-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68844/#review209059
-----------------------------------------------------------


Ship it!




Ship It!

- Sarath Subramanian


On Sept. 26, 2018, 12:38 p.m., Madhan Neethiraj wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68844/
> -----------------------------------------------------------
> 
> (Updated Sept. 26, 2018, 12:38 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, keval bhatt, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2891
>     https://issues.apache.org/jira/browse/ATLAS-2891
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Processing of hook notifications at Atlas server was updated to skip 
> hive_column_lineage that is likely to contain incorrect lineage information 
> due to HIVE-20633. hive_column_lineage is skipped only when both the 
> conditions listed below are met:
> - hive_column_lineage should have more than 5 inputs (configurable via 
> atlas.notification.consumer.skip.hive_column_lineage.inputs.threshold)
> - all hive_column_lineage entities in the message should have same number of 
> inputs
> 
> This is can be disabled by setting configuration 
> atlas.notification.consumer.skip.hive_column_lineage.inputs.threshold to a 
> negative value.
> 
> 
> Diffs
> -----
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  b9e4256be 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 19075f61c 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
>  c99a699b2 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> e3bb71ca8 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1a567af5c 
> 
> 
> Diff: https://reviews.apache.org/r/68844/diff/3/
> 
> 
> Testing
> -------
> 
> Verified that the notification processing skips potentially incorrect 
> hive_column_lineage; and following WARN logs is printed for each message 
> where hive_column_lineage was skiped.
> 
> 2018-09-25 12:41:27,536 WARN  - [NotificationHookConsumer thread-0:] ~ 
> removed 120 hive_column_lineage entities, each having 241 inputs. 
> offset=27648, partition=0 (NotificationHookConsumer:687)
> 2018-09-25 12:41:46,799 WARN  - [NotificationHookConsumer thread-0:] ~ 
> removed 21 hive_column_lineage entities, each having 43 inputs. offset=27786, 
> partition=0 (NotificationHookConsumer:687)
> 2018-09-25 12:42:07,646 WARN  - [NotificationHookConsumer thread-0:] ~ 
> removed 26 hive_column_lineage entities, each having 53 inputs. offset=27947, 
> partition=0 (NotificationHookConsumer:687)
> 2018-09-25 12:42:19,210 WARN  - [NotificationHookConsumer thread-0:] ~ 
> removed 28 hive_column_lineage entities, each having 57 inputs. offset=28031, 
> partition=0 (NotificationHookConsumer:687)
> 2018-09-25 12:45:26,062 WARN  - [NotificationHookConsumer thread-0:] ~ 
> removed 584 hive_column_lineage entities, each having 1172 inputs. 
> offset=28605, partition=0 (NotificationHookConsumer:687)
> 2018-09-25 12:45:39,153 WARN  - [NotificationHookConsumer thread-0:] ~ 
> removed 584 hive_column_lineage entities, each having 1172 inputs. 
> offset=28628, partition=0 (NotificationHookConsumer:687)
> 2018-09-25 12:46:05,157 WARN  - [NotificationHookConsumer thread-0:] ~ 
> removed 584 hive_column_lineage entities, each having 1172 inputs. 
> offset=28653, partition=0 (NotificationHookConsumer:687)
> 2018-09-25 12:46:14,670 WARN  - [NotificationHookConsumer thread-0:] ~ 
> removed 33 hive_column_lineage entities, each having 68 inputs. offset=28665, 
> partition=0 (NotificationHookConsumer:687)
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Reply via email to