-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68844/
-----------------------------------------------------------

Review request for atlas, Apoorv Naik, Ashutosh Mestry, keval bhatt, Nixon 
Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-2891
    https://issues.apache.org/jira/browse/ATLAS-2891


Repository: atlas


Description
-------

Processing of hook notifications at Atlas server was updated to skip 
hive_column_lineage that is likely to contain incorrect lineage information due 
to HIVE-20633. hive_column_lineage is skipped only when both the conditions 
listed below are met:
- hive_column_lineage should have more than 5 inputs (configurable via 
atlas.notification.consumer.skip.hive_column_lineage.inputs.threshold)
- all hive_column_lineage entities in the message should have same number of 
inputs

This is can be disabled by setting configuration 
atlas.notification.consumer.skip.hive_column_lineage.inputs.threshold to a 
negative value.


Diffs
-----

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
 f465e49b8 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
e3bb71ca8 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 1a567af5c 


Diff: https://reviews.apache.org/r/68844/diff/1/


Testing
-------

Verified that the notification processing skips potentially incorrect 
hive_column_lineage; and following WARN logs is printed for each message where 
hive_column_lineage was skiped.

2018-09-25 12:41:27,536 WARN  - [NotificationHookConsumer thread-0:] ~ removed 
120 hive_column_lineage entities, each having 241 inputs. offset=27648, 
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:41:46,799 WARN  - [NotificationHookConsumer thread-0:] ~ removed 
21 hive_column_lineage entities, each having 43 inputs. offset=27786, 
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:42:07,646 WARN  - [NotificationHookConsumer thread-0:] ~ removed 
26 hive_column_lineage entities, each having 53 inputs. offset=27947, 
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:42:19,210 WARN  - [NotificationHookConsumer thread-0:] ~ removed 
28 hive_column_lineage entities, each having 57 inputs. offset=28031, 
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:45:26,062 WARN  - [NotificationHookConsumer thread-0:] ~ removed 
584 hive_column_lineage entities, each having 1172 inputs. offset=28605, 
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:45:39,153 WARN  - [NotificationHookConsumer thread-0:] ~ removed 
584 hive_column_lineage entities, each having 1172 inputs. offset=28628, 
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:46:05,157 WARN  - [NotificationHookConsumer thread-0:] ~ removed 
584 hive_column_lineage entities, each having 1172 inputs. offset=28653, 
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:46:14,670 WARN  - [NotificationHookConsumer thread-0:] ~ removed 
33 hive_column_lineage entities, each having 68 inputs. offset=28665, 
partition=0 (NotificationHookConsumer:687)


Thanks,

Madhan Neethiraj

Reply via email to