----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68844/#review209059 -----------------------------------------------------------
Ship it! Ship It! - Sarath Subramanian On Sept. 26, 2018, 12:38 p.m., Madhan Neethiraj wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68844/ > ----------------------------------------------------------- > > (Updated Sept. 26, 2018, 12:38 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, keval bhatt, Nixon > Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-2891 > https://issues.apache.org/jira/browse/ATLAS-2891 > > > Repository: atlas > > > Description > ------- > > Processing of hook notifications at Atlas server was updated to skip > hive_column_lineage that is likely to contain incorrect lineage information > due to HIVE-20633. hive_column_lineage is skipped only when both the > conditions listed below are met: > - hive_column_lineage should have more than 5 inputs (configurable via > atlas.notification.consumer.skip.hive_column_lineage.inputs.threshold) > - all hive_column_lineage entities in the message should have same number of > inputs > > This is can be disabled by setting configuration > atlas.notification.consumer.skip.hive_column_lineage.inputs.threshold to a > negative value. > > > Diffs > ----- > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java > b9e4256be > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > 19075f61c > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java > c99a699b2 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > e3bb71ca8 > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > 1a567af5c > > > Diff: https://reviews.apache.org/r/68844/diff/3/ > > > Testing > ------- > > Verified that the notification processing skips potentially incorrect > hive_column_lineage; and following WARN logs is printed for each message > where hive_column_lineage was skiped. > > 2018-09-25 12:41:27,536 WARN - [NotificationHookConsumer thread-0:] ~ > removed 120 hive_column_lineage entities, each having 241 inputs. > offset=27648, partition=0 (NotificationHookConsumer:687) > 2018-09-25 12:41:46,799 WARN - [NotificationHookConsumer thread-0:] ~ > removed 21 hive_column_lineage entities, each having 43 inputs. offset=27786, > partition=0 (NotificationHookConsumer:687) > 2018-09-25 12:42:07,646 WARN - [NotificationHookConsumer thread-0:] ~ > removed 26 hive_column_lineage entities, each having 53 inputs. offset=27947, > partition=0 (NotificationHookConsumer:687) > 2018-09-25 12:42:19,210 WARN - [NotificationHookConsumer thread-0:] ~ > removed 28 hive_column_lineage entities, each having 57 inputs. offset=28031, > partition=0 (NotificationHookConsumer:687) > 2018-09-25 12:45:26,062 WARN - [NotificationHookConsumer thread-0:] ~ > removed 584 hive_column_lineage entities, each having 1172 inputs. > offset=28605, partition=0 (NotificationHookConsumer:687) > 2018-09-25 12:45:39,153 WARN - [NotificationHookConsumer thread-0:] ~ > removed 584 hive_column_lineage entities, each having 1172 inputs. > offset=28628, partition=0 (NotificationHookConsumer:687) > 2018-09-25 12:46:05,157 WARN - [NotificationHookConsumer thread-0:] ~ > removed 584 hive_column_lineage entities, each having 1172 inputs. > offset=28653, partition=0 (NotificationHookConsumer:687) > 2018-09-25 12:46:14,670 WARN - [NotificationHookConsumer thread-0:] ~ > removed 33 hive_column_lineage entities, each having 68 inputs. offset=28665, > partition=0 (NotificationHookConsumer:687) > > > Thanks, > > Madhan Neethiraj > >
