----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73329/#review223017 -----------------------------------------------------------
common/src/main/java/org/apache/atlas/repository/Constants.java Lines 138 (patched) <https://reviews.apache.org/r/73329/#comment312171> "deletedMessageTimestamp" => entityDeletedTimestamp DELETED_MESSAGE_TIMESTAMP_PROPERTY_KEY => ENTITY_DELETED_TIMESTAMP_PROPERTY_KEY notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java Lines 216 (patched) <https://reviews.apache.org/r/73329/#comment312175> consider adding a comment - explaining the reason for adding sleep for hs2 messages in spool repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java Lines 377 (patched) <https://reviews.apache.org/r/73329/#comment312172> reorder the garph query conditions for better perf: graph.query().has(STATE_PROPERTY_KEY, Status.DELETED.name()) .has(Constants.DELETED_MESSAGE_TIMESTAMP_PROPERTY_KEY, GREATER_THAN, timestamp) .has(Constants.QUALIFIED_NAME, qualifiedName) .orderBy(Constants.DELETED_MESSAGE_TIMESTAMP_PROPERTY_KEY, ASC); webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDbDDLPreprocessor.java Lines 38 (patched) <https://reviews.apache.org/r/73329/#comment312173> dbRelationship => dbObject webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveTableDDLPreprocessor.java Lines 38 (patched) <https://reviews.apache.org/r/73329/#comment312174> tableObject - Sarath Subramanian On May 19, 2021, 9 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73329/ > ----------------------------------------------------------- > > (Updated May 19, 2021, 9 p.m.) > > > Review request for atlas, Radhika Kundam and Sarath Subramanian. > > > Bugs: ATLAS-4152 > https://issues.apache.org/jira/browse/ATLAS-4152 > > > Repository: atlas > > > Description > ------- > > **Background** > As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided > the poperty is enabled). > > When Spooling enabled the order of messages can potentially change. The > notification messages coming from HS2 and HMS may not be in the same order as > when they arrived with direct notification. > > Problem: > Consider the sequence of arriving messages: > > This is the sequence of messages for Entity 1 (C = create, U = update, D = > delete, L?x = Lineage of type 'x') > No problem: C1, U1, L1x, L1y, D1 > Problem: C1, U1, D1, L1x, L1y > > This implementation attempts to handle ths problem mentioned above. If the > above case is not handled, it will end up creating shell entities, since > deleted entities are not looked up as part of entity creation. > > **Approach** > Used bounded stream approach where an incoming stream of messages is bounded > with an indicator that it originates from spool. This helps makes localized > decisions on the incoming stream of messages. > > High-level approach: > - Messages when written to the spool are tagged with a timestamp. > - Deleted entities are maintained in a cache. > - Lineage-only message are checked if they refer to a deleted entity. > - If they refer to deleted entity, they are stitched to the one present in > the cache only if it falls within the threshold. > - Using step-climbing approach for locating right entity to stitch lineage to. > > New: _EntityCorrelationsManager_: Uses message timestamp and cached entity > qualifiedName-GUID map. > Modifed: _NotificationHookConsumer_ Uses the new class. > New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted > entities. > Modified: _SpoolConfiguration_: Added new configuration to pause message > sending after destination is available: > _atlas.hook.spool.pause.before.send.sec_. > In-memory lookup approach changed to persistent loookup. > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/repository/Constants.java ffcec9743 > > intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java > 810ba97c9 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > f7d9668ec > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java > 22bd79fdf > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 3d1b3ccf1 > > notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java > 3264e264c > > notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java > edd8ed931 > > notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java > 2d7d19595 > > notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java > 22242c933 > > notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java > a9a3a78cc > notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java > 2cacaaadc > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java > d7e4959f7 > > notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java > 167efbecc > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > cc727c6ba > > repository/src/main/java/org/apache/atlas/repository/store/graph/EntityCorrelationStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 0a9470815 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java > PRE-CREATION > > webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java > PRE-CREATION > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > 84cc8d813 > > webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java > 89568e236 > > webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDbDDLPreprocessor.java > PRE-CREATION > > webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java > 86e3384ee > > webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveTableDDLPreprocessor.java > PRE-CREATION > > webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java > 608b4a304 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java > 65e8b5001 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java > 15a19005e > > > Diff: https://reviews.apache.org/r/73329/diff/8/ > > > Testing > ------- > > **Functional tests** > Manual verification of scenarios. > > **Unit tests** > New tests added. > > **Test data** > 11:55: Kafka Down! > 12:00: create table t01(c01 string); > 12:10: create view t06_vw as select * from t01; > 12:20: create view t06_vw_1 as select * from t01; > 12:30: create view t06_vw_2 as select * from t01; > 12:40: create view t06_vw_3 as select * from t01; > 12:50: DROP TABLE t01; > 12:52: create table t01(c01 string); > 12:53: create view t06_vwx as select * from t01; > 12:54: create view t06_vwx_1 as select * from t01; > 12:55: create view t06_vwx_2 as select * from t01; > 12:56: create view t06_vwx_3 as select * from t01; > 12:57: DROP TABLE T01; > 12:58: create table t01(c01 string); > 01:00: Kafka UP! > > > Thanks, > > Ashutosh Mestry > >
