-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/
-----------------------------------------------------------
(Updated May 3, 2021, 8:59 p.m.)
Review request for atlas, Radhika Kundam and Sarath Subramanian.
Changes
-------
Additional explanation.
Bugs: ATLAS-4152
https://issues.apache.org/jira/browse/ATLAS-4152
Repository: atlas
Description (updated)
-------
**Background**
As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided the
poperty is enabled).
When Spooling enabled the order of messages can potentially change. The
notification messages coming from HS2 and HMS may not be in the same order as
when they arrived with direct notification.
Problem:
Consider the sequence of arriving messages:
This is the sequence of messages for Entity 1 (C = create, U = update, D =
delete, L?x = Lineage of type 'x')
No problem: C1, U1, L1x, L1y, D1
Problem: C1, U1, D1, L1x, L1y
This implementation attempts to handle ths problem mentioned above. If the
above case is not handled, it will end up creating shell entities, since
deleted entities are not looked up as part of entity creation.
**Approach**
Used bounded stream approach where an incoming stream of messages is bounded
with an indicator that it originates from spool. This helps makes localized
decisions on the incoming stream of messages.
High-level approach:
- Messages when written to the spool are tagged with a timestamp.
- Deleted entities are maintained in a cache.
- Lineage-only message are checked if they refer to a deleted entity.
- If they refer to deleted entity, they are stitched to the one present in the
cache only if it falls within the threshold.
New: _EntityCorrelationsManager_: Uses message timestamp and cached entity
qualifiedName-GUID map.
Modifed: _NotificationHookConsumer_ Uses the new class.
New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted
entities.
Diffs
-----
intg/src/main/java/org/apache/atlas/model/notification/HookNotification.java
5b5fa04e2
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
2d7d19595
notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
a9a3a78cc
webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
84cc8d813
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java
89568e236
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDdlLineagePreprocessor.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
e69d63e3a
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java
608b4a304
webapp/src/test/resources/json/hive-ddl.json PRE-CREATION
Diff: https://reviews.apache.org/r/73329/diff/1/
Testing
-------
**Functional tests**
Manual verification of scenarios.
Thanks,
Ashutosh Mestry
