-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/
-----------------------------------------------------------
Review request for atlas, Radhika Kundam and Sarath Subramanian.
Bugs: ATLAS-4152
https://issues.apache.org/jira/browse/ATLAS-4152
Repository: atlas
Description
-------
**Background**
As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided the
poperty is enabled).
When Spooling enabled the order of messages can potentially change. The
notification messages coming from HS2 and HMS may not be in the same order as
when they arrived with direct notification.
Problem:
Consider the sequence of arriving messages:
This is the sequence of messages for Entity 1 (C = create, U = update, D =
delete, L?x = Lineage of type 'x')
No problem: C1, U1, L1x, L1y, D1
Problem: C1, U1, D1, L1x, L1y
This implementation attempts to handle ths problem mentioned above.
**Approach**
Used bounded stream approach where an incoming stream of messages is bounded
with an indicator that it originates from spool. This helps makes localized
decisions on the incoming stream of messages.
Diffs
-----
intg/src/main/java/org/apache/atlas/model/notification/HookNotification.java
5b5fa04e2
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
2d7d19595
notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
a9a3a78cc
webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
84cc8d813
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java
89568e236
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDdlLineagePreprocessor.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
e69d63e3a
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java
608b4a304
webapp/src/test/resources/json/hive-ddl.json PRE-CREATION
Diff: https://reviews.apache.org/r/73329/diff/1/
Testing
-------
**Functional tests**
Manual verification of scenarios.
Thanks,
Ashutosh Mestry
