-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73418/
-----------------------------------------------------------
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Radhika Kundam,
Sarath Subramanian, Sidharth Mishra, and Umesh Padashetty.
Bugs: ATLAS-4335
https://issues.apache.org/jira/browse/ATLAS-4335
Repository: atlas
Description
-------
There are 4 parts to this project.
---------------------------------------------------------
The first part is Notification Rest Service.
In this part a new post API (api/atlas/v2/notification/topic/\<topicName>) is
created.
NotificationREST.java is the entry point having a method handleNotifications.
It does verify the authorization of the incoming requests.
It further checks if the provided topic-name is among valid supported topic
names.
After validations it handover incoming messages to the Kafka notifier which in
turn does the job of bumping messages in Kafka topic.
An Integration test(NotificationRestIT.java) is also created to cover this new
API
Impacted files are listed below.
webapp/src/main/java/org/apache/atlas/web/rest/NotificationREST.java
authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
webapp/src/test/java/org/apache/atlas/web/integration/NotificationRestIT.java
webapp/src/test/resources/json/notifications/create-db-ddl.json
webapp/src/test/resources/json/notifications/create-db.json
webapp/src/test/resources/json/notifications/delete-db.json
-----------------------------------------------------------
The second part is Rest Notifier
In this part, a new Notifier(RestNotification.java) is created. It extends the
AbstractNotification interface and works on the same line as KafkaNotification
and AtlasFileSpool, it just that it uses AtlasClientV2 to send the
notifications to the abovementioned rest API.
A new API endpoint is added in the AtlasClientV2. NotificationProvider is
modified to instantiate the new rest notifier based on the configuration
flag(atlas.hook.rest.notification.enabled)
Unit test(RestNotificationTest.java) has been added to cover basic flows.
AtlasBaseClient is modified to make one of the static variables available to
the mentioned unit test.
Impacted files are listed below.
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
notification/src/main/java/org/apache/atlas/kafka/NotificationProvider.java
notification/src/main/java/org/apache/atlas/notification/rest/RestNotification.java
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
notification/src/test/java/org/apache/atlas/notification/RestNotificationTest.java
client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
-------------------------------------------------------------
The third part is Buffered Kafka consumer (tumbling window solution)
This part addresses the reordering of out or order notifications. With the
introduction of a new notification delivery mechanism based on
rest(RestNotification), it was observed that notifications were landing on the
Kafka, not in the same order as they were produced at the services.
Particularly when two different services generate events by the same
trigger(HMS and HS2). This problem was there with the KafkaNotification
delivery mechanism as well, but it never surfaces so frequently as there is a
delay in the HS2 events, and the Kafka client delivers messages almost
instantaneously to Kafka. With the rest route, the delivery time is not
instantaneous, especially in a non-kerbarised or non-token-based authentication
setup.
To cover this AtlasKafkaBufferedConsumer is created. It’s a tumbling window
solution on the stream of events. Here while consuming events, instead of
reading events one by one, it loads events from Kafka in batches(size = 10
events by default). It has 2 such batches in memory, the current batch, and the
lookup batch. In a cycle, it processes the current batch, but before
processing, it pulls into the current batch all events in the lookup batch
having timestamp lower than the greatest event timestamp in the current batch.
It also sorts events in the current batch by event timestamp, thus fixes the
order of messages.
AtlasKafkaBufferedConsumer's constructor can throw AtlasException therefore
KafkaNotification and NotificationHookConsumer are modified to cover for the
exception.
Unit test(AtlasKafkaBufferedConsumer) was added to cover the flow extensively.
Impacted files are listed below.
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaBufferedConsumer.java
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
notification/src/test/java/org/apache/atlas/kafka/AtlasKafkaBufferedConsumerTest.java
-----------------------------------------------------------------
The fourth part is Passing Message creation time along with a message to Hook
Executor.
Hook messages are wrapped into AtlasNotificationMessage, which also stores the
time stamp. However, this timestamp bares the wrapper creation time and the
actual message creation time. AtlasHook buffers the incoming messages from the
underlying service into a queue(Courtesy ThreadPoolExecutor). It processes
these queued events sequentially one by one (as by default pool has a single
thread) and, the event is wrapped into AtlasNotificationMessage when it is
picked for processing. Therefore timestamp wrapper has lagged the actual event
creation time by the number of milliseconds the event stays on the queue. This
has not been a problem with KafkaNotifier as the delivery mechanism was very
fast and events were not staying on for more than a few milliseconds. However,
in the case of RestNotification, events end us staying on the queue for a
significant amount of milliseconds as the delivery mechanism is comparatively
slow. This is a critical problem is the Buffered Kafka consumer mention
ed in the previous part relies on the event creation time.
To fix this problem, event creation time is passed along with the messages to
the queue. After dequeue, actual creation time is passed into the constructor
of the AtlasNotificationMessage.
A unit test is added in AbstractNotificationTest to cover this case. Also,
AtlasHookTest is modified to cover additional timestamp parameters.
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationStringMessage.java
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java
Diffs
-----
addons/falcon-bridge/pom.xml e0d2f3be7
addons/hbase-bridge/pom.xml 50fb9e838
addons/hive-bridge/pom.xml ea2de4d3b
addons/impala-bridge/pom.xml a1bd5920b
addons/sqoop-bridge/pom.xml 9d5ac52d9
authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java
5d06e1b29
client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java eb0e630f9
client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java cb35c9446
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 9ef848752
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 2febff469
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
ff45d5713
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
5869910cb
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationStringMessage.java
bc8442785
notification/pom.xml 28d13bc13
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaBufferedConsumer.java
PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java
af3727df4
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
32f5183a0
notification/src/main/java/org/apache/atlas/kafka/NotificationProvider.java
b35af97fd
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
c45a1da95
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
b43bc7c66
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
3d8d9cc0a
notification/src/main/java/org/apache/atlas/notification/rest/RestNotification.java
PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
0c92c300e
notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c
notification/src/test/java/org/apache/atlas/kafka/AtlasKafkaBufferedConsumerTest.java
PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
8078a6ceb
notification/src/test/java/org/apache/atlas/notification/RestNotificationTest.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
5643af95c
webapp/src/main/java/org/apache/atlas/web/rest/NotificationREST.java
PRE-CREATION
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
fdfc2560d
webapp/src/test/java/org/apache/atlas/web/integration/NotificationRestIT.java
PRE-CREATION
webapp/src/test/resources/json/notifications/create-db-ddl.json PRE-CREATION
webapp/src/test/resources/json/notifications/create-db.json PRE-CREATION
webapp/src/test/resources/json/notifications/delete-db.json PRE-CREATION
Diff: https://reviews.apache.org/r/73418/diff/1/
Testing
-------
Thanks,
Deep Singh