-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/
-----------------------------------------------------------
(Updated Sept. 30, 2017, 11:51 a.m.)
Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, Sarath
Subramanian, and Suma Shivaprasad.
Bugs: ATLAS-2075
https://issues.apache.org/jira/browse/ATLAS-2075
Repository: atlas
Description
-------
Update notification module to compress and split large messages (> 1mb) that
can't be sent via Kafka. When such a message is sent, the message is first
compressed via GZip. If compressed message is within the size limit, the
message is sent via Kafka. If the compressed size is larger, then the message
is split into multiple small messages and sent via Kafka.
Notification consumer is updated to merge and uncompress messages before the
message is processed further.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
d3b4e49e
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
38889ef8
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
ec99372d
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
cb44fc68
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/MessageVersion.java
6ef407ac
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
956c85e0
notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
1929eb46
notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
cc2099e7
notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java
9b712f44
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
b7474a0e
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
3b2a093b
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
61107a9f
notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
587b7ebd
notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
be324277
notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
3724fd5f
Diff: https://reviews.apache.org/r/62681/diff/5/
Changes: https://reviews.apache.org/r/62681/diff/4-5/
Testing
-------
- verified that large messages are compressed and split by Atlas hook, before
sending to Kafka
- verified that Atlas server processes compressed, multi-part messages correctly
Thanks,
Madhan Neethiraj
