-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61807/
-----------------------------------------------------------
(Updated Sept. 25, 2017, 6:10 p.m.)
Review request for atlas, Madhan Neethiraj and Nixon Rodrigues.
Changes
-------
Updates include:
- Moved changes related to JSON compression to _JsonUtils_.
- Additional refactoring for simplicity.
Bugs: ATLAS-2075
https://issues.apache.org/jira/browse/ATLAS-2075
Repository: atlas
Description (updated)
-------
**Approach**
- Message size is checked. It is split into parts if it exceeds the size
threshold.
- New message type _Split_ derived from _VersionedMessage_ holds information of
the individual parts of the message.
- _MessageSplitter_ splits message into smaller packets and encloses them in a
message of type _Split_.
- _MessageCombiner_ combines array of split messages.
- _VersionedMessageDeserializer_ is modified to support the new message type.
**Additional configuration item**
- _atlas.notification.message.size.threshold_ Needs to be added to Hive's atlas
configuration as well as _Atlas_'s own _atlas-application.properties_.
- Default is set to a number closer to 1MB.
**For testing**
- Set threshold to a low number.
- Update _atlas-application.properties_:
- Add _atlas.notification.message.size.threshold_ to 1024.
Diffs (updated)
-----
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
d3b4e49e
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
38889ef8
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
cb44fc68
notification/src/main/java/org/apache/atlas/notification/MessageCombiner.java
PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/MessageTransformer.java
PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/MessageVersion.java
6ef407ac
notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
1929eb46
notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
cc2099e7
notification/src/main/java/org/apache/atlas/notification/VersionedMessageHeader.java
PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/messages/Compressed.java
PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/messages/Split.java
PRE-CREATION
notification/src/main/java/org/apache/atlas/utils/JsonUtils.java PRE-CREATION
notification/src/main/java/org/apache/atlas/utils/StringSplitterCombiner.java
PRE-CREATION
notification/src/test/java/org/apache/atlas/hook/StringSplitCombineTest.java
PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
3b2a093b
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
61107a9f
notification/src/test/java/org/apache/atlas/notification/MessageCombinerTest.java
PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/MessageSplitterTest.java
PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/SplitCombinerUtils.java
PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
be324277
notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
3724fd5f
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java
dd3257e0
notification/src/test/java/org/apache/atlas/utils/TestJsonUtils.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
2669c739
webapp/src/test/java/org/apache/atlas/util/JsonBenchmarkTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/61807/diff/3/
Changes: https://reviews.apache.org/r/61807/diff/2-3/
Testing
-------
**Unit tests**
- Added unit tests for new functionality.
- Refactored few of existing unit tests.
- Added tests to verify backward compatibility.
**Functional tests**
- Verified regular flow.
- Verified cases:
- Case where server is shutdown before all the parts of a message are
committed.
Thanks,
Ashutosh Mestry
