-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/
-----------------------------------------------------------
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and
Sarath Subramanian.
Bugs: ATLAS-3878
https://issues.apache.org/jira/browse/ATLAS-3878
Repository: atlas
Description
-------
**Background**
See JIRA for details.
*Analysis* Using memory profiling tools, it was observed that large number of
notification objects were created. These stayed in memory and later were
promoted to higher generation, thereby taking even longer to be collected.
**Approach**
Using the fixed-buffer approach to address the problem of creating large number
of small objects.
New *FixedBufferList* This is an encapsulation over *ArrayList*. During initial
allocation, list is populated with default values. Features:
- Setting of values to these pre-allocated objects is achieved by first doing a
*get* on the element and then assigning values to it.
- *toList* fetches the sub-list from the encapsulating list. This uses the
state within the class to fetch the right length for the returning array.
New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is
necessary since the list is now part class's state.
Modified *EntityAuditListenerV2* Uses the new classes.
Modifed *EntityNotificationListener* Uses the new classes.
**Verification**
- Using the test setup, the memory usage was observed over a period of 24 hrs.
- Memory usage and object allocation was obvserved using memory profiler.
Diffs
-----
intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION
intg/src/main/java/org/apache/atlas/utils/NamedFixedBufferList.java
PRE-CREATION
intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
79527acfa
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
a677b315c
Diff: https://reviews.apache.org/r/72666/diff/1/
Testing
-------
**Unit testing**
Unit tests added for the new classes.
**Volume testing**
Setup:
- Node: Threads 40, Core: 40, Allocated Memory: 12 GB
- Multiple Kafka queues ingesting data.
- Bulk entity creation using custom script ingesting 100M entities.
Memory usage stayed between 0 and 5% during the 24 hr period.
**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
Thanks,
Ashutosh Mestry
