-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/
-----------------------------------------------------------

(Updated July 22, 2020, 4:16 p.m.)


Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and 
Sarath Subramanian.


Changes
-------

Updates include:
- Added configuration property 
atlas.notification.fixed.buffer.items.increment.count with default set to 10.


Bugs: ATLAS-3878
    https://issues.apache.org/jira/browse/ATLAS-3878


Repository: atlas


Description
-------

**Background**
See JIRA for details.

*Analysis* Using memory profiling tools, it was observed that large number of 
notification objects were created. These stayed in memory and later were 
promoted to higher generation, thereby taking even longer to be collected.

**Approach**
Using the fixed-buffer approach to address the problem of creating large number 
of small objects.

New *FixedBufferList* This is an encapsulation over *ArrayList*. During initial 
allocation, list is populated with default values. Features:
- Setting of values to these pre-allocated objects is achieved by first doing a 
*get* on the element and then assigning values to it.
- *toList* fetches the sub-list from the encapsulating list. This uses the 
state within the class to fetch the right length for the returning array.

New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
necessary since the list is now part class's state.
Modified *EntityAuditListenerV2* Uses the new classes.
Modifed *EntityNotificationListener* Uses the new classes.

**Verification**
- Using the test setup, the memory usage was observed over a period of 24 hrs. 
- Memory usage and object allocation was obvserved using memory profiler.


Diffs (updated)
-----

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
  intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/utils/FixedBufferListAccessor.java 
PRE-CREATION 
  intg/src/test/java/org/apache/atlas/utils/FixedBufferListAccessorTest.java 
PRE-CREATION 
  intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
 79527acfa 
  
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
 a677b315c 


Diff: https://reviews.apache.org/r/72666/diff/6/

Changes: https://reviews.apache.org/r/72666/diff/5-6/


Testing
-------

**Unit testing**
Unit tests added for the new classes.

**Volume testing**
Setup:
- Node: Threads 40, Core: 40, Allocated Memory: 12 GB
- Multiple Kafka queues ingesting data.
- Bulk entity creation using custom script ingesting 100M entities.

Memory usage stayed between 0 and 5% during the 24 hr period.

With:
- Workers: 64
- Batch size: 50 (fewer elements in batch improve commit time and audit write 
time).
- Throughput: ~1.2 M entities per hour. Without out of memory error.

**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/


Thanks,

Ashutosh Mestry

Reply via email to