[ https://issues.apache.org/jira/browse/ATLAS-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Mestry resolved ATLAS-3878. ------------------------------------ Resolution: Fixed > Notifications: Improve Memory Usage in Scale Enviroment > ------------------------------------------------------- > > Key: ATLAS-3878 > URL: https://issues.apache.org/jira/browse/ATLAS-3878 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Affects Versions: 2.0.0, trunk > Reporter: Ashutosh Mestry > Assignee: Ashutosh Mestry > Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3878-Notifications-Improve-Memory-Usage-in-Sca-v2.patch > > > *Background* > As part of entity creation, Atlas sends notifications of different types. > Current implementation, to listeners. Listeners in turn perform specific > tasks. > At a more concrete level, the _EntityAuditListenerV2_ will write audits and > the _NotificationEntityChangeListener_ will send Kafka notifications. > Each of the listeners create notification objects. The notification objects > are large in number and are short lived. > The transient nature of the notification objects causes memory pressure in > scale environment. > *Solution* > Create object pool for notification objects. This way objects can be > reused.and existing design can be kept in tact. This will also offer benefit > of using existing test setup for verification. > *Tests Used* > _Setup_ > Create a test rig that will spawn multiple works that will invoke Atlas' bulk > APIs for entity creation. > Node: 40 workers, 8 GB allocated memory and 40 cores. > _Observation_ > About 40 mins into the exercise, memory pressure builds up causing GC > collects to take longer. This causes ZK timeout and finally Atlas process > crashes. -- This message was sent by Atlassian Jira (v8.3.4#803005)