Pavi Subenderan created RANGER-3442:
---------------------------------------
Summary: Ranger KMS DAO memory issues when many new keys are
created
Key: RANGER-3442
URL: https://issues.apache.org/jira/browse/RANGER-3442
Project: Ranger
Issue Type: Bug
Components: kms
Affects Versions: 2.0.0
Reporter: Pavi Subenderan
We have many keys created in our KMS keystore and recently we found that when
we create new keys, the KMS instances easily hit against the memory limit.
We can reproduce this with a script to call KMS createKey and then getMetadata
for new keys in a loop. Basically we restart our instances and memory usage is
approximately 1.5GB out of 8GB, but after running this script for a bit (1-5
minutes), we hit close to the 8GB limit and the memory usage does not go back
down after that.
I did a heap dump and saw that most of the memory was being retained by
XXRangerKeystore and eclipse EntityManagerImpl.
org.eclipse.persistence.internal.jpa.EntityManagerImpl
org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork
And the largest shallow size object was char[] with 4GB+...
I was ultimately able to solve this issue by adding an
getEntityManager().clear() call in BaseDao.java getAllKeys().
After I added this fix, we can now run as many KMS CreateKey / getMetadata
calls as we want without any increase in memory usage whatsoever. Memory usage
now stays constant at <1.7GB.
My understanding is that Ranger KMS has a many instances of ThreadLocal
EntityManager (160+ according to my heap dump) which each held a cache of the
results for getAllKeys. Since we have so many keys in our KMS, this would
easily put as at the memory limit.
Not sure if there are any drawbacks to clearing EntityManager in
BaseDao.getAllKeys() but we are seeing greatly improved performance in our case
since we aren't constantly hitting the memory limit anymore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
