Wei-Chiu Chuang created HADOOP-16876:
----------------------------------------
Summary: KMS delegation tokens are memory expensive
Key: HADOOP-16876
URL: https://issues.apache.org/jira/browse/HADOOP-16876
Project: Hadoop Common
Issue Type: Bug
Components: kms
Reporter: Wei-Chiu Chuang
Attachments: Screen Shot 2020-02-20 at 5.04.12 PM.png
We recently saw a number of users reporting memory consumption in KMS.
Part of the reason being HADOOP-14445. Without that, the number of kms
delegation tokens that zookeeper stores is proportional to the number of KMS
servers.
There are two problems:
(1) it exceeds zookeeper jute buffer length and operations fail.
(2) KMS use more heap memory to store KMS DTs.
But even with HADOOP-14445, KMS DTs are still expensive. Looking at a heap dump
from KMS, the majority of the heap is occupied by znode and KMS DT objects.
With the growing number of encrypted clusters and use cases, this is
increasingly a problem our users encounter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
