Dongjoon Hyun created SPARK-22374:
-------------------------------------
Summary: STS ran into OOM in a secure cluster
Key: SPARK-22374
URL: https://issues.apache.org/jira/browse/SPARK-22374
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0
Reporter: Dongjoon Hyun
In a secure cluster, FileSystem.CACHE grows indefinitely.
*ENVIRONMENT*
1. `spark.yarn.principal` and `spark.yarn.keytab` is used.
2. Spark Thrift Server run with `doAs` false.
{code}
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
{code}
With 6GB (-Xmx6144m) options, `HiveConf` consumes 4GB inside FileSystem.CACHE.
{code}
20,030 instances of "org.apache.hadoop.hive.conf.HiveConf", loaded by
"sun.misc.Launcher$AppClassLoader @ 0x64001c160" occupy 4,418,101,352 (73.42%)
bytes. These instances are referenced from one instance of
"java.util.HashMap$Node[]", loaded by "<system class loader>"
{code}
Please see the attached images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
