GitHub user rajeshbalamohan opened a pull request:
https://github.com/apache/spark/pull/10846
SPARK-12920. [SQL]. Spark thrift server can run at very high CPU withâ¦
Spark thrift server runs at very high CPU when concurrent users submit
queries to the system over a period of time. Profiling revealed it is due to
many Conf objects getting cacheed in HadoopRDD and this causing lots of
pressure for GC when running queries on datasets with large number of
partitions. Also, job UI retention causes issues with large jobs. Details are
mentioned in SPARK-12920 and profiler snapshots are attached. Fix introduces
"spark.hadoop.cacheConf" to optionally cache the jobConf in HadoopRDD.
JobProgressListener fixes are related to trimming the job/stage details.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rajeshbalamohan/spark SPARK-12920
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10846.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10846
----
commit 12d77beb0c6c4e3f336aab43a5907775da715753
Author: Rajesh Balamohan <[email protected]>
Date: 2016-01-20T09:14:36Z
SPARK-12920. [SQL]. Spark thrift server can run at very high CPU with
concurrent users
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
