[
https://issues.apache.org/jira/browse/SPARK-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088092#comment-16088092
]
Apache Spark commented on SPARK-21408:
--------------------------------------
User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/18639
> Default RPC dispatcher thread pool size too large for small executors
> ---------------------------------------------------------------------
>
> Key: SPARK-21408
> URL: https://issues.apache.org/jira/browse/SPARK-21408
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.3.0
> Reporter: Marcelo Vanzin
> Priority: Minor
>
> This is the code that sizes the RPC dispatcher thread pool:
> {noformat}
> private val threadpool: ThreadPoolExecutor = {
> val numThreads =
> nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
> math.max(2, Runtime.getRuntime.availableProcessors()))
> val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads,
> "dispatcher-event-loop")
> {noformat}
> That is based on the number of available cores on the host, instead of the
> number of cores the executor was told to use. Meaning if you start an
> executor with a single "core" on a host with 64 CPUs, you'll get 64 threads,
> which is kinda overkill.
> Using the allocated cores + a lower bound is probably a better approach.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
