[
https://issues.apache.org/jira/browse/SPARK-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259730#comment-15259730
]
Dongjoon Hyun commented on SPARK-14936:
---------------------------------------
Hi, [~joshrosen].
I investigated a little and found that the root cause is
`ThreadPoolExecutor.awaitTermination` of `Dispatcher.scala`.
{code}
def awaitTermination(): Unit = {
// val startTime = System.nanoTime()
threadpool.awaitTermination(Long.MaxValue, TimeUnit.MILLISECONDS)
// val elapsedTime = (System.nanoTime() - startTime) / 1e9
// System.err.println(s"Dispatcher.threadpool.awaitTermination:
$elapsedTime")
}
{code}
When I measured this, it took 56 seconds.
{code}
Dispatcher.threadpool.awaitTermination: 56.225577089
{code}
> FlumePollingStreamSuite is slow
> -------------------------------
>
> Key: SPARK-14936
> URL: https://issues.apache.org/jira/browse/SPARK-14936
> Project: Spark
> Issue Type: Sub-task
> Components: Build, Tests
> Reporter: Josh Rosen
>
> FlumePollingStreamSuite contains two tests which run for a minute each. This
> seems excessively slow and we should speed it up if possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
