[
https://issues.apache.org/jira/browse/SPARK-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15537498#comment-15537498
]
Nick Orka commented on SPARK-17707:
-----------------------------------
I noticed this a couple of days ago. We've just switched to Spark 2.0. I'm on
Mac OS usitng start-all to spin up standalone spark. While I was developing
just spark part on scala, I was using Scala Console for testing and debugging
and it looks fine there because it holds all the variables during one session.
We are using Luigi workflow management for our data pipeline. This is where
I've noticed that the same Spark task has unstable behavior. Sometime it passes
through sometime it stucks. Luigi opens separate thread to run spark-submit as
a shell command. I intercepted the exact command line and started it just in
shell. And I've noticed that if you open the running application details in Web
UI the application opens socket for piping the details to web listener as a
separate thread. I can see the very last statement execution of my app
(println("I'm done")) but the shell is still waiting. Ctrl-C is the only way to
finish the process. If I don't open app details in Web UI, it prints "I'm done"
closes all accumulators' and shuffles' processes and returns to shell without
any problem.
> Web UI prevents spark-submit application to be finished
> -------------------------------------------------------
>
> Key: SPARK-17707
> URL: https://issues.apache.org/jira/browse/SPARK-17707
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Nick Orka
>
> Here are re-production steps:
> 1. create any scala spark application which will work long enough to open the
> application details in Web UI
> 2. run spark-submit command for standalone cluster, like: --master
> spark:\\localhost:7077
> 3. open running application details in Web UI, like: localhost:4040
> 4. spark-submit will never finish, you will have to kill the process
> Cause: The application creates a thread with infinite loop for web UI
> communication and never stops it. The application is waiting for the thread
> to be finished instead, even if you close the web page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
