Tathagata Das created SPARK-10649:
-------------------------------------
Summary: Streaming jobs unexpectedly inherits job group, job
descriptions from context starting thread
Key: SPARK-10649
URL: https://issues.apache.org/jira/browse/SPARK-10649
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.5.0, 1.4.1, 1.3.1
Reporter: Tathagata Das
Assignee: Tathagata Das
The job group, job descriptions and scheduler pool information is passed
through thread local properties, and get inherited by child threads. In case of
spark streaming, the streaming jobs inherit these properties from the thread
that called streamingContext.start(). This may not make sense.
1. Job group: This is mainly used for cancelling a group of jobs together. It
does not make sense to cancel streaming jobs like this, as the effect will be
unpredictable. And its not a valid usecase any way, to cancel a streaming
context, call streamingContext.stop()
2. Job description: This is used to pass on nice text descriptions for jobs to
show up in the UI. The job description of the thread that calls
streamingContext.start() is not useful for all the streaming jobs, as it does
not make sense for all of the streaming jobs to have the same description, and
the description may or may not be related to streaming.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
