[
https://issues.apache.org/jira/browse/SPARK-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or updated SPARK-10649:
------------------------------
Fix Version/s: 1.5.1
> Streaming jobs unexpectedly inherits job group, job descriptions from context
> starting thread
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-10649
> URL: https://issues.apache.org/jira/browse/SPARK-10649
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.3.1, 1.4.1, 1.5.0
> Reporter: Tathagata Das
> Assignee: Tathagata Das
> Fix For: 1.6.0, 1.5.1
>
>
> The job group, and job descriptions information is passed through thread
> local properties, and get inherited by child threads. In case of spark
> streaming, the streaming jobs inherit these properties from the thread that
> called streamingContext.start(). This may not make sense.
> 1. Job group: This is mainly used for cancelling a group of jobs together. It
> does not make sense to cancel streaming jobs like this, as the effect will be
> unpredictable. And its not a valid usecase any way, to cancel a streaming
> context, call streamingContext.stop()
> 2. Job description: This is used to pass on nice text descriptions for jobs
> to show up in the UI. The job description of the thread that calls
> streamingContext.start() is not useful for all the streaming jobs, as it does
> not make sense for all of the streaming jobs to have the same description,
> and the description may or may not be related to streaming.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
