[
https://issues.apache.org/jira/browse/SPARK-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14082118#comment-14082118
]
Tathagata Das commented on SPARK-2780:
--------------------------------------
Yeah, this isnt very intuitive. Two possible solutions
1. Explicitly forward the local properties to the job executor thread.
That is, when streamingContext.setLocalProperty is set, all the
properties are explicitly picked up and set in the job executor
thread(s).
2. Leave it as it is, and document this better!
TD
> Create a StreamingContext.setLocalProperty for setting local property of jobs
> launched by streaming
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-2780
> URL: https://issues.apache.org/jira/browse/SPARK-2780
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.0.0, 1.1.0
> Reporter: Tathagata Das
> Priority: Minor
>
> SparkContext.setLocalProperty makes all Spark jobs submitted using
> the current thread belong to the set pool. However, in Spark
> Streaming, all the jobs are actually launched in the background from a
> different thread. So this setting does not work.
> Currently, there is a
> work around. If you are doing any kind of output operations on
> DStreams, like DStream.foreachRDD(), you can set the property inside
> that
> dstream.foreachRDD(rdd =>
> rdd.sparkContext.setLocalProperty(...)
> )
--
This message was sent by Atlassian JIRA
(v6.2#6252)
