Tathagata Das created SPARK-2780:
------------------------------------
Summary: Create a StreamingContext.setLocalProperty for setting
local property of jobs launched by streaming
Key: SPARK-2780
URL: https://issues.apache.org/jira/browse/SPARK-2780
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.0.0, 1.1.0
Reporter: Tathagata Das
Priority: Minor
SparkContext.setLocalProperty makes all Spark jobs submitted using
the current thread belong to the set pool. However, in Spark
Streaming, all the jobs are actually launched in the background from a
different thread. So this setting does not work.
Currently, there is a
work around. If you are doing any kind of output operations on
DStreams, like DStream.foreachRDD(), you can set the property inside
that
dstream.foreachRDD(rdd =>
rdd.sparkContext.setLocalProperty(...)
)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
