[
https://issues.apache.org/jira/browse/SPARK-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Ash updated SPARK-534:
-----------------------------
Description:
SparkEnv (used by SparkContext) is not thread-safe and it causes issues with
scala's Futures and parrallel collections.
For example, this will not work:
{code}
val f = Futures.future({
sc.textFile("hdfs://....")
})
f.apply()
{code}
Workaround for now:
{code}
val f = Futures.future({
SparkEnv.set(sc.env)
sc.textFile("hdfs://....")
})
f.apply()
{code}
was:
SparkEnv (used by SparkContext) is not thread-safe and it causes issues with
scala's Futures and parrallel collections.
For example, this will not work:
val f = Futures.future({
sc.textFile("hdfs://....")
})
f.apply()
Workaround for now:
val f = Futures.future({
SparkEnv.set(sc.env)
sc.textFile("hdfs://....")
})
f.apply()
> Make SparkContext thread-safe
> -----------------------------
>
> Key: SPARK-534
> URL: https://issues.apache.org/jira/browse/SPARK-534
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 0.5.0, 0.5.1, 0.6.0, 0.6.1, 0.7.0, 0.6.2, 0.5.2, 0.7.1,
> 0.7.2, 0.7.3
> Reporter: tjhunter
> Priority: Blocker
>
> SparkEnv (used by SparkContext) is not thread-safe and it causes issues with
> scala's Futures and parrallel collections.
> For example, this will not work:
> {code}
> val f = Futures.future({
> sc.textFile("hdfs://....")
> })
> f.apply()
> {code}
> Workaround for now:
> {code}
> val f = Futures.future({
> SparkEnv.set(sc.env)
> sc.textFile("hdfs://....")
> })
> f.apply()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
