[
https://issues.apache.org/jira/browse/SPARK-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-1980.
------------------------------
Resolution: Invalid
Target Version/s: (was: 1.0.0)
> problems introduced by broadcast
> --------------------------------
>
> Key: SPARK-1980
> URL: https://issues.apache.org/jira/browse/SPARK-1980
> Project: Spark
> Issue Type: Bug
> Reporter: zhoudi
>
> i am writing a word embedding on SPARK. The scale of the model is about 60w *
> 100 * Float.size. Because of the large scale, I have to use broadcast to
> deliver the current model to executors. After each iteration, I would update
> the model, and then broadcast it again. The pseudo-code is as follows,
> for (i <- 0 to 100) {
> broadcast_model <- broadcast(model);
> e_model = xxx.map(Func(broadcast_model)) // handle of broadcast_model to
> Func;
> .reduce(_ + _)
> model <- model + e_model // Update the model
> }
> My problem is that an Error would come out after six iteration. The Error
> Info is as follows,
> ./bin/spark-submit: line 44: 28232 killed
> $SPARK_HOME/bin/spark-class org.apache.spark.deploy.Spark "${ORIG_ARGS[@]}"
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
