[
https://issues.apache.org/jira/browse/SPARK-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhoudi updated SPARK-1980:
--------------------------
Description:
i am writing a word embedding on SPARK. The scale of the model is about 60w *
100 * Float.size. Because of the large scale, I have to use broadcast to
deliver the current model to executors. After each iteration, I would update
the model, and then broadcast it again. The pseudo-code is as follows,
for (i <- 0 to 100) {
broadcast_model <- broadcast(model);
e_model = xxx.map(Func(broadcast_model)) // handle of broadcast_model to
Func;
.reduce(_ + _)
model <- model + e_model // Update the model
}
My problem is that an Error would come out after six iteration. The Error Info
is as follows,
./bin/spark-submit: line 44: 28232 killed
$SPARK_HOME/bin/spark-class org.apache.spark.deploy.Spark "${ORIG_ARGS[@]}"
was:
i am writing a word embedding on SPARK. The scale of the model is about 60w *
100 * Float.size. Because of the large scale, I have to use broadcast to
deliver the current model to executors. After each iteration, I would update
the model, and then broadcast it again. The pseudo-code is as follows,
for (i <- 0 to 100) {
broadcast_model <- broadcast(model)
e_model = xxx.map(Func(broadcast_model)) // handle of broadcast_model to
Func
.reduce(_ + _)
model <- model + e_model
}
My problem is that an Error would come out after six iteration. The Error Info
is as follows,
./bin/spark-submit: line 44: 28232 killed
$SPARK_HOME/bin/spark-class org.apache.spark.deploy.Spark "${ORIG_ARGS[@]}"
> problems introduced by broadcast
> --------------------------------
>
> Key: SPARK-1980
> URL: https://issues.apache.org/jira/browse/SPARK-1980
> Project: Spark
> Issue Type: Bug
> Reporter: zhoudi
>
> i am writing a word embedding on SPARK. The scale of the model is about 60w *
> 100 * Float.size. Because of the large scale, I have to use broadcast to
> deliver the current model to executors. After each iteration, I would update
> the model, and then broadcast it again. The pseudo-code is as follows,
> for (i <- 0 to 100) {
> broadcast_model <- broadcast(model);
> e_model = xxx.map(Func(broadcast_model)) // handle of broadcast_model to
> Func;
> .reduce(_ + _)
> model <- model + e_model // Update the model
> }
> My problem is that an Error would come out after six iteration. The Error
> Info is as follows,
> ./bin/spark-submit: line 44: 28232 killed
> $SPARK_HOME/bin/spark-class org.apache.spark.deploy.Spark "${ORIG_ARGS[@]}"
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
