[
https://issues.apache.org/jira/browse/SPARK-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Ye updated SPARK-13133:
--------------------------
Description:
When the option --master of spark-submit script is inconsistent with
SparkConf.setMaster in Spark application code, the behavior is difficult to
understand. For example, if the option --master of spark-submit script is
yarn-cluster while there is SparkConf.setMaster("local") in Spark Streaming
application code, the application exit abnormally after about 2 minutes. In
driver's log there is an error whose content is "SparkContext did not
initialize after waiting for 100000 ms. Please check earlier log output for
errors. Failing the application".
When SparkContext is launched, it should be checked whether the option --master
of spark-submit script and SparkConf.setMaster in Spark application code is
different. If they are different, there should be a clear hint in the driver's
log for the developer to troubleshoot.
was:When the option --master of spark-submit script is inconsistent with
SparkConf.setMaster in Spark application code, the behavior is Difficult to
understand. For example, if the option --master of spark-submit script is
yarn-cluster while SparkConf.setMaster("local") in Spark Streaming application
code, the application exit abnormally after about 2 minutes. In driver's log
there is an error whose content is "SparkContext did not initialize after
waiting for 100000 ms. Please check earlier log output for errors. Failing the
application".
> When the option --master of spark-submit script is inconsistent with
> SparkConf.setMaster in Spark appliction code, the behavior is Difficult to
> understand
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-13133
> URL: https://issues.apache.org/jira/browse/SPARK-13133
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.3.0, 1.4.0, 1.5.0, 1.6.0
> Reporter: Li Ye
> Priority: Minor
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> When the option --master of spark-submit script is inconsistent with
> SparkConf.setMaster in Spark application code, the behavior is difficult to
> understand. For example, if the option --master of spark-submit script is
> yarn-cluster while there is SparkConf.setMaster("local") in Spark Streaming
> application code, the application exit abnormally after about 2 minutes. In
> driver's log there is an error whose content is "SparkContext did not
> initialize after waiting for 100000 ms. Please check earlier log output for
> errors. Failing the application".
> When SparkContext is launched, it should be checked whether the option
> --master of spark-submit script and SparkConf.setMaster in Spark application
> code is different. If they are different, there should be a clear hint in the
> driver's log for the developer to troubleshoot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
