[jira] [Commented] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319018#comment-14319018
]
Apache Spark commented on SPARK-4267:
-
User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/4575
Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
--
Key: SPARK-4267
URL: https://issues.apache.org/jira/browse/SPARK-4267
Project: Spark
Issue Type: Bug
Components: YARN
Reporter: Tsuyoshi OZAWA
Assignee: Sean Owen
Priority: Blocker
Fix For: 1.3.0
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5
uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
{code}
./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn
-Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
{code}
Then Spark on YARN fails to launch jobs with NPE.
{code}
$ bin/spark-shell --master yarn-client
scala sc.textFile(hdfs:///user/ozawa/wordcountInput20G).flatMap(line
= line.split( )).map(word = (word, 1)).persist().reduceByKey((a, b) = a
+ b, 16).saveAsTextFile(hdfs:///user/ozawa/sparkWordcountOutNew2);
java.lang.NullPointerException
at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)
at
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.init(console:13)
at $iwC$$iwC$$iwC.init(console:18)
at $iwC$$iwC.init(console:20)
at $iwC.init(console:22)
at init(console:24)
at .init(console:28)
at .clinit(console)
at .init(console:7)
at .clinit(console)
at $print(console)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
[jira] [Commented] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310753#comment-14310753
]
Apache Spark commented on SPARK-4267:
-
User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/4452
Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
--
Key: SPARK-4267
URL: https://issues.apache.org/jira/browse/SPARK-4267
Project: Spark
Issue Type: Bug
Components: YARN
Reporter: Tsuyoshi OZAWA
Priority: Blocker
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5
uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
{code}
./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn
-Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
{code}
Then Spark on YARN fails to launch jobs with NPE.
{code}
$ bin/spark-shell --master yarn-client
scala sc.textFile(hdfs:///user/ozawa/wordcountInput20G).flatMap(line
= line.split( )).map(word = (word, 1)).persist().reduceByKey((a, b) = a
+ b, 16).saveAsTextFile(hdfs:///user/ozawa/sparkWordcountOutNew2);
java.lang.NullPointerException
at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)
at
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.init(console:13)
at $iwC$$iwC$$iwC.init(console:18)
at $iwC$$iwC.init(console:20)
at $iwC.init(console:22)
at init(console:24)
at .init(console:28)
at .clinit(console)
at .init(console:7)
at .clinit(console)
at $print(console)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
[jira] [Commented] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290744#comment-14290744
]
Apache Spark commented on SPARK-4267:
-
User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/4188
Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
--
Key: SPARK-4267
URL: https://issues.apache.org/jira/browse/SPARK-4267
Project: Spark
Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5
uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
{code}
./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn
-Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
{code}
Then Spark on YARN fails to launch jobs with NPE.
{code}
$ bin/spark-shell --master yarn-client
scala sc.textFile(hdfs:///user/ozawa/wordcountInput20G).flatMap(line
= line.split( )).map(word = (word, 1)).persist().reduceByKey((a, b) = a
+ b, 16).saveAsTextFile(hdfs:///user/ozawa/sparkWordcountOutNew2);
java.lang.NullPointerException
at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)
at
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.init(console:13)
at $iwC$$iwC$$iwC.init(console:18)
at $iwC$$iwC.init(console:20)
at $iwC.init(console:22)
at init(console:24)
at .init(console:28)
at .clinit(console)
at .init(console:7)
at .clinit(console)
at $print(console)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
[jira] [Commented] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290742#comment-14290742
]
Sean Owen commented on SPARK-4267:
--
The warning is from YARN, I believe, rather than Spark. Yeah maybe should be an
error.
Your info however points to the problem; I'm sure it's {{-Dnumbers=one two
three}}. {{Utils.splitCommandString}} strips quotes as it parses them, so will
turn it into {{-Dnumbers=one two three}} so the command is becoming {{java
-Dnumbers=one two three ...}} and this isn't valid.
I suggest that {{Utils.splitCommandString}} not strip the quotes that it
parses, so that the reconstructed command line is exactly like the original.
It's just splitting, not interpreting the command. This also seems less
surprising. PR coming to demonstrate.
Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
--
Key: SPARK-4267
URL: https://issues.apache.org/jira/browse/SPARK-4267
Project: Spark
Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5
uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
{code}
./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn
-Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
{code}
Then Spark on YARN fails to launch jobs with NPE.
{code}
$ bin/spark-shell --master yarn-client
scala sc.textFile(hdfs:///user/ozawa/wordcountInput20G).flatMap(line
= line.split( )).map(word = (word, 1)).persist().reduceByKey((a, b) = a
+ b, 16).saveAsTextFile(hdfs:///user/ozawa/sparkWordcountOutNew2);
java.lang.NullPointerException
at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)
at
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.init(console:13)
at $iwC$$iwC$$iwC.init(console:18)
at $iwC$$iwC.init(console:20)
at $iwC.init(console:22)
at init(console:24)
at .init(console:28)
at .clinit(console)
at .init(console:7)
at .clinit(console)
at $print(console)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
at
[jira] [Commented] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208595#comment-14208595
]
Kousuke Saruta commented on SPARK-4267:
---
Hi [~ozawa], On my YARN-2.5.1(JDK 1.7.0_60) cluster, Spark Shell works well.
I built with following command.
{code}
sbt/sbt -Dhadoop.version=2.5.1 -Pyarn assembly
{code}
And launched Spark Shell with following command.
{code}
bin/spark-shell --master yarn --deploy-mode client --executor-cores 1
--driver-memory 512M --executor-memory 512M --num-executors 1
{code}
And then, I ran job with following script.
{code}
sc.textFile(hdfs:///user/kou/LICENSE.txt).flatMap(line = line.split(
)).map(word = (word, 1)).persist().reduceByKey((a, b) = a + b,
16).saveAsTextFile(hdfs:///user/kou/LICENSE.txt.count)
{code}
So I think the problem is not caused by the version of Hadoop.
One possible case is that SparkContext#stop is called between instantiating
SparkContext and running job accidentally.
Did you see any ERROR log on the shell?
Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
--
Key: SPARK-4267
URL: https://issues.apache.org/jira/browse/SPARK-4267
Project: Spark
Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5
uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
{code}
./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn
-Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
{code}
Then Spark on YARN fails to launch jobs with NPE.
{code}
$ bin/spark-shell --master yarn-client
scala sc.textFile(hdfs:///user/ozawa/wordcountInput20G).flatMap(line
= line.split( )).map(word = (word, 1)).persist().reduceByKey((a, b) = a
+ b, 16).saveAsTextFile(hdfs:///user/ozawa/sparkWordcountOutNew2);
java.lang.NullPointerException
at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)
at
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.init(console:13)
at $iwC$$iwC$$iwC.init(console:18)
at $iwC$$iwC.init(console:20)
at $iwC.init(console:22)
at init(console:24)
at .init(console:28)
at .clinit(console)
at .init(console:7)
at .clinit(console)
at $print(console)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
[jira] [Commented] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209183#comment-14209183
]
Matthew Daniel commented on SPARK-4267:
---
Apologies, I don't know if we want log verbiage inline or as an attachment.
I experienced this NPE on an EMR cluster, AMI 3.3.0 which is Amazon Hadoop
2.4.0 against a {{make-distribution.sh}} version with {{-Pyarn}} and
{{-Phadoop-2.2}} with {{-Dhadoop.version=2.2.0}}. I built it against 2.2
because some of our jobs run on 2.2, and I thought 2.4 would be backwards
compatible.
I will try building as you said, using {{sbt assembly}}, but I wanted to reply
to your comment that yes, I do see an {{ERROR}} line but it isn't helpful to
me, so I hope it's meaningful to others.
{noformat}
14/11/13 02:58:23 INFO cluster.YarnClientSchedulerBackend: Application report
from ASM:
appMasterRpcPort: -1
appStartTime: 1415847498993
yarnAppState: ACCEPTED
14/11/13 02:58:23 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter.
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,
PROXY_HOST=10.166.39.198,PROXY_URI_BASE=http://10.166.39.198:9046/proxy/application_1415840940647_0001,
/proxy/application_1415840940647_0001
14/11/13 02:58:23 INFO ui.JettyUtils: Adding filter:
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
14/11/13 02:58:24 INFO cluster.YarnClientSchedulerBackend: Application report
from ASM:
appMasterRpcPort: 0
appStartTime: 1415847498993
yarnAppState: RUNNING
14/11/13 02:58:29 ERROR cluster.YarnClientSchedulerBackend: Yarn application
already ended: FINISHED
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/metrics/json,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/stages/stage/kill,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/static,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/executors/json,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/executors,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/environment/json,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/environment,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/storage/rdd/json,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/storage/rdd,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/storage/json,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/storage,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/stages/pool/json,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/stages/pool,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/stages/stage/json,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/stages/stage,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/stages/json,null}
14/11/13 02:58:29 INFO handler.ContextHandler: stopped
o.e.j.s.ServletContextHandler{/stages,null}
14/11/13 02:58:29 INFO ui.SparkUI: Stopped Spark web UI at
http://ip-10-166-39-198.ec2.internal:4040
14/11/13 02:58:29 INFO scheduler.DAGScheduler: Stopping DAGScheduler
14/11/13 02:58:29 INFO cluster.YarnClientSchedulerBackend: Shutting down all
executors
14/11/13 02:58:29 INFO cluster.YarnClientSchedulerBackend: Asking each executor
to shut down
14/11/13 02:58:29 INFO cluster.YarnClientSchedulerBackend: Stopped
14/11/13 02:58:30 INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor
stopped!
14/11/13 02:58:30 INFO network.ConnectionManager: Selector thread was
interrupted!
14/11/13 02:58:30 INFO network.ConnectionManager: ConnectionManager stopped
14/11/13 02:58:30 INFO storage.MemoryStore: MemoryStore cleared
14/11/13 02:58:30 INFO storage.BlockManager: BlockManager stopped
14/11/13 02:58:30 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
14/11/13 02:58:30 INFO spark.SparkContext: Successfully stopped SparkContext
14/11/13 02:58:30 INFO remote.RemoteActorRefProvider$RemotingTerminator:
Shutting down remote daemon.
14/11/13 02:58:30 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote
daemon shut down; proceeding with flushing remote transports.
14/11/13 02:58:30 INFO Remoting: Remoting shut down
14/11/13 02:58:30 INFO remote.RemoteActorRefProvider$RemotingTerminator:
Remoting shut down.
14/11/13 02:58:47 INFO
[jira] [Commented] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209239#comment-14209239
]
Kousuke Saruta commented on SPARK-4267:
---
Hi [~bugzi...@mdaniel.scdi.com].
The NPE is caused by SparkContext stopped because Application finished
accidentally.
I don't know why your application finished before running job for now.
Can you see some ERROR message on the logs of ApplicationMaster or
ResourceManager?
Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
--
Key: SPARK-4267
URL: https://issues.apache.org/jira/browse/SPARK-4267
Project: Spark
Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5
uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
{code}
./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn
-Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
{code}
Then Spark on YARN fails to launch jobs with NPE.
{code}
$ bin/spark-shell --master yarn-client
scala sc.textFile(hdfs:///user/ozawa/wordcountInput20G).flatMap(line
= line.split( )).map(word = (word, 1)).persist().reduceByKey((a, b) = a
+ b, 16).saveAsTextFile(hdfs:///user/ozawa/sparkWordcountOutNew2);
java.lang.NullPointerException
at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)
at
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.init(console:13)
at $iwC$$iwC$$iwC.init(console:18)
at $iwC$$iwC.init(console:20)
at $iwC.init(console:22)
at init(console:24)
at .init(console:28)
at .clinit(console)
at .init(console:7)
at .clinit(console)
at $print(console)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
at
[jira] [Commented] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209325#comment-14209325
]
Matthew Daniel commented on SPARK-4267:
---
My searches for {{ERROR}} didn't yield anything, but I found the text at the
bottom of this comment in a file
{{yarn-hadoop-nodemanager-ip-10-171-57-176.ec2.internal.log.2014-11-13-03}} on
one of the yarn slaves which sheds light on the situation.
I reverted {{spark-defaults.conf}} to just the bare bones:
{noformat}
spark.master yarn
spark.driver.memory 1G
spark.executor.memory 5G
{noformat}
and then the {{SparkContext}} was initialized as expected. To be honest, I
perhaps should not have uncommented the {{# spark.executor.extraJavaOptions
-XX:+PrintGCDetails -Dkey=value -Dnumbers=one two three}} but I wanted to see
what it did. Now I know what it does: bring down yarn containers. :-)
It's too bad such a grave *error* is reported at _warn_ level, and I hope in
the master branch that NPE has been cleaned up because those exceptions are not
helpful at all.
Nevertheless, I hope this helps the original submitter track down their
problem, too.
{noformat}
2014-11-13 03:57:14,085 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor
(ContainersLauncher #3): Exception from container-launch with container ID:
container_1415840940647_0002_01_02 and exit code: 1
org.apache.hadoop.util.Shell$ExitCodeException: Usage: java [-options] class
[args...]
(to execute a class)
or java [-options] -jar jarfile [args...]
(to execute a jar file)
where options include:
{noformat}
Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
--
Key: SPARK-4267
URL: https://issues.apache.org/jira/browse/SPARK-4267
Project: Spark
Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5
uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
{code}
./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn
-Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
{code}
Then Spark on YARN fails to launch jobs with NPE.
{code}
$ bin/spark-shell --master yarn-client
scala sc.textFile(hdfs:///user/ozawa/wordcountInput20G).flatMap(line
= line.split( )).map(word = (word, 1)).persist().reduceByKey((a, b) = a
+ b, 16).saveAsTextFile(hdfs:///user/ozawa/sparkWordcountOutNew2);
java.lang.NullPointerException
at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)
at
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.init(console:13)
at $iwC$$iwC$$iwC.init(console:18)
at $iwC$$iwC.init(console:20)
at $iwC.init(console:22)
at init(console:24)
at .init(console:28)
at .clinit(console)
at .init(console:7)
at .clinit(console)
at $print(console)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[jira] [Commented] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201959#comment-14201959
]
Tsuyoshi OZAWA commented on SPARK-4267:
---
[~sandyr] [~pwendell] do you have any workarounds to deal with this problem?
Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
--
Key: SPARK-4267
URL: https://issues.apache.org/jira/browse/SPARK-4267
Project: Spark
Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5
uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
{code}
./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn
-Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
{code}
Then Spark on YARN fails to launch jobs with NPE.
{code}
$ bin/spark-shell --master yarn-client
scala sc.textFile(hdfs:///user/ozawa/wordcountInput20G).flatMap(line
= line.split( )).map(word = (word, 1)).persist().reduceByKey((a, b) = a
+ b, 16).saveAsTextFile(hdfs:///user/ozawa/sparkWordcountOutNew2);
java.lang.NullPointerException
at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)
at
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.init(console:13)
at $iwC$$iwC$$iwC.init(console:18)
at $iwC$$iwC.init(console:20)
at $iwC.init(console:22)
at init(console:24)
at .init(console:28)
at .clinit(console)
at .init(console:7)
at .clinit(console)
at $print(console)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
[jira] [Commented] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14202319#comment-14202319
]
Sandy Ryza commented on SPARK-4267:
---
Strange. Checked in the code and it seems like this must mean the
taskScheduler is null. Did you see any errors farther up in the shell before
this happened? Does it work in local mode?
Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
--
Key: SPARK-4267
URL: https://issues.apache.org/jira/browse/SPARK-4267
Project: Spark
Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5
uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
{code}
./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn
-Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
{code}
Then Spark on YARN fails to launch jobs with NPE.
{code}
$ bin/spark-shell --master yarn-client
scala sc.textFile(hdfs:///user/ozawa/wordcountInput20G).flatMap(line
= line.split( )).map(word = (word, 1)).persist().reduceByKey((a, b) = a
+ b, 16).saveAsTextFile(hdfs:///user/ozawa/sparkWordcountOutNew2);
java.lang.NullPointerException
at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)
at
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.init(console:13)
at $iwC$$iwC$$iwC.init(console:18)
at $iwC$$iwC.init(console:20)
at $iwC.init(console:22)
at init(console:24)
at .init(console:28)
at .clinit(console)
at .init(console:7)
at .clinit(console)
at $print(console)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
