Guoqiang Li created SPARK-1951:
----------------------------------
Summary: spark on yarn can't start
Key: SPARK-1951
URL: https://issues.apache.org/jira/browse/SPARK-1951
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.0.0
Reporter: Guoqiang Li
Priority: Blocker
{{HADOOP_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit --archives
/input/lbs/recommend/toona/spark/conf toona-assembly.jar 20140521}}throw an
exception:
{code}
Exception in thread "main" java.io.FileNotFoundException: File
file:/input/lbs/recommend/toona/spark/conf does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:402)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at
org.apache.spark.deploy.yarn.ClientBase$class.org$apache$spark$deploy$yarn$ClientBase$$copyRemoteFile(ClientBase.scala:162)
at
org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$4$$anonfun$apply$2.apply(ClientBase.scala:237)
at
org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$4$$anonfun$apply$2.apply(ClientBase.scala:232)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at
org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$4.apply(ClientBase.scala:232)
at
org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$4.apply(ClientBase.scala:230)
at scala.collection.immutable.List.foreach(List.scala:318)
at
org.apache.spark.deploy.yarn.ClientBase$class.prepareLocalResources(ClientBase.scala:230)
at
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:39)
at org.apache.spark.deploy.yarn.Client.runApp(Client.scala:74)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:96)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:186)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
{{HADOOP_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit --archives
hdfs://10dian72:8020/input/lbs/recommend/toona/spark/conf toona-assembly.jar
20140521}} work.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
