Thanks a lot for your suggestion. I had modified HADOOP_CONF_DIR in spark-env.sh so that core-site.xml is under HADOOP_CONF_DIR. i can able to see the logs like that you had shown above. Now i can able to run for 3 minutes and store results between every minutes. After sometimes, there is an exception. How to fix this exception ? and Can you please explain where its going wrong ?
*Log Link : http://pastebin.com/xL9jaRUa <http://pastebin.com/xL9jaRUa> * *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Wed, Aug 19, 2015 at 1:54 PM, Jeff Zhang <zjf...@gmail.com> wrote: > HADOOP_CONF_DIR is the environment variable point to the hadoop conf > directory. Not sure how CDH organize that, make sure core-site.xml is > under HADOOP_CONF_DIR. > > On Wed, Aug 19, 2015 at 4:06 PM, Ramkumar V <ramkumar.c...@gmail.com> > wrote: > >> We are using Cloudera-5.3.1. since it is one of the earlier version of >> CDH, it doesnt supports the latest version of spark. So i installed >> spark-1.4.1 separately in my machine. I couldnt able to do spark-submit in >> cluster mode. How to core-site.xml under classpath ? it will be very >> helpful if you could explain in detail to solve this issue. >> >> *Thanks*, >> <https://in.linkedin.com/in/ramkumarcs31> >> >> >> On Fri, Aug 14, 2015 at 8:25 AM, Jeff Zhang <zjf...@gmail.com> wrote: >> >>> >>> 1. 15/08/12 13:24:49 INFO Client: Source and destination file >>> systems are the same. Not copying >>> >>> file:/home/hdfs/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.5.0-cdh5.3.5.jar >>> 2. 15/08/12 13:24:49 INFO Client: Source and destination file >>> systems are the same. Not copying >>> >>> file:/home/hdfs/spark-1.4.1/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-1.4.1.jar >>> 3. 15/08/12 13:24:49 INFO Client: Source and destination file >>> systems are the same. Not copying >>> file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip >>> 4. 15/08/12 13:24:49 INFO Client: Source and destination file >>> systems are the same. Not copying >>> file:/home/hdfs/spark-1.4.1/python/lib/py4j-0.8.2.1-src.zip >>> 5. 15/08/12 13:24:49 INFO Client: Source and destination file >>> systems are the same. Not copying >>> file:/home/hdfs/spark-1.4.1/examples/src/main/python/streaming/kyt.py >>> 6. >>> >>> >>> 1. diagnostics: Application application_1437639737006_3808 failed 2 >>> times due to AM Container for appattempt_1437639737006_3808_000002 exited >>> with exitCode: -1000 due to: File >>> file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip does not exist >>> 2. .Failing this attempt.. Failing the application. >>> >>> >>> >>> The machine you run spark is the client machine, while the yarn AM is >>> running on another machine. And the yarn AM complains that the files are >>> not found as your logs shown. >>> From the logs, its seems that these files are not copied to the HDFS as >>> local resources. I doubt that you didn't put core-site.xml under your >>> classpath, so that spark can not detect your remote file system and won't >>> copy the files to hdfs as local resources. Usually in yarn-cluster mode, >>> you should be able to see the logs like following. >>> >>> > 15/08/14 10:48:49 INFO yarn.Client: Preparing resources for our AM >>> container >>> > 15/08/14 10:48:49 INFO yarn.Client: Uploading resource >>> file:/Users/abc/github/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar >>> -> hdfs:// >>> 0.0.0.0:9000/user/abc/.sparkStaging/application_1439432662178_0019/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar >>> > 15/08/14 10:48:50 INFO yarn.Client: Uploading resource >>> file:/Users/abc/github/spark/spark.py -> hdfs:// >>> 0.0.0.0:9000/user/abc/.sparkStaging/application_1439432662178_0019/spark.py >>> > 15/08/14 10:48:50 INFO yarn.Client: Uploading resource >>> file:/Users/abc/github/spark/python/lib/pyspark.zip -> hdfs:// >>> 0.0.0.0:9000/user/abc/.sparkStaging/application_1439432662178_0019/pyspark.zip >>> >>> On Thu, Aug 13, 2015 at 2:50 PM, Ramkumar V <ramkumar.c...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I have a cluster of 1 master and 2 slaves. I'm running a spark >>>> streaming in master and I want to utilize all nodes in my cluster. i had >>>> specified some parameters like driver memory and executor memory in my >>>> code. when i give --deploy-mode cluster --master yarn-cluster in my >>>> spark-submit, it gives the following error. >>>> >>>> Log link : *http://pastebin.com/kfyVWDGR >>>> <http://pastebin.com/kfyVWDGR>* >>>> >>>> How to fix this issue ? Please help me if i'm doing wrong. >>>> >>>> >>>> *Thanks*, >>>> Ramkumar V >>>> >>>> >>> >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >>> >> >> > > > -- > Best Regards > > Jeff Zhang >
