Can you look in your worker logs for more detailed stack-trace? If its about winutils.exe you can look at these links to get it resolved.
- http://qnalist.com/questions/4994960/run-spark-unit-test-on-windows-7 - https://issues.apache.org/jira/browse/SPARK-2356 Thanks Best Regards On Mon, Jun 8, 2015 at 9:01 AM, Dong Lei <dong...@microsoft.com> wrote: > Hi spark users: > > > > After I submitted a SparkPi job to spark, the driver crashed at the end of > the job with the following log: > > > > WARN EventLoggingListener: Event log dir > file:/d:/data/SparkWorker/work/driver-20150607200517-0002/logs/event does > not exists, will newly create one. > > Exception in thread "main" java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:59) > > at > org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) > > Caused by: java.lang.NullPointerException > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010) > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:445) > > at org.apache.hadoop.util.Shell.run(Shell.java:418) > > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650) > > at org.apache.hadoop.util.Shell.execCommand(Shell.java:739) > > at org.apache.hadoop.util.Shell.execCommand(Shell.java:722) > > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:633) > > at > org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:467) > > at > org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:135) > > at > org.apache.spark.SparkContext.<init>(SparkContext.scala:401) > > at > org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28) > > at org.apache.spark.examples.SparkPi.main(SparkPi.scala) > > > > From the log, I can see that the driver has added jars from HDFS, > connected to master, scheduled executors and all the executors were > running. And then this error occurred. > > > > The command I use to submit job(I’m running spark 1.3.1 with *standalone* > mode on windows): > > ./bin/spark-submit \ > > --class org.apache.spark.examples.SparkPi \ > > --master spark://localhost:7077 \ > > --deploy-mode cluster > > Hdfs://localhost:443/spark-examples-1.3.1-hadoop2.4.0.jar \ > > 1000 > > > > > > Any ideas about the error? > > I’ve found a similar error in JIRA > https://issues.apache.org/jira/browse/SPARK-1407 but It only occurred at > FileLogger when using yarn and eventlog set to HDFS. In my case, I use > standalone mode and event log set to local, and my error is caused by > Hadoop.util.Shell.runCommand. > > > > > > Best Regards > > Dong Lei >
