Re: FileNotFoundException, while file is actually available

2017-02-06 Thread censj 
If you deploy yarn model，you can used yarn logs -applicationId youApplicationId 
get yarn logs. You can get logs in details。
Then Looking error info,
===
Name: cen sujun
Mobile: 13067874572
Mail: ce...@lotuseed.com

> 在 2017年2月6日，05:33，Evgenii Morozov  写道：
> 
> Hi, 
> 
> I see a lot of exceptions like the following during our machine learning 
> pipeline calculation. Spark version 2.0.2.
> Sometimes it’s just few executors that fails with this message, but the job 
> is successful. 
> 
> I’d appreciate any hint you might have.
> Thank you.
> 
> 2017-02-05 07:56:47.022 [task-result-getter-1] WARN  
> o.a.spark.scheduler.TaskSetManager - Lost task 0.0 in stage 151558.0 (TID 
> 993070, 10.61.12.43):
> java.io.FileNotFoundException: File file:/path/to/file does not exist
>at 
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
>at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
>at 
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
>at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>at 
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:142)
>at 
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
>at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
>at 
> org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:109)
>at 
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
>at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
>at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
>at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
>at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
>at org.apache.spark.scheduler.Task.run(Task.scala:86)
>at 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>at java.lang.Thread.run(Thread.java:745)
> 
> 
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>

FileNotFoundException, while file is actually available

2017-02-05 Thread Evgenii Morozov 
Hi, 

I see a lot of exceptions like the following during our machine learning 
pipeline calculation. Spark version 2.0.2.
Sometimes it’s just few executors that fails with this message, but the job is 
successful. 

I’d appreciate any hint you might have.
Thank you.

2017-02-05 07:56:47.022 [task-result-getter-1] WARN  
o.a.spark.scheduler.TaskSetManager - Lost task 0.0 in stage 151558.0 (TID 
993070, 10.61.12.43):
java.io.FileNotFoundException: File file:/path/to/file does not exist
at 
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
at 
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
at 
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:142)
at 
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at 
org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:109)
at 
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org