Find the log from rm below, before FNFE there is no earlier errors in driver log,
16/07/04 00:27:56 INFO mapreduce.TableInputFormatBase: Input split length: 0 bytes. 16/07/04 00:27:56 INFO executor.Executor: Executor is trying to kill task 56.0 in stage 2437.0 (TID 328047) 16/07/04 00:27:56 INFO executor.Executor: Executor killed task 266.0 in stage 2433.0 (TID 328005) 16/07/04 00:27:56 INFO executor.Executor: Executor killed task 206.0 in stage 2433.0 (TID 327977) 16/07/04 00:27:56 INFO executor.Executor: Executor killed task 318.0 in stage 2433.0 (TID 328006) 16/07/04 00:27:57 INFO executor.Executor: Executor killed task 56.0 in stage 2437.0 (TID 328047) 16/07/04 00:27:57 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown 16/07/04 00:27:57 INFO storage.MemoryStore: MemoryStore cleared 16/07/04 00:27:57 INFO storage.BlockManager: BlockManager stopped 16/07/04 00:27:57 WARN executor.CoarseGrainedExecutorBackend: An unknown ( driver.domain.com:56055) driver disconnected. 16/07/04 00:27:57 ERROR executor.CoarseGrainedExecutorBackend: Driver xx:xx:xx:xx:56055 disassociated! Shutting down. 16/07/04 00:27:57 INFO util.ShutdownHookManager: Shutdown hook called 16/07/04 00:27:57 INFO util.ShutdownHookManager: Deleting directory /opt/mapr/tmp/hadoop-tmp/hadoop-mapr/nm-local-dir/usercache/user/appcache/application_1467474162580_29353/spark-9c0bfccc-74c3-4541-a2fd-19101e47b49a End of LogType:stderr On Mon, Jul 4, 2016 at 4:21 PM, Jacek Laskowski <ja...@japila.pl> wrote: > Can you share some stats from Web UI just before the failure? Any earlier > errors before FNFE? > > Jacek > On 4 Jul 2016 12:34 p.m., "kishore kumar" <akishore...@gmail.com> wrote: > >> @jacek: It is running on yarn-client mode, our code don't support running >> in yarn-cluster mode and the job is running for around an hour and giving >> the exception. >> >> @karhi: yarn application status is successful, resourcemanager logs did >> not give any failure info except >> 16/07/04 00:27:57 INFO executor.CoarseGrainedExecutorBackend: Driver >> commanded a shutdown >> 16/07/04 00:27:57 INFO storage.MemoryStore: MemoryStore cleared >> 16/07/04 00:27:57 INFO storage.BlockManager: BlockManager stopped >> 16/07/04 00:27:57 WARN executor.CoarseGrainedExecutorBackend: An unknown ( >> slave1.domain.com:56055) driver disconnected. >> 16/07/04 00:27:57 ERROR executor.CoarseGrainedExecutorBackend: Driver >> 173.36.88.26:56055 disassociated! Shutting down. >> 16/07/04 00:27:57 INFO util.ShutdownHookManager: Shutdown hook called >> 16/07/04 00:27:57 INFO util.ShutdownHookManager: Deleting directory >> /opt/mapr/tmp/hadoop-tmp/hadoop-mapr/nm-local-dir/usercache/user/appcache/application_1467474162580_29353/spark-9c0bfccc-74c3-4541-a2fd-19101e47b49a >> End of LogType:stderr >> >> >> On Mon, Jul 4, 2016 at 3:20 PM, Jacek Laskowski <ja...@japila.pl> wrote: >> >>> Hi, >>> >>> You seem to be using yarn. Is this cluster or client deploy mode? Have >>> you seen any other exceptions before? How long did the application run >>> before the exception? >>> >>> Pozdrawiam, >>> Jacek Laskowski >>> ---- >>> https://medium.com/@jaceklaskowski/ >>> Mastering Apache Spark http://bit.ly/mastering-apache-spark >>> Follow me at https://twitter.com/jaceklaskowski >>> >>> >>> On Mon, Jul 4, 2016 at 10:57 AM, kishore kumar <akishore...@gmail.com> >>> wrote: >>> > We've upgraded spark version from 1.2 to 1.6 still the same problem, >>> > >>> > Exception in thread "main" org.apache.spark.SparkException: Job >>> aborted due >>> > to stage failure: Task 286 in stage >>> > 2397.0 failed 4 times, most recent failure: Lost task 286.3 in stage >>> 2397.0 >>> > (TID 314416, salve-06.domain.com): java.io.FileNotFoundException: >>> > /opt/mapr/tmp/h >>> > >>> adoop-tmp/hadoop-mapr/nm-local-dir/usercache/user1/appcache/application_1467474162580_29353/blockmgr-bd075392-19c2-4cb8-8033-0fe54d683c8f/12/shuffle_530_286_0.inde >>> > x.c374502a-4cf2-4052-abcf-42977f1623d0 (No such file or directory) >>> > >>> > Kindly help me to get rid from this. >>> > >>> > On Sun, Jun 5, 2016 at 9:43 AM, kishore kumar <akishore...@gmail.com> >>> wrote: >>> >> >>> >> Hi, >>> >> >>> >> Could anyone help me about this error ? why this error comes ? >>> >> >>> >> Thanks, >>> >> KishoreKuamr. >>> >> >>> >> On Fri, Jun 3, 2016 at 9:12 PM, kishore kumar <akishore...@gmail.com> >>> >> wrote: >>> >>> >>> >>> Hi Jeff Zhang, >>> >>> >>> >>> Thanks for response, could you explain me why this error occurs ? >>> >>> >>> >>> On Fri, Jun 3, 2016 at 6:15 PM, Jeff Zhang <zjf...@gmail.com> wrote: >>> >>>> >>> >>>> One quick solution is to use spark 1.6.1. >>> >>>> >>> >>>> On Fri, Jun 3, 2016 at 8:35 PM, kishore kumar < >>> akishore...@gmail.com> >>> >>>> wrote: >>> >>>>> >>> >>>>> Could anyone help me on this issue ? >>> >>>>> >>> >>>>> On Tue, May 31, 2016 at 8:00 PM, kishore kumar < >>> akishore...@gmail.com> >>> >>>>> wrote: >>> >>>>>> >>> >>>>>> Hi, >>> >>>>>> >>> >>>>>> We installed spark1.2.1 in single node, running a job in >>> yarn-client >>> >>>>>> mode on yarn which loads data into hbase and elasticsearch, >>> >>>>>> >>> >>>>>> the error which we are encountering is >>> >>>>>> Exception in thread "main" org.apache.spark.SparkException: Job >>> >>>>>> aborted due to stage failure: Task 38 in stage 26800.0 failed 4 >>> times, most >>> >>>>>> recent failure: Lost task 38.3 in stage 26800.0 (TID 4990082, >>> >>>>>> hdprd-c01-r04-03): java.io.FileNotFoundException: >>> >>>>>> >>> /opt/mapr/tmp/hadoop-tmp/hadoop-mapr/nm-local-dir/usercache/sparkuser/appcache/application_1463194314221_211370/spark-3cc37dc7-fa3c-4b98-aa60-0acdfc79c725/28/shuffle_8553_38_0.index >>> >>>>>> (No such file or directory) >>> >>>>>> >>> >>>>>> any idea about this error ? >>> >>>>>> -- >>> >>>>>> Thanks, >>> >>>>>> Kishore. >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> Thanks, >>> >>>>> Kishore. >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> Best Regards >>> >>>> >>> >>>> Jeff Zhang >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Thanks, >>> >>> Kishore. >>> >> >>> >> >>> >> >>> >> >>> >> -- >>> >> Thanks, >>> >> Kishore. >>> > >>> > >>> > >>> > >>> > -- >>> > Thanks, >>> > Kishore. >>> >> >> >> >> -- >> Thanks, >> Kishore. >> > -- Thanks, Kishore.
