Usually there is another error or log message before FileNotFoundException.
Try to check your logs for something like that.
2015-12-10 10:47 GMT+01:00 kendal :
> I have similar issues... Exception only with very large data.
> And I tried to double the memory or partition as suggested by some goog
I have similar issues... Exception only with very large data.
And I tried to double the memory or partition as suggested by some google
search, but in vain..
any idea?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/FileNotFoundException-in-appcache-shuffl
As Jerry said, this is not related to shuffle file consolidation.
The unique thing about this problem is that it's failing to find a file
while trying to _write_ to it, in append mode. The simplest explanation for
this would be that the file is deleted in between some check for existence
and openi
Hey,
I am having a "similar" issue, did you manage to find a solution yet? Please
check my post below for reference:
http://apache-spark-user-list.1001560.n3.nabble.com/IOError-Errno-2-No-such-file-or-directory-tmp-spark-9e23f17e-2e23-4c26-9621-3cb4d8b832da-tmp3i3xno-td21076.html
Thank you,
Luc
ser/201410.mbox/
-Original Message-
From: Shao, Saisai [saisai.s...@intel.com<mailto:saisai.s...@intel.com>]
Sent: Wednesday, October 29, 2014 01:46 AM Eastern Standard Time
To: Ryan Williams
Cc: user
Subject: RE: FileNotFoundException in appcache shuffle files
Hi Ryan,
This is an issue from so
Hi, Ryan
We have met similar errors and increasing executor memory solved it. Though
I am not sure about the detailed reason, it might be worth a try.
On Wed, Oct 29, 2014 at 1:34 PM, Ryan Williams [via Apache Spark User List]
wrote:
> My job is failing with the following error:
>
> 14/10/29 02
Hi Ryan,
This is an issue from sort-based shuffle, not consolidated hash-based shuffle.
I guess mostly this issue occurs when Spark cluster is in abnormal situation,
maybe long time of GC pause or some others, you can check the system status or
if there’s any other exceptions beside this one.
