Dhruv sharma created SPARK-23006:
------------------------------------
Summary: S3 sync issue
Key: SPARK-23006
URL: https://issues.apache.org/jira/browse/SPARK-23006
Project: Spark
Issue Type: Brainstorming
Components: Spark Core
Affects Versions: 2.1.1
Environment: AWS EMR 5.7
Reporter: Dhruv sharma
Priority: Blocker
Fix For: 2.1.1
I am using S3 to read and write for our various jobs.
And facing some read and write inconsistencies, which are leading to the job
failures.
Details are as below:
- WRITE ERRORS :
while writing to s3 I have seen the exception "File already exists".
My understanding is while writing to s3 if some executors die then the same
task is delegated to another executor, which then retries the write. But file
being partially written gives us the above exception.
Tried tuning the time multiplier so that WRITE tasks are not killed. It's
working for now but when it might stop working, not sure.
- READ AFTER WRITE ERRORS
One of the jobs is dealing with S3 using overwrite mode.
It first deletes the files from the folder and then re-write to it.
S3 doesn't guarantee immediate deletion.
The time I get the list of objects using spark.read.json("bucket/key1/*") , It
generates another exception "FILE NOT FOUND". Because anyway the file is
deleted.
Also, have faced the issue when reading after new write as list operation is
not strongly consistent there are errors like "FILE NOT FOUND or KEY NOT FOUND".
Is there a way we can tune our spark configuration, to remove such exceptions.
Further what should be kept in mind while interacting with S3 for read and
write.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
