Most likely directory write permission not permission. The app user doesn't have permission to write files to that directory.
> Sent: Friday, July 17, 2020 at 6:03 PM > From: "Nagendra Darla" <dvv.nagen...@gmail.com> > To: "Hulio andres" <hulioand...@usa.com> > Cc: user@spark.apache.org > Subject: Re: File not found exceptions on S3 while running spark jobs > > Hi, > > Thanks I know about FileNotFound Exception. > > This error is with S3 buckets which has a delay in showing newly created > files. These files eventually shows up after some time. > > These errors are coming up while running a parquet table into Delta table. > > My question is more around avoiding this error with spark jobs which create > / updates / deletes lots of files on S3 buckets. > > On Thu, Jul 16, 2020 at 10:28 PM Hulio andres <hulioand...@usa.com> wrote: > > > > > https://examples.javacodegeeks.com/java-io-filenotfoundexception-how-to-solve-file-not-found-exception/ > > > > Are you a programmer ? > > > > Regards, > > > > Hulio > > > > > > > > > Sent: Friday, July 17, 2020 at 2:41 AM > > > From: "Nagendra Darla" <dvv.nagen...@gmail.com> > > > To: user@spark.apache.org > > > Subject: File not found exceptions on S3 while running spark jobs > > > > > > Hello All, > > > I am converting existing parquet table (size: 50GB) into Delta format. It > > > took around 1hr 45 mins to convert. > > > And I see that there are lot of FileNotFoundExceptions in the logs > > > > > > Caused by: java.io.FileNotFoundException: No such file or directory: > > > > > s3a://old-data/delta-data/PL1/output/denorm_table/part-00031-183e54ef-50bc-46fc-83a3-7836baa28f86-c000.snappy.parquet > > > > > > *How do I fix these errors?* I am using below options in spark-submit > > > command > > > > > > spark-submit --packages > > > io.delta:delta-core_2.11:0.6.0,org.apache.hadoop:hadoop-aws:2.8.5 > > > --conf > > spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore > > > --class Pipeline1 Pipeline.jar > > > > > > Thank You, > > > Nagendra Darla > > > > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > > > -- > Sent from iPhone > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
