Hello All, I am converting existing parquet table (size: 50GB) into Delta format. It took around 1hr 45 mins to convert. And I see that there are lot of FileNotFoundExceptions in the logs
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://old-data/delta-data/PL1/output/denorm_table/part-00031-183e54ef-50bc-46fc-83a3-7836baa28f86-c000.snappy.parquet *How do I fix these errors?* I am using below options in spark-submit command spark-submit --packages io.delta:delta-core_2.11:0.6.0,org.apache.hadoop:hadoop-aws:2.8.5 --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore --class Pipeline1 Pipeline.jar Thank You, Nagendra Darla
