Where is this list of files coming from? If you made the list, then yes, the expectation is generally "supply a list of files which are present" on the basis that general convention is "missing files are considered bad"
Though you could try setting spark.sql.files.ignoreCorruptFiles=true to see what happens Past discussion on the topic of : what if the set of files off s3 includes files which have been moved offline, where the conclusion was "you get to filter, sorry" https://issues.apache.org/jira/browse/SPARK-21797 On Mon, Jul 1, 2019 at 2:52 AM Naresh Peshwe <nareshpeshwe12...@gmail.com> wrote: > Hi All, > When I try to read a list parquet files from S3, my application errors out > if even one of the files are absent. When I searched for solutions most of > them suggested filtering the list of files (on presence) before calling > read. > Shouldn't this be handled by Spark by providing an option for continuing > without throwing an error? If not, could you point me to the thread where > this was discussed upon. > > > Regards, > Naresh >
