Hi there, I'm trying to read a list of paths from S3 into a dataframe for a window of time using the following:
sparkSession.read.parquet(listOfPaths:_*) In some cases, the path may not be there because there is no data, which is an acceptable scenario. However, Spark throws an AnalysisException: Path does not exist. Is there an option I can set to tell it to gracefully return an empty dataframe if a particular path is missing? Looking at the spark code, there is an option checkFilesExist, but I don't believe that is set in the particular flow of code that I'm accessing. Thanks! Sumona
