Burak Yavuz created SPARK-17599: ----------------------------------- Summary: Folder deletion after globbing may fail StructuredStreaming jobs Key: SPARK-17599 URL: https://issues.apache.org/jira/browse/SPARK-17599 Project: Spark Issue Type: Bug Components: SQL, Streaming Affects Versions: 2.0.0 Reporter: Burak Yavuz
The FileStreamSource used by StructuredStreaming first resolves globs, and then creates a ListingFileCatalog which listFiles with the resolved glob patterns. If a folder is deleted after glob resolution but before the ListingFileCatalog can list the files, we can run into a 'FileNotFoundException'. This should not be a fatal exception for a streaming job. However we should include a warn message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org