[ https://issues.apache.org/jira/browse/SPARK-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Saul Shanabrook updated SPARK-20568: ------------------------------------ Description: It would be great to be able to delete files after processing them with structured streaming. For example, I am reading in a bunch of JSON files and converting them into Parquet. If the JSON files are not deleted after they are processed, it quickly fills up my hard drive. I originally [posted this on Stack Overflow|http://stackoverflow.com/q/43671757/907060] and was recommended to make a feature request for it. was: It would be great to be able to delete files after processing them with structured streaming. For example, I am reading in a bunch of JSON files and converting them into Parquet. If the JSON files are not deleted after they are processed, it quickly fills up my hard drive. I originally [posted this on Stack Overflow](http://stackoverflow.com/q/43671757/907060) and was recommended to make a feature request for it. > Delete files after processing > ----------------------------- > > Key: SPARK-20568 > URL: https://issues.apache.org/jira/browse/SPARK-20568 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming > Affects Versions: 2.1.0 > Reporter: Saul Shanabrook > > It would be great to be able to delete files after processing them with > structured streaming. > For example, I am reading in a bunch of JSON files and converting them into > Parquet. If the JSON files are not deleted after they are processed, it > quickly fills up my hard drive. I originally [posted this on Stack > Overflow|http://stackoverflow.com/q/43671757/907060] and was recommended to > make a feature request for it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org