[ https://issues.apache.org/jira/browse/SPARK-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust updated SPARK-17879: ------------------------------------- Target Version/s: 2.1.0 > Don't compact metadata logs constantly into a single compacted file > ------------------------------------------------------------------- > > Key: SPARK-17879 > URL: https://issues.apache.org/jira/browse/SPARK-17879 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.0.1 > Reporter: Burak Yavuz > > With metadata log compaction, we compact all files into a single file every > "n" batches. The problem is, over time, this single file becomes huge, and > could become an issue to constantly write out in the driver. > It would be a good idea to cap the compacted file size, so that we don't end > up writing huge files in the driver. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org