Burak Yavuz created SPARK-17879: ----------------------------------- Summary: Don't compact metadata logs constantly into a single compacted file Key: SPARK-17879 URL: https://issues.apache.org/jira/browse/SPARK-17879 Project: Spark Issue Type: Bug Components: SQL, Streaming Affects Versions: 2.0.1 Reporter: Burak Yavuz
With metadata log compaction, we compact all files into a single file every "n" batches. The problem is, over time, this single file becomes huge, and could become an issue to constantly write out in the driver. It would be a good idea to cap the compacted file size, so that we don't end up writing huge files in the driver. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org