Github user aalobaidi commented on the issue:
https://github.com/apache/spark/pull/21500
Sorry for the late reply. The option is useful for specific use case which
is micro-batches with relatively large number partitions with each of the
partitions is very big in size. When this option is enabled, Spark will load
the state of a partition from disk, process all events belonging to the
partition and then commit the new state (delta) to disk and unloaded the entire
partition state from memory. And go to the next partition(task). This way each
executor will keep in memory the state of the partitions running concurrently
as opposite to keeping all the state of all partitions executed.
You can control the balance between memory usage and IOs by setting
`spark.sql.shuffle.partitions` (should be set before the first run of the
query).
I did JVM profiling and benchmarks with 5M events micro-batchs of total
state of ~600M key 6 nodes EMR cluster. The memory usage was much better (in
fact the default behavior failed with less than 200M key) and performance
wasn't affected significantly. (I will have to compile more specific numbers).
@HeartSaVioR brings a good point regarding state compaction (snapshots). I
canât confirm if compactions was working or not during the test, I will have
to get back to you guys about this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
