Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21733
@arunmahadevan @jose-torres
https://issues.apache.org/jira/browse/SPARK-24763?focusedCommentId=16541367&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16541367
I had a chance to test this patch with more kinds of use cases, and in
overall enabling option shows on far or slightly better performance whereas it
reduces state size according to the ratio of size of key-value pair. I'm now
feeling that it would make sense to adopt new strategy to the default and use
old behavior as fallback of supporting old app, but the numbers is for
persuading committers and I still agree decision would be necessary from
committer(s).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
