Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21698
@tgravescs I'm still working on this but I would be glad if you can also
work on the "sort the serialized bytes of T" approach, actually the
retry-all-tasks approach seems more complex than I expected when it involves
commit protocol (currently a task can be only committed once, so if you already
have some tasks committed and then hit a ExecutorLost then retry-all-tasks
won't work), so I hope we can have other approaches like "sort the serialized
bytes of T" get merged into 2.4 release.
I'll post the benchmark result of DF.repartition() fix later.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
