Takeshi Yamamuro created SPARK-4725:
---------------------------------------
Summary: Re-think custom shuffle serializers for vertex messages
Key: SPARK-4725
URL: https://issues.apache.org/jira/browse/SPARK-4725
Project: Spark
Issue Type: Improvement
Components: GraphX
Reporter: Takeshi Yamamuro
Priority: Minor
These serializers are removed in Spark-3649 because some type mismatch
errors occur in SortShuffleWriter.
https://www.mail-archive.com/[email protected]/msg04125.html
However, messages between executors might be of critical performance issues in
PageRank and other communication-intensive graph tasks.
Ankur reported that the removal caused a slowdown and the increase
of per-iteration communications in the commit log.
I made a patch to avoid the type-mismatch error in
https://github.com/maropu/spark/commit/20e74f0e41ed99cb0a89ec5e5fc0e3c9e3f1038e#diff-68f4d319d5a58cbe0729476e0cb8594aR39
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
