Emil Ejbyfeldt created SPARK-40912:
--------------------------------------
Summary: Large overhead of Exceptions in DeserializationStream
Key: SPARK-40912
URL: https://issues.apache.org/jira/browse/SPARK-40912
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.3.0
Reporter: Emil Ejbyfeldt
The interface of DeserializationStream forces implementation to raise
EOFException to indicate that there is no more data. And for the
KryoDeserializtionStream it even worse since the kryo library does not raise
EOFException we pay for the price of two exceptions for each stream. For large
shuffles with lots of small stream this is quite a bit large overhead (seen
couple % of cpu time).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
