Josh Rosen created SPARK-5077:
---------------------------------
Summary: Map output statuses can still exceed spark.akka.frameSize
Key: SPARK-5077
URL: https://issues.apache.org/jira/browse/SPARK-5077
Project: Spark
Issue Type: Bug
Components: Shuffle
Affects Versions: 1.2.0, 1.3.0
Reporter: Josh Rosen
Since HighlyCompressedMapOutputStatuses uses a bitmap for tracking empty
blocks, its size is not bounded and thus Spark is still susceptible to
"MapOutputTrackerMasterActor: Map output statuses
were 11141547 bytes which exceeds spark.akka.frameSize"-type errors, even in
1.2.0.
We needed to use a bitmap for tracking zero-sized blocks (see SPARK-3740; this
isn't just a performance issue; it's necessary for correctness). This will
require a bit more effort to fix, since we'll either have to find a way to use
a fixed size / capped size encoding for MapOutputStatuses (which might require
changes to let us fetch empty blocks safely) or figure out some other strategy
for shipping these statues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
