Josh Rosen created SPARK-42205:
----------------------------------
Summary: Remove logging of Accumulables in Task/Stage start events
in JsonProtocol
Key: SPARK-42205
URL: https://issues.apache.org/jira/browse/SPARK-42205
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.0.0
Reporter: Josh Rosen
Spark's JsonProtocol event logs (used by the history server) are impacted by a
race condition when tasks / stages finish very quickly:
The SparkListenerTaskStart and SparkListenerStageSubmitted events contain
mutable TaskInfo and StageInfo objects, which in turn contain Accumulables
fields. When a task or stage is submitted, Accumulables is initially empty.
When the task or stage finishes, this field is updated with values from the
task.
If a task or stage finishes before the start event has been logged by the event
logging listener then the _start_ event will contain the Accumulable values
from the task or stage _end_ event.
This information isn't used by the History Server and contributes to wasteful
bloat in event log sizes. In one real-world log, I found that ~10% of the
uncompressed log size was due to these redundant Accumulable fields.
I propose that we update JsonProtocol to skip the logging of this field for
Start/Submitted events.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
