This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ef142c4f204f [SPARK-49898][CORE] Fix documentation and default for
event log task metrics accumulator logging flag from SPARK-42204
ef142c4f204f is described below
commit ef142c4f204fcdc2382212ae55a71c3f0f6cae1f
Author: Josh Rosen <[email protected]>
AuthorDate: Tue Oct 8 17:28:19 2024 +0900
[SPARK-49898][CORE] Fix documentation and default for event log task
metrics accumulator logging flag from SPARK-42204
### What changes were proposed in this pull request?
This PR corrects an unintentional default behavior change from
https://github.com/apache/spark/pull/39763
That PR introduced a new configuration,
`spark.eventLog.includeTaskMetricsAccumulators`, to provide an ability for
users to disable the redundant logging of task metrics information via the
Accumulables field in the Spark event log task end logs.
I made a mistake in updating that PR description and code from the original
version: the description says that the intent is to not change out of the box
behavior, but the actual flag default was the opposite.
This new PR corrects both the flag default and the flag description to
reflect the original intent of not changing default behavior.
### Why are the changes needed?
Roll back an unintentional behavior change.
### Does this PR introduce _any_ user-facing change?
Yes, it rolls back an unintentional default behavior change.
### How was this patch tested?
Existing unit tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48372 from JoshRosen/fix-event-log-accumulable-defaults.
Authored-by: Josh Rosen <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../scala/org/apache/spark/internal/config/package.scala | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index c5646d2956ae..134d1d6bdb88 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -273,15 +273,15 @@ package object config {
private[spark] val EVENT_LOG_INCLUDE_TASK_METRICS_ACCUMULATORS =
ConfigBuilder("spark.eventLog.includeTaskMetricsAccumulators")
- .doc("Whether to include TaskMetrics' underlying accumulator values in
the event log (as " +
- "part of the Task/Stage/Job metrics' 'Accumulables' fields. This
configuration defaults " +
- "to false because the TaskMetrics values are already logged in the
'Task Metrics' " +
- "fields (so the accumulator updates are redundant). This flag exists
only as a " +
- "backwards-compatibility escape hatch for applications that might rely
on the old " +
- "behavior. See SPARK-42204 for details.")
+ .doc("Whether to include TaskMetrics' underlying accumulator values in
the event log " +
+ "(as part of the Task/Stage/Job metrics' 'Accumulables' fields. The
TaskMetrics " +
+ "values are already logged in the 'Task Metrics' fields (so the
accumulator updates " +
+ "are redundant). This flag defaults to true for behavioral backwards
compatibility " +
+ "for applications that might rely on the redundant logging. " +
+ "See SPARK-42204 for details.")
.version("4.0.0")
.booleanConf
- .createWithDefault(false)
+ .createWithDefault(true)
private[spark] val EVENT_LOG_OVERWRITE =
ConfigBuilder("spark.eventLog.overwrite")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
