viirya commented on code in PR #59074:
URL: https://github.com/apache/spark/pull/59074#discussion_r4131003225
##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -165,18 +165,25 @@ private[spark] object History {
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("1m")
+ val EVENT_LOG_MAX_LINE_LENGTH_LIMIT: Int = 512 * 1024 * 1024
+
val EVENT_LOG_MAX_LINE_LENGTH =
ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
- .doc("Maximum length of a single event log line during replay. Lines
longer than " +
- "this are skipped with a warning instead of being read into memory,
bounding the " +
- "memory replay can use when an event log is corrupt or unexpectedly
large. Setting " +
- "this to 0 or a negative value disables the limit. " +
+ .doc("Maximum UTF-8 byte length of a single event log line during
replay, excluding " +
+ "the line ending. Longer lines are skipped with a warning, bounding
the " +
+ "memory replay can use when an event log is corrupt or unexpectedly
large. Buffer " +
+ "growth, UTF-16 storage and JSON parsing can require several times
this limit in heap " +
+ "space. Buffer capacity grows in steps: reducing 256m to 200m or 150m
may not reduce " +
+ "the retained buffer; use 128m to reach a smaller capacity. Values at
or below 0, " +
+ s"or above $EVENT_LOG_MAX_LINE_LENGTH_LIMIT, use the maximum supported
limit of " +
+ s"$EVENT_LOG_MAX_LINE_LENGTH_LIMIT bytes (512 MiB). This cap avoids
JVM array-size " +
+ "limits but does not guarantee sufficient heap space. " +
"Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and
4.2.1; and in " +
"all versions after 4.3.0.")
.version("4.3.0")
Review Comment:
Note that I found the backport PRs of
[SPARK-59407](https://issues.apache.org/jira/browse/SPARK-59407) fills this
`.version` field of this config with the version number in each branch, i.e.,
`.version("4.1.4")` on branch-4.1 backport, `.version("3.5.10")` on branch-3.5
backport.
It makes this config look a bit strange as on each version, this config has
different version value on different branch. But what `.version` specifies
should be the version that the config is added.
It seems because this is an improvement ticket and adds a new config, but it
backports to many previous branches so the config in earlier versions looks
strange.
I'm not sure how to deal with it when to backport this fix to these
branches. Just backport this so all branches have "4.3.0" in `.version` of this
config? Or leave the strange `version` unchanged? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
