dongjoon-hyun commented on code in PR #59187:
URL: https://github.com/apache/spark/pull/59187#discussion_r4158564232
##########
docs/running-on-kubernetes.md:
##########
@@ -2110,14 +2112,22 @@ See the [configuration page](configuration.html) for
information on Spark config
TOTAL_DURATION policy chooses an executor with the biggest total task time.
AVERAGE_DURATION policy chooses an executor with the biggest average task
time.
FAILED_TASKS policy chooses an executor with the most number of failed
tasks.
+ PEAK_JVM_ONHEAP_MEMORY policy chooses an executor with the biggest peak
JVM on-heap memory.
+ PEAK_JVM_OFFHEAP_MEMORY policy chooses an executor with the biggest peak
JVM off-heap memory.
+ TOTAL_SHUFFLE_WRITE policy chooses an executor with the biggest total
shuffle write.
+ DISK_USED policy chooses an executor with the biggest used disk size.
ACTIVE_TASKS policy chooses an executor with the smallest number of active
tasks.
If there is a tie, it chooses an executor with the smallest add-time.
It is recommended to use it with
spark.kubernetes.executor.minTasksPerExecutorBeforeRolling
because newly started executors usually have no active tasks.
OUTLIER policy chooses an executor with outstanding statistics which is
bigger than
at least two standard deviation from the mean in average task time,
- total task time, total task GC time, and the number of failed tasks if
exists.
+ total task time, total task GC time, the number of failed tasks,
+ peak JVM on-heap memory, peak JVM off-heap memory, total shuffle write,
+ and disk used if exists.
Review Comment:
**Nit (P3):** This lists eight dimensions, but
`outliersFromMultipleDimensions` concatenates the per-dimension outlier lists
in a fixed order (average task time > total task time > total task GC time >
failed tasks > peak JVM on-heap > peak JVM off-heap > total shuffle write >
disk used) and `choose` takes the first element. When different executors are
outliers in different dimensions, the documented list alone does not tell the
reader which one is rolled. Since the list is written in that same order, one
sentence such as "The dimensions are checked in this order and the first
outlier found is chosen." would make it explicit. The same applies to the
`Config.scala` doc string.
##########
docs/running-on-kubernetes.md:
##########
@@ -2110,14 +2112,22 @@ See the [configuration page](configuration.html) for
information on Spark config
TOTAL_DURATION policy chooses an executor with the biggest total task time.
AVERAGE_DURATION policy chooses an executor with the biggest average task
time.
FAILED_TASKS policy chooses an executor with the most number of failed
tasks.
+ PEAK_JVM_ONHEAP_MEMORY policy chooses an executor with the biggest peak
JVM on-heap memory.
+ PEAK_JVM_OFFHEAP_MEMORY policy chooses an executor with the biggest peak
JVM off-heap memory.
+ TOTAL_SHUFFLE_WRITE policy chooses an executor with the biggest total
shuffle write.
+ DISK_USED policy chooses an executor with the biggest used disk size.
Review Comment:
**Nit (P3):** "the biggest used disk size" can be read as the executor's
local disk usage (scratch space or shuffle files). The metric behind
`DISK_USED` is `ExecutorSummary.diskUsed`, which `AppStatusListener`
accumulates only from `SparkListenerBlockUpdated` events for RDD, stream and
broadcast blocks (`updateExecutorMemoryDiskInfo`). Shuffle files are not
counted. An executor that holds large shuffle data but no disk-persisted blocks
therefore never ranks first under this policy, and the same applies to the new
"disk used" dimension of `OUTLIER`.
Suggestion: "DISK_USED policy chooses an executor with the biggest disk size
used by its stored blocks (e.g., disk-persisted RDD blocks)." The same sentence
is in the `Config.scala` doc string (line 264), which is outside this diff's
hunks, so please update both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
