This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 388824c4488 [SPARK-41327][CORE] Fix
`SparkStatusTracker.getExecutorInfos` by switch On/OffHeapStorageMemory info
388824c4488 is described below
commit 388824c448804161b076507f0f39ef0596e0a0bf
Author: Lingyun Yuan <ylyb...@gmail.com>
AuthorDate: Wed Nov 30 11:47:15 2022 -0800
[SPARK-41327][CORE] Fix `SparkStatusTracker.getExecutorInfos` by switch
On/OffHeapStorageMemory info
### What changes were proposed in this pull request?
This PR aims to fix `SparkStatusTracker.getExecutorInfos` to return a
correct `on/offHeapStorageMemory`.
### Why are the changes needed?
`SparkExecutorInfoImpl` used the following parameter order.
https://github.com/apache/spark/blob/54c57fa86906f933e089a33ef25ae0c053769cc8/core/src/main/scala/org/apache/spark/StatusAPIImpl.scala#L42-L45
SPARK-20659 introduced a bug with wrong parameter order at Apache Spark
2.4.0.
-
https://github.com/apache/spark/pull/20546/files#diff-7daca909d33ff8e9b4938e2b4a4aaa1558fbdf4604273b9e38cce32c55e1508cR118-R121
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
Manually review.
Closes #38843 from ylybest/master.
Lead-authored-by: Lingyun Yuan <ylyb...@gmail.com>
Co-authored-by: ylybest <119458293+ylyb...@users.noreply.github.com>
Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
core/src/main/scala/org/apache/spark/SparkStatusTracker.scala | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala
b/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala
index 37e673cd8c7..22dc1d056ec 100644
--- a/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala
+++ b/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala
@@ -114,10 +114,10 @@ class SparkStatusTracker private[spark] (sc:
SparkContext, store: AppStatusStore
port,
cachedMem,
exec.activeTasks,
- exec.memoryMetrics.map(_.usedOffHeapStorageMemory).getOrElse(0L),
exec.memoryMetrics.map(_.usedOnHeapStorageMemory).getOrElse(0L),
- exec.memoryMetrics.map(_.totalOffHeapStorageMemory).getOrElse(0L),
- exec.memoryMetrics.map(_.totalOnHeapStorageMemory).getOrElse(0L))
+ exec.memoryMetrics.map(_.usedOffHeapStorageMemory).getOrElse(0L),
+ exec.memoryMetrics.map(_.totalOnHeapStorageMemory).getOrElse(0L),
+ exec.memoryMetrics.map(_.totalOffHeapStorageMemory).getOrElse(0L))
}.toArray
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
