[spark] branch master updated: [SPARK-33893][CORE] Exclude fallback block manager from executorList

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new d467d81  [SPARK-33893][CORE] Exclude fallback block manager from 
executorList
d467d81 is described below

commit d467d817260d6ca605c34f493e68d0877209170f
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Wed Dec 23 15:31:56 2020 -0800

    [SPARK-33893][CORE] Exclude fallback block manager from executorList
    
    ### What changes were proposed in this pull request?
    
    This PR aims to exclude fallback block manager from `executorList` function.
    
    ### Why are the changes needed?
    
    When a fallback storage is used, the executors UI tab hangs because the 
executor list REST API result doesn't have `peakMemoryMetrics` of 
`ExecutorMetrics`. The root cause is that the block manager id used by fallback 
storage is included in the API result and it doesn't have `peakMemoryMetrics` 
because it's populated during HeartBeat reporting. We should hide it.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a bug fix on UI.
    
    ### How was this patch tested?
    
    Manual. Run the following and visit Spark `executors` tab UI with browser.
    ```
    bin/spark-shell -c 
spark.storage.decommission.fallbackStorage.path=file:///tmp/spark-storage/
    ```
    
    Closes #30911 from dongjoon-hyun/SPARK-33893.
    
    Authored-by: Dongjoon Hyun <dh...@apple.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 core/src/main/scala/org/apache/spark/status/AppStatusStore.scala | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala 
b/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
index affa85b..b9cc914 100644
--- a/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
+++ b/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
@@ -24,6 +24,7 @@ import scala.collection.mutable.HashMap
 
 import org.apache.spark.{JobExecutionStatus, SparkConf}
 import org.apache.spark.status.api.v1
+import org.apache.spark.storage.FallbackStorage.FALLBACK_BLOCK_MANAGER_ID
 import org.apache.spark.ui.scope._
 import org.apache.spark.util.Utils
 import org.apache.spark.util.kvstore.{InMemoryStore, KVStore}
@@ -88,7 +89,7 @@ private[spark] class AppStatusStore(
     } else {
       base
     }
-    filtered.asScala.map(_.info).toSeq
+    filtered.asScala.map(_.info).filter(_.id != 
FALLBACK_BLOCK_MANAGER_ID.executorId).toSeq
   }
 
   def executorSummary(executorId: String): v1.ExecutorSummary = {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org