This is an automated email from the ASF dual-hosted git repository.
ulyssesyou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 08554732f4fd [SPARK-48880][CORE] Avoid throw NullPointerException if
driver plugin fails to initialize
08554732f4fd is described below
commit 08554732f4fdd0695dfc4b07557e344009d325f4
Author: ulysses-you <[email protected]>
AuthorDate: Mon Jul 15 11:26:13 2024 +0800
[SPARK-48880][CORE] Avoid throw NullPointerException if driver plugin fails
to initialize
### What changes were proposed in this pull request?
This pr skips clear memoryStore if memoryManager is null. This could happen
if driver plugin fails to initialize, since we initialize MemoryManager after
DriverPlugin.
### Why are the changes needed?
before it would throw:
```
{"class":"java.lang.NullPointerException","msg":"Cannot invoke
\"org.apache.spark.memory.MemoryManager.maxOnHeapStorageMemory()\" because
\"this.memoryManager\" is
null","stacktrace":[{"class":"org.apache.spark.storage.memory.MemoryStore","method":"maxMemory","file":"MemoryStore.scala","line":110},
{"class":"org.apache.spark.storage.memory.MemoryStore","method":"<init>","file":"MemoryStore.scala","line":113},
{"class":"org.apache.spark.storage.BlockManager","method":"memoryStore$lzycompute","file":"BlockManager.scala","line":234},
{"class":"org.apache.spark.storage.BlockManager","method":"memoryStore","file":"BlockManager.scala","line":233},
{"class":"org.apache.spark.storage.BlockManager","method":"stop","file":"BlockManager.scala","line":2167},
{"class":"org.apache.spark.SparkEnv","method":"stop","file":"SparkEnv.scala","line":118},
{"class":"org.apache.spark.SparkContext","method":"$anonfun$stop$25","file":"SparkContext.scala","line":2369},
{"class":"org.apache.spark.util.Utils$","method":"tryLogNonFatalError","file":"Utils.scala","line":1299},
{"class":"org.apache.spark.SparkContext","method":"stop","file":"SparkContext.scala","line":2369}
```
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
manually test
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #47321 from ulysses-you/minor.
Authored-by: ulysses-you <[email protected]>
Signed-off-by: youxiduo <[email protected]>
---
core/src/main/scala/org/apache/spark/storage/BlockManager.scala | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
index 8655b7231079..a6d62005e4e6 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
@@ -2164,7 +2164,10 @@ private[spark] class BlockManager(
diskBlockManager.stop()
rpcEnv.stop(storageEndpoint)
blockInfoManager.clear()
- memoryStore.clear()
+ // The memoryManager may be null if the driver plugin fails to initialize
+ if (memoryManager != null) {
+ memoryStore.clear()
+ }
futureExecutionContext.shutdownNow()
logInfo("BlockManager stopped")
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
