This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new efae5362cbe [SPARK-42421][CORE] Use the utils to get the switch for
dynamic allocation used in local checkpoint
efae5362cbe is described below
commit efae5362cbeaf1594b18edf594e83b2cf72afce6
Author: Wanqiang Ji <[email protected]>
AuthorDate: Mon May 29 10:47:13 2023 +0800
[SPARK-42421][CORE] Use the utils to get the switch for dynamic allocation
used in local checkpoint
### What changes were proposed in this pull request?
Use the utils to get the switch for dynamic allocation used in local
checkpoint
### Why are the changes needed?
In RDD's local checkpoint, only through retrieve the value from
configuration, but not adjudge the local master and testing for dynamic
allocation which unified in Utils.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Used the existing UTs
Closes #39998 from jiwq/SPARK-42421.
Authored-by: Wanqiang Ji <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
(cherry picked from commit c4b880f8d3936c1b7c8ddd9621ab392c70ace1e7)
Signed-off-by: Kent Yao <[email protected]>
---
core/src/main/scala/org/apache/spark/rdd/RDD.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/rdd/RDD.scala
b/core/src/main/scala/org/apache/spark/rdd/RDD.scala
index 18f3f87f30f..407820b663a 100644
--- a/core/src/main/scala/org/apache/spark/rdd/RDD.scala
+++ b/core/src/main/scala/org/apache/spark/rdd/RDD.scala
@@ -1652,8 +1652,8 @@ abstract class RDD[T: ClassTag](
* The checkpoint directory set through `SparkContext#setCheckpointDir` is
not used.
*/
def localCheckpoint(): this.type = RDDCheckpointData.synchronized {
- if (conf.get(DYN_ALLOCATION_ENABLED) &&
- conf.contains(DYN_ALLOCATION_CACHED_EXECUTOR_IDLE_TIMEOUT)) {
+ if (Utils.isDynamicAllocationEnabled(conf) &&
+ conf.contains(DYN_ALLOCATION_CACHED_EXECUTOR_IDLE_TIMEOUT)) {
logWarning("Local checkpointing is NOT safe to use with dynamic
allocation, " +
"which removes executors along with their cached blocks. If you must
use both " +
"features, you are advised to set
`spark.dynamicAllocation.cachedExecutorIdleTimeout` " +
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
