This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c4b880f8d39 [SPARK-42421][CORE] Use the utils to get the switch for
dynamic allocation used in local checkpoint
c4b880f8d39 is described below
commit c4b880f8d3936c1b7c8ddd9621ab392c70ace1e7
Author: Wanqiang Ji <[email protected]>
AuthorDate: Mon May 29 10:47:13 2023 +0800
[SPARK-42421][CORE] Use the utils to get the switch for dynamic allocation
used in local checkpoint
### What changes were proposed in this pull request?
Use the utils to get the switch for dynamic allocation used in local
checkpoint
### Why are the changes needed?
In RDD's local checkpoint, only through retrieve the value from
configuration, but not adjudge the local master and testing for dynamic
allocation which unified in Utils.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Used the existing UTs
Closes #39998 from jiwq/SPARK-42421.
Authored-by: Wanqiang Ji <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
---
core/src/main/scala/org/apache/spark/rdd/RDD.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/rdd/RDD.scala
b/core/src/main/scala/org/apache/spark/rdd/RDD.scala
index e9a2d7a5331..a21d2ae7739 100644
--- a/core/src/main/scala/org/apache/spark/rdd/RDD.scala
+++ b/core/src/main/scala/org/apache/spark/rdd/RDD.scala
@@ -1679,8 +1679,8 @@ abstract class RDD[T: ClassTag](
* The checkpoint directory set through `SparkContext#setCheckpointDir` is
not used.
*/
def localCheckpoint(): this.type = RDDCheckpointData.synchronized {
- if (conf.get(DYN_ALLOCATION_ENABLED) &&
- conf.contains(DYN_ALLOCATION_CACHED_EXECUTOR_IDLE_TIMEOUT)) {
+ if (Utils.isDynamicAllocationEnabled(conf) &&
+ conf.contains(DYN_ALLOCATION_CACHED_EXECUTOR_IDLE_TIMEOUT)) {
logWarning("Local checkpointing is NOT safe to use with dynamic
allocation, " +
"which removes executors along with their cached blocks. If you must
use both " +
"features, you are advised to set
`spark.dynamicAllocation.cachedExecutorIdleTimeout` " +
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
