This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2f58ef2ba7e [SPARK-45457][DOCS] Surface sc.setLocalProperty()
value=NULL param meaning
2f58ef2ba7e is described below
commit 2f58ef2ba7e01556ed881d8f1cfa38643f4bb06a
Author: khalidmammadov <[email protected]>
AuthorDate: Sun Oct 8 13:51:29 2023 -0700
[SPARK-45457][DOCS] Surface sc.setLocalProperty() value=NULL param meaning
### What changes were proposed in this pull request?
sc.setLocalProperty() has special meaning/feature for `null` value as a
parameter and when supplied then removes the property associated with the key
parameter. It's only mentioned in the [Fair Scheduler section of the
documentation](https://spark.apache.org/docs/latest/job-scheduling.html#fair-scheduler-pools)
although it is not limited to scheduler pool config and has got more general
use cases.
So, it would be nice to document that on the APIs as well for users'
benefit.
### Why are the changes needed?
API documentation
### Does this PR introduce _any_ user-facing change?
Yes
### How was this patch tested?
Local lint
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #43269 from khalidmammadov/doc_set_local_prop.
Authored-by: khalidmammadov <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
R/pkg/R/sparkR.R | 1 +
core/src/main/scala/org/apache/spark/SparkContext.scala | 2 ++
python/pyspark/context.py | 5 ++++-
3 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/R/pkg/R/sparkR.R b/R/pkg/R/sparkR.R
index 1cbaad6d178..0be7e5da24d 100644
--- a/R/pkg/R/sparkR.R
+++ b/R/pkg/R/sparkR.R
@@ -662,6 +662,7 @@ setJobDescription <- function(value) {
#' Set a local property that affects jobs submitted from this thread, such as
the
#' Spark fair scheduler pool.
+#' To remove/unset property simply set `value` to NULL e.g.
setLocalProperty("key", NULL)
#'
#' @param key The key for a local property.
#' @param value The value for a local property.
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index ced26144e07..008b963d644 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -794,6 +794,8 @@ class SparkContext(config: SparkConf) extends Logging {
* may have unexpected consequences when working with thread pools. The
standard java
* implementation of thread pools have worker threads spawn other worker
threads.
* As a result, local properties may propagate unpredictably.
+ *
+ * To remove/unset property simply set `value` to null e.g.
sc.setLocalProperty("key", null)
*/
def setLocalProperty(key: String, value: String): Unit = {
if (value == null) {
diff --git a/python/pyspark/context.py b/python/pyspark/context.py
index 82cd325c279..19d3608c382 100644
--- a/python/pyspark/context.py
+++ b/python/pyspark/context.py
@@ -2340,6 +2340,8 @@ class SparkContext:
Set a local property that affects jobs submitted from this thread,
such as the
Spark fair scheduler pool.
+ To remove/unset property simply set `value` to None e.g.
sc.setLocalProperty("key", None)
+
.. versionadded:: 1.0.0
Parameters
@@ -2347,7 +2349,8 @@ class SparkContext:
key : str
The key of the local property to set.
value : str
- The value of the local property to set.
+ The value of the local property to set. If set to `None` then the
+ property will be removed
See Also
--------
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
