This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d9c596c040b0 [SPARK-50251][PYTHON] Add `getSystemProperty` to PySpark
`SparkContext`
d9c596c040b0 is described below
commit d9c596c040b021f8062bbe6fd38e711a25536421
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 6 21:55:11 2024 -0800
[SPARK-50251][PYTHON] Add `getSystemProperty` to PySpark `SparkContext`
### What changes were proposed in this pull request?
This PR aims to add `getSystemProperty` to PySpark `SparkContext` like
`setSystemProperty` at Apache Spark 4.0.0.
https://spark.apache.org/docs/4.0.0-preview2/api/python/reference/api/pyspark.SparkContext.setSystemProperty.html
### Why are the changes needed?
Since Apache Spark 0.9.0, `setSystemProperty` has been provided because
Python doesn't have JVM's `SystemProperties` concept. This is usefully used
like the following.
https://github.com/apache/spark/blob/99d27c9701019a0f534cc13c084895a00badee12/python/pyspark/shell.py#L84
This PR aims to add `getSystemProperty` additionally and symmetrically to
provide an easier and better experience for both Spark developers and Spark App
developers.
### Does this PR introduce _any_ user-facing change?
No. This is a new API.
### How was this patch tested?
Pass the CIs with newly added doctests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48781 from dongjoon-hyun/SPARK-50251.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/docs/source/reference/pyspark.rst | 1 +
python/pyspark/core/context.py | 22 ++++++++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/python/docs/source/reference/pyspark.rst
b/python/docs/source/reference/pyspark.rst
index 9a6fbb651716..798cda64d4d0 100644
--- a/python/docs/source/reference/pyspark.rst
+++ b/python/docs/source/reference/pyspark.rst
@@ -74,6 +74,7 @@ Spark Context APIs
SparkContext.getJobTags
SparkContext.getLocalProperty
SparkContext.getOrCreate
+ SparkContext.getSystemProperty
SparkContext.hadoopFile
SparkContext.hadoopRDD
SparkContext.listArchives
diff --git a/python/pyspark/core/context.py b/python/pyspark/core/context.py
index 4225a20f64c9..63d41c11dafd 100644
--- a/python/pyspark/core/context.py
+++ b/python/pyspark/core/context.py
@@ -554,6 +554,28 @@ class SparkContext:
assert SparkContext._jvm is not None
SparkContext._jvm.java.lang.System.setProperty(key, value)
+ @classmethod
+ def getSystemProperty(cls, key: str) -> str:
+ """
+ Get a Java system property, such as `java.home`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ key : str
+ The key of a new Java system property.
+
+ Examples
+ --------
+ >>> sc.getSystemProperty("SPARK_SUBMIT")
+ 'true'
+ >>> _ = sc.getSystemProperty("java.home")
+ """
+ SparkContext._ensure_initialized()
+ assert SparkContext._jvm is not None
+ return SparkContext._jvm.java.lang.System.getProperty(key)
+
@property
def version(self) -> str:
"""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
