This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b23905ac0c62 [SPARK-50222][PYTHON][FOLLOWUP] Support
`spark.submit.appName` in PySpark
b23905ac0c62 is described below
commit b23905ac0c628a32ac5062c2fdb90e1b3564dcde
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 8 15:45:24 2024 -0800
[SPARK-50222][PYTHON][FOLLOWUP] Support `spark.submit.appName` in PySpark
### What changes were proposed in this pull request?
This PR is a follow-up in order to support `spark.submit.appName` in
PySpark.
- #48755
### Why are the changes needed?
This follow-up will override the static PySpark application's appName like
the following in the same way like Java/Scala.
https://github.com/apache/spark/blob/d9c596c040b021f8062bbe6fd38e711a25536421/examples/src/main/python/pi.py#L29-L32
**BEFORE**
```
$ bin/spark-submit --name NAME examples/src/main/python/pi.py 2>&1 | grep
app_name | jq
{
"ts": "2024-11-07T08:07:20.806Z",
"level": "INFO",
"msg": "Submitted application: PythonPi",
"context": {
"app_name": "PythonPi"
},
"logger": "SparkContext"
}
```
```
$ bin/spark-submit -c spark.app.name=NAME examples/src/main/python/pi.py
2>&1 | grep app_name | jq
{
"ts": "2024-11-07T08:08:23.685Z",
"level": "INFO",
"msg": "Submitted application: PythonPi",
"context": {
"app_name": "PythonPi"
},
"logger": "SparkContext"
}
```
**AFTER**
```
$ bin/spark-submit -c spark.submit.appName=NAME
examples/src/main/python/pi.py 2>&1 | grep app_name | jq
{
"ts": "2024-11-07T08:09:02.084Z",
"level": "INFO",
"msg": "Submitted application: NAME",
"context": {
"app_name": "NAME"
},
"logger": "SparkContext"
}
```
### Does this PR introduce _any_ user-facing change?
No, this is a new configuration.
### How was this patch tested?
Pass the CIs with a newly added test case.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48788 from dongjoon-hyun/SPARK-50222-3.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/sql/session.py | 3 +++
python/pyspark/tests/test_appsubmit.py | 20 ++++++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/python/pyspark/sql/session.py b/python/pyspark/sql/session.py
index 748dd2cafa7c..a5b76a27b296 100644
--- a/python/pyspark/sql/session.py
+++ b/python/pyspark/sql/session.py
@@ -543,9 +543,12 @@ class SparkSession(SparkConversionMixin):
session = SparkSession._instantiatedSession
if session is None or session._sc._jsc is None:
+ SparkContext._ensure_initialized()
sparkConf = SparkConf()
for key, value in self._options.items():
sparkConf.set(key, value)
+ if sparkConf.contains("spark.submit.appName"):
+
sparkConf.setAppName(sparkConf.get("spark.submit.appName", ""))
# This SparkContext may be an existing one.
sc = SparkContext.getOrCreate(sparkConf)
# Do not update `SparkConf` for existing `SparkContext`,
as it's shared
diff --git a/python/pyspark/tests/test_appsubmit.py
b/python/pyspark/tests/test_appsubmit.py
index 79b6b4fa91a7..0645bf2dc64b 100644
--- a/python/pyspark/tests/test_appsubmit.py
+++ b/python/pyspark/tests/test_appsubmit.py
@@ -293,6 +293,26 @@ class SparkSubmitTests(unittest.TestCase):
out, err = proc.communicate()
self.assertEqual(0, proc.returncode, msg="Process failed with error:\n
{0}".format(out))
+ def test_session(self):
+ """Make sure spark.submit.appName overrides the appName in script"""
+ script = self.createTempFile(
+ "test.py",
+ """
+ |from pyspark.sql import SparkSession
+ |spark = SparkSession.builder.appName("PythonPi").getOrCreate()
+ |print(spark.sparkContext.appName)
+ |spark.stop()
+ """,
+ )
+ proc = subprocess.Popen(
+ self.sparkSubmit + ["--master", "local", "-c",
"spark.submit.appName=NEW", script],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ )
+ out, err = proc.communicate()
+ self.assertEqual(0, proc.returncode)
+ self.assertIn("NEW", out.decode("utf-8"))
+
if __name__ == "__main__":
from pyspark.tests.test_appsubmit import * # noqa: F401
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
