[
https://issues.apache.org/jira/browse/SPARK-52669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-52669:
----------------------------------
Fix Version/s: (was: 4.1.0)
> Improvement PySpark run with python directly could not find correct python
> exec
> -------------------------------------------------------------------------------
>
> Key: SPARK-52669
> URL: https://issues.apache.org/jira/browse/SPARK-52669
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.0.0
> Environment: Spark version: the latest (3.3.2+)
> OS: Centos
> JDK: 8.0.422-kona
> Python: 3.10.15
> Reporter: gongwendong
> Priority: Minor
> Labels: pull-request-available
> Attachments: image-2025-07-03-15-03-16-079.png
>
>
> * issue information
> run in cluster: yarn, deploy mode: client with run.py not work fine. cause
> error about driver and executor about python version.
> {code:java}
> // in run.py python code
> SparkSession.builder
> .appName('sample on conflict python exec error')
> .master('yarn')
> .config('spark.submit.pyFiles',emr.project_pack())
> .config('spark.ui.enabled','true')
> .config('spark.pyspark.driver.python','./environment/bin/python')
> .config('spark.pyspark.python','./environment/bin/python')
> .config("spark.archives",
> f"hdfs:///spark/env/xxxxx.dist.archives.tar.gz#environment")
> .enableHiveSupport()
> .getOrCreate()
> spark.range(1).rdd.map(lambda x: (x, f"Executor Python version:
> {sys.version}",
> f"#Executor Python executable:
> {sys.executable}")).collect(){code}
>
> * exception & error:RuntimeError: Python in worker has different version 3.6
> than that in driver 3.10, PySpark cannot run with different minor versions.
> Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
> are correctly set.
> !image-2025-07-03-15-03-16-079.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
