[
https://issues.apache.org/jira/browse/SPARK-26404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463384#comment-17463384
]
jingxiong zhong commented on SPARK-26404:
-----------------------------------------
@gollum999Tim Sanders,hey sir, I have a question about that how can I add my
python dependency into spark job, as following
{code:sh}
spark-submit \
--archives s3a://path/python3.6.9.tgz#python3.6.9 \
--conf "spark.pyspark.driver.python=python3.6.9/bin/python3" \
--conf "spark.pyspark.python=python3.6.9/bin/python3" \
--name "piroottest" \
./examples/src/main/python/pi.py 10
{code}
this can't run my job sucessfully，it throw error
{code:sh}
Traceback (most recent call last):
File "/tmp/spark-63b77184-6e89-4121-bc32-6a1b793e0c85/pi.py", line 21, in
<module>
from pyspark.sql import SparkSession
File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 121, in
<module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/__init__.py", line 42, in
<module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/types.py", line 27, in
<module>
async def _ag():
File "/opt/spark/work-dir/python3.6.9/lib/python3.6/ctypes/__init__.py", line
7, in <module>
from _ctypes import Union, Structure, Array
ImportError: libffi.so.6: cannot open shared object file: No such file or
directory
{code}
Or is there another way to add Python dependencies？
> set spark.pyspark.python or PYSPARK_PYTHON doesn't work in k8s client-cluster
> mode.
> -----------------------------------------------------------------------------------
>
> Key: SPARK-26404
> URL: https://issues.apache.org/jira/browse/SPARK-26404
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Spark Core
> Affects Versions: 2.4.0
> Reporter: Dongqing Liu
> Priority: Major
>
> Neither
> conf.set("spark.executorEnv.PYSPARK_PYTHON", "/opt/pythonenvs/bin/python")
> nor
> conf.set("spark.pyspark.python", "/opt/pythonenvs/bin/python")
> works.
> Looks like the executor always picks python from PATH.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
