Bobby Wang created SPARK-57192:
----------------------------------
Summary: SparkSession.addArtifacts fails with multiple paths in
classic PySpark
Key: SPARK-57192
URL: https://issues.apache.org/jira/browse/SPARK-57192
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 5.0.0
Reporter: Bobby Wang
h1. Description
In classic PySpark, SparkSession.addArtifacts accepts multiple paths via *path,
but calling it with multiple Python files fails when pyfile=True.
h1. Reproducer:
spark.addArtifacts("a.py", "b.py", "c.py", pyfile=True)
h1. Actual result:
TypeError: SparkContext.addPyFile() takes 2 positional arguments but 4 were
given
The classic implementation forwards all paths at once to
SparkContext.addPyFile/addArchive/addFile, but those APIs accept one path at a
time.
h1. Expected result:
SparkSession.addArtifacts should add each artifact successfully when multiple
paths are provided.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
