Kaicheng Zhou created SPARK-61671:
-------------------------------------
Summary: Add SparkSession.listFiles and
SparkSession.listArchives for Spark Connect
Key: SPARK-61671
URL: https://issues.apache.org/jira/browse/SPARK-61671
Project: Spark
Issue Type: Improvement
Components: Connect, PySpark, SQL
Affects Versions: 4.4.0
Reporter: Kaicheng Zhou
Fix For: 4.4.0
SparkContext.listFiles and SparkContext.listArchives return the files and
archives added to resources, e.g. with SparkContext.addFile, addPyFile and
addArchive, or SparkSession.addArtifacts. Spark Connect has no SparkContext, so
they have no equivalent there, and code that moves to Spark Connect has to keep
its own list of what it added with SparkSession.addArtifacts.
The LIST FILES / LIST ARCHIVES SQL commands do run over Spark Connect, but they
read SparkContext.listFiles / listArchives, which hold the artifacts of every
session on the server. A Spark Connect session therefore sees the files and
archives of other sessions, including other users' (their
spark://.../artifacts/<session-uuid>/... URIs), and the list does not match
what its own tasks receive, which is only the session's own artifacts
(JobArtifactSet). LIST JARS has the same problem.
Proposal:
1. Add SparkSession.listFiles() and SparkSession.listArchives() to PySpark, for
both Spark Classic and Spark Connect, returning what SparkContext.listFiles /
listArchives return. In Spark Classic they delegate to the SparkContext. In
Spark Connect they run LIST FILES / LIST ARCHIVES, so no protocol change is
needed and the client also works with existing servers.
2. Make LIST FILES, LIST JARS and LIST ARCHIVES list the session's
JobArtifactSet: only the session's artifacts when artifacts are isolated
(spark.sql.artifact.isolation.enabled, always on for Spark Connect), and
everything on the SparkContext otherwise, which leaves Spark Classic unchanged.
Behavior matches SparkContext: Python files are listed as files, an archive
keeps the #fragment it was added with, and adding the same path twice lists it
once.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]