This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new d2d3c4e [SPARK-60956][CONNECT] Upload the jars, Python files and
archives of addArtifacts as PySpark does
d2d3c4e is described below
commit d2d3c4e301d240a16b6ebb791e592bfc62e72418
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 13:13:07 2026 +0900
[SPARK-60956][CONNECT] Upload the jars, Python files and archives of
addArtifacts as PySpark does
### What changes were proposed in this pull request?
[SPARK-60956](https://issues.apache.org/jira/browse/SPARK-60956):
`SparkSession.addArtifacts(*path, pyfile=False, archive=False, file=False)` now
uploads each path as PySpark's `ArtifactManager._parse_artifacts` names it, by
the flags and its file extension:
- With `pyfile`, a `.py`, `.egg`, `.zip` or `.jar` goes to `pyfiles/`,
which Python UDFs import from.
- With `archive`, a `.zip`, `.jar`, `.tar.gz`, `.tgz` or `.tar` goes to
`archives/`. A `#<name>` fragment of the path names the directory that the
archive is extracted to.
- With `file`, any file goes to `files/`.
- Otherwise a `.class` goes to `classes/` and a `.jar` to `jars/`, the
classpath. Any other file raises `UNSUPPORTED_OPERATION` (`<path> file format`).
As in PySpark:
- A path can be a `file:` URI. A URI of another scheme raises
`UNSUPPORTED_OPERATION` (`<scheme> scheme`).
- More than one flag raises `INVALID_MULTIPLE_ARGUMENT_CONDITIONS`.
- `addArtifact` is the same method.
In the Rust crates:
- `spark_connect_core::artifact::parse_artifact` names a path.
- `SparkSession::add_artifacts` takes PySpark's defaults.
- The new `SparkSession::add_artifacts_with(paths, pyfile, archive, file)`
takes the flags.
### Why are the changes needed?
The Python client accepted the flags and dropped them, and the Rust session
uploaded every path to `files/`:
- A jar was not added to the classpath.
- An archive with a fragment failed to upload.
- `addArtifacts("note.txt")` succeeded, where PySpark raises.
- More than one flag was accepted.
- `addArtifacts("s3://b/x.jar")` failed with an unclassified error.
### Does this PR introduce _any_ user-facing change?
Yes, `addArtifacts` behaves as in PySpark 4.3.0.
In the Rust API, `SparkSession::add_artifacts` and `add_artifact` no longer
upload a file that is neither a jar nor a class; `add_artifacts_with(paths,
false, false, true)` uploads one to `files/`.
### How was this patch tested?
- Against a local Spark 4.3.0 server, a probe of six cases gives the same
results as official PySpark 4.3.0:
- the errors for a text file without a flag, an `s3:` URI and two flags;
- a UDF that imports a `pyfile=True` module, reads a `file=True` file
with `SparkFiles`, and reads from a `#alias` archive.
Before this change, four of them differed.
- New unit tests check the names and errors of `parse_artifact` for each
kind, fragments, queries, `file:` URIs and relative paths.
- A new mock-server test checks the name of each upload.
- A new offline test checks the errors.
- A new live test checks the UDF reads, and it also passes with official
PySpark 4.3.0.
- The e2e coverage test of artifacts passes the `file` flag for its text
files.
- The lib tests and every integration test, e2e included, against the local
server (643 passed), the build of all tests with `wasm-udf`, the offline Python
suites (615 passed, 8 live cases skipped), and the live drop-in and Arrow
suites (58 passed) and live cases (10 passed). Clippy reports no diagnostics in
the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #176 from HyukjinKwon/SPARK-60956-artifacts.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 59 +++++++--
crates/spark-connect-core/src/artifact.rs | 204 +++++++++++++++++++++++++++++
crates/spark-connect-core/src/client.rs | 53 ++------
crates/spark-connect/src/session.rs | 35 ++++-
crates/spark-connect/tests/e2e_coverage.rs | 14 +-
python/tests/test_arrow_handoff.py | 23 +++-
python/tests/test_dropin_e2e.py | 32 +++++
python/tests/test_dropin_offline.py | 19 +++
8 files changed, 374 insertions(+), 65 deletions(-)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 7a4d00c..4f13fcb 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1156,9 +1156,10 @@ impl PySparkSession {
self.session.clear_tags()
}
- // Mirrors reference `SparkSession.addArtifacts(*path, pyfile=False,
archive=False,
- // file=False)`. The pyfile/archive/file flags are accepted for signature
parity;
- // artifact classification is handled by the Rust transport / server.
+ /// Add local files, or `file:` URIs, as artifacts of the session, as
PySpark's
+ /// `addArtifacts(*path, pyfile=False, archive=False, file=False)`: each
path is named by
+ /// the flags and its extension, and the errors for a path that is not
supported are
+ /// PySpark's (see `spark_connect_core::artifact::parse_artifact`).
#[pyo3(name = "addArtifacts", signature = (*path, pyfile=false,
archive=false, file=false))]
fn add_artifacts(
&self,
@@ -1168,11 +1169,53 @@ impl PySparkSession {
archive: bool,
file: bool,
) -> PyResult<()> {
- let _ = (pyfile, archive, file);
+ use spark_connect_core::artifact::{parse_artifact,
UnsupportedArtifact};
+
+ if [pyfile, archive, file].iter().filter(|flag| **flag).count() > 1 {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "INVALID_MULTIPLE_ARGUMENT_CONDITIONS",
+ &[
+ ("arg_names", "'pyfile', 'archive' and/or 'file'"),
+ ("condition", "True together"),
+ ],
+ ));
+ }
+ for p in &path {
+ if let Err(unsupported) = parse_artifact(p, pyfile, archive, file)
{
+ return Err(match unsupported {
+ UnsupportedArtifact::Format(local) =>
crate::column::raise_pyspark(
+ py,
+ "PySparkRuntimeError",
+ "UNSUPPORTED_OPERATION",
+ &[("operation", &format!("{local} file format"))],
+ ),
+ UnsupportedArtifact::Scheme(scheme) =>
crate::column::raise_pyspark(
+ py,
+ "PySparkRuntimeError",
+ "UNSUPPORTED_OPERATION",
+ &[("operation", &format!("{scheme} scheme"))],
+ ),
+ UnsupportedArtifact::HashInArchivePath =>
crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "VALUE_ALLOWED",
+ &[("arg_name", "artifact path"), ("disallowed_value",
"#")],
+ ),
+ });
+ }
+ }
let refs: Vec<&str> = path.iter().map(|s| s.as_str()).collect();
- py.detach(|| self.session.add_artifacts(&refs)).to_pyerr()
+ py.detach(|| {
+ self.session
+ .add_artifacts_with(&refs, pyfile, archive, file)
+ .map_err(Box::new)
+ })
+ .to_pyerr()
}
+ /// `addArtifact` is `addArtifacts`, as in PySpark.
#[pyo3(name = "addArtifact", signature = (*path, pyfile=false,
archive=false, file=false))]
fn add_artifact(
&self,
@@ -1182,11 +1225,7 @@ impl PySparkSession {
archive: bool,
file: bool,
) -> PyResult<()> {
- // pyfile/archive/file are accepted for PySpark signature parity;
artifact
- // classification is handled by the Rust transport / server.
- let _ = (pyfile, archive, file);
- let refs: Vec<&str> = path.iter().map(|s| s.as_str()).collect();
- py.detach(|| self.session.add_artifacts(&refs)).to_pyerr()
+ self.add_artifacts(py, path, pyfile, archive, file)
}
#[pyo3(name = "newSession")]
diff --git a/crates/spark-connect-core/src/artifact.rs
b/crates/spark-connect-core/src/artifact.rs
index ace2bba..d254571 100644
--- a/crates/spark-connect-core/src/artifact.rs
+++ b/crates/spark-connect-core/src/artifact.rs
@@ -305,8 +305,212 @@ fn build_chunked_artifact_requests(
Ok(requests)
}
+/// A path of `addArtifacts` that PySpark's `ArtifactManager` does not upload.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum UnsupportedArtifact {
+ /// A file of a format that is not uploaded with the flags given, by its
local path.
+ Format(String),
+ /// A URI of a scheme other than `file`.
+ Scheme(String),
+ /// The local path of an archive with a fragment has a `#`, which the
fragment would
+ /// be taken from.
+ HashInArchivePath,
+}
+
+impl std::fmt::Display for UnsupportedArtifact {
+ /// The message of PySpark's error for the path.
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match self {
+ UnsupportedArtifact::Format(path) => {
+ write!(
+ f,
+ "[UNSUPPORTED_OPERATION] {path} file format is not
supported."
+ )
+ }
+ UnsupportedArtifact::Scheme(scheme) => {
+ write!(
+ f,
+ "[UNSUPPORTED_OPERATION] {scheme} scheme is not supported."
+ )
+ }
+ UnsupportedArtifact::HashInArchivePath => write!(
+ f,
+ "[VALUE_ALLOWED] Value for `artifact path` does not allow #."
+ ),
+ }
+ }
+}
+
+/// `%XX` escapes decoded, as `url2pathname` does on POSIX.
+fn unquote(text: &str) -> String {
+ let bytes = text.as_bytes();
+ let mut out = Vec::with_capacity(bytes.len());
+ let mut i = 0;
+ while i < bytes.len() {
+ let hex = |b: u8| (b as char).to_digit(16);
+ if bytes[i] == b'%' && i + 2 < bytes.len() {
+ if let (Some(high), Some(low)) = (hex(bytes[i + 1]), hex(bytes[i +
2])) {
+ out.push((high * 16 + low) as u8);
+ i += 3;
+ continue;
+ }
+ }
+ out.push(bytes[i]);
+ i += 1;
+ }
+ String::from_utf8_lossy(&out).into_owned()
+}
+
+/// The name that a path of `addArtifacts` is uploaded as, and the local file
to upload, as
+/// PySpark's `ArtifactManager._parse_artifacts` parses a local path or a
`file:` URI:
+///
+/// - With `pyfile`, a `.py`, `.egg`, `.zip` or `.jar` is `pyfiles/<name>`.
+/// - With `archive`, a `.zip`, `.jar`, `.tar.gz`, `.tgz` or `.tar` is
`archives/<name>`, or
+/// `archives/<name>#<fragment>` for a path with a fragment, which names the
directory
+/// that the archive is extracted to.
+/// - With `file`, any file is `files/<name>`.
+/// - Otherwise a `.class` is `classes/<name>`, a `.jar` is `jars/<name>`, and
any other file
+/// is not supported.
+pub fn parse_artifact(
+ path_or_uri: &str,
+ pyfile: bool,
+ archive: bool,
+ file: bool,
+) -> std::result::Result<(String, std::path::PathBuf), UnsupportedArtifact> {
+ // As `urlparse`: a scheme of a letter and then letters, digits, `+`, `-`
or `.` before
+ // `:`, a netloc after `//`, and the fragment after `#` and the query
after `?`.
+ let (scheme, rest) = match path_or_uri.find(':') {
+ Some(i)
+ if path_or_uri[..i].starts_with(|c: char| c.is_ascii_alphabetic())
+ && path_or_uri[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c)) =>
+ {
+ (path_or_uri[..i].to_ascii_lowercase(), &path_or_uri[i + 1..])
+ }
+ _ => (String::new(), path_or_uri),
+ };
+ let rest = match rest.strip_prefix("//") {
+ Some(after) => &after[after.find(['/', '?',
'#']).unwrap_or(after.len())..],
+ None => rest,
+ };
+ let (rest, fragment) = rest.split_once('#').unwrap_or((rest, ""));
+ let path = rest.split_once('?').map_or(rest, |(path, _)| path);
+ let local = match scheme.as_str() {
+ "" => {
+ let path = std::path::PathBuf::from(unquote(path));
+ if path.is_absolute() {
+ path
+ } else {
+ std::env::current_dir().unwrap_or_default().join(path)
+ }
+ }
+ "file" => std::path::PathBuf::from(unquote(path)),
+ _ => return Err(UnsupportedArtifact::Scheme(scheme)),
+ };
+ let local_path = local.to_string_lossy().into_owned();
+ let name = local
+ .file_name()
+ .map(|name| name.to_string_lossy().into_owned())
+ .unwrap_or_default();
+ let ends = |suffixes: &[&str]| suffixes.iter().any(|suffix|
name.ends_with(suffix));
+ let artifact = if pyfile && ends(&[".py", ".egg", ".zip", ".jar"]) {
+ format!("pyfiles/{name}")
+ } else if archive && ends(&[".zip", ".jar", ".tar.gz", ".tgz", ".tar"]) {
+ if fragment.is_empty() {
+ format!("archives/{name}")
+ } else if local_path.contains('#') {
+ return Err(UnsupportedArtifact::HashInArchivePath);
+ } else {
+ format!("archives/{name}#{fragment}")
+ }
+ } else if file {
+ format!("files/{name}")
+ } else if ends(&[".class"]) {
+ format!("classes/{name}")
+ } else if ends(&[".jar"]) {
+ format!("jars/{name}")
+ } else {
+ return Err(UnsupportedArtifact::Format(local_path));
+ };
+ Ok((artifact, local))
+}
+
#[cfg(test)]
mod tests {
+ use super::{parse_artifact, UnsupportedArtifact};
+
+ #[test]
+ fn artifacts_are_named_as_pyspark_parses_their_paths() {
+ use UnsupportedArtifact::*;
+ let named = |name: &str| Ok(name.to_string());
+ let cases = [
+ ("/d/a.jar", (false, false, false), named("jars/a.jar")),
+ (
+ "/d/A.class",
+ (false, false, false),
+ named("classes/A.class"),
+ ),
+ ("/d/m.py", (true, false, false), named("pyfiles/m.py")),
+ ("/d/m.zip", (true, false, false), named("pyfiles/m.zip")),
+ (
+ "/d/b.tar.gz",
+ (false, true, false),
+ named("archives/b.tar.gz"),
+ ),
+ // The fragment of an archive names the directory that it is
extracted to.
+ (
+ "/d/b.zip#alias",
+ (false, true, false),
+ named("archives/b.zip#alias"),
+ ),
+ ("/d/n.txt", (false, false, true), named("files/n.txt")),
+ // Another path drops its fragment, and its query, as `urlparse`
takes them.
+ ("/d/a.jar#x", (false, false, false), named("jars/a.jar")),
+ (
+ "file:/d/a.jar?q=1",
+ (false, false, false),
+ named("jars/a.jar"),
+ ),
+ (
+ "file:///d/a%20b.jar",
+ (false, false, false),
+ named("jars/a b.jar"),
+ ),
+ (
+ "/d/m.py",
+ (false, false, false),
+ Err(Format("/d/m.py".to_string())),
+ ),
+ (
+ "/d/m.py",
+ (false, true, false),
+ Err(Format("/d/m.py".to_string())),
+ ),
+ (
+ "s3://bucket/a.jar",
+ (false, false, false),
+ Err(Scheme("s3".to_string())),
+ ),
+ (
+ "/d/b%23c.zip#alias",
+ (false, true, false),
+ Err(HashInArchivePath),
+ ),
+ ];
+ for (path, (pyfile, archive, file), expected) in cases {
+ let parsed = parse_artifact(path, pyfile, archive,
file).map(|(name, _)| name);
+ assert_eq!(parsed, expected, "{path}");
+ }
+ }
+
+ #[test]
+ fn a_relative_artifact_path_is_in_the_current_directory() {
+ let (name, local) = parse_artifact("a.jar", false, false,
false).unwrap();
+ assert_eq!(name, "jars/a.jar");
+ assert_eq!(local, std::env::current_dir().unwrap().join("a.jar"));
+ }
+
use super::*;
#[test]
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 60fc68b..347c7a3 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -812,54 +812,17 @@ impl SparkConnectClient {
pub async fn add_artifacts(
&self,
paths: &[&str],
- _pyfile: bool,
- _archive: bool,
- _file: bool,
+ pyfile: bool,
+ archive: bool,
+ file: bool,
) -> Result<()> {
- // Build list of named artifacts from file paths
+ // Each path is named as PySpark's ArtifactManager names it, by the
flags and its file
+ // extension (see `parse_artifact`).
let mut artifacts = Vec::new();
for path in paths {
- // Extract just the filename as the artifact name
- let path_obj = std::path::Path::new(path);
- let file_name = path_obj
- .file_name()
- .and_then(|n| n.to_str())
- .ok_or_else(|| {
- SparkError::connect_msg(format!("Invalid artifact path:
{}", path))
- })?;
-
- // Determine artifact prefix based on file extension
- let artifact_name = if _pyfile
- && (file_name.ends_with(".py")
- || file_name.ends_with(".zip")
- || file_name.ends_with(".egg")
- || file_name.ends_with(".jar"))
- {
- format!("pyfiles/{}", file_name)
- } else if _archive
- && (file_name.ends_with(".zip")
- || file_name.ends_with(".jar")
- || file_name.ends_with(".tar.gz")
- || file_name.ends_with(".tgz")
- || file_name.ends_with(".tar"))
- {
- format!("archives/{}", file_name)
- } else if _file {
- format!("files/{}", file_name)
- } else if file_name.ends_with(".jar") {
- format!("jars/{}", file_name)
- } else {
- return Err(SparkError::connect_msg(format!(
- "Unsupported artifact type: {}",
- path
- )));
- };
-
- // Create artifact from file
- artifacts.push(NamedArtifact::new(
- artifact_name,
- Box::new(FileArtifact::new(path)),
- ));
+ let (name, local) = crate::artifact::parse_artifact(path, pyfile,
archive, file)
+ .map_err(|e| SparkError::connect_msg(e.to_string()))?;
+ artifacts.push(NamedArtifact::new(name,
Box::new(FileArtifact::new(local))));
}
if artifacts.is_empty() {
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 000e9d2..67e8870 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -569,18 +569,43 @@ impl SparkSession {
block_on(self.client.interrupt_operation(operation_id))
}
- /// Add local files as artifacts to the session (e.g. `.py`, `.jar`,
`.zip`).
- ///
- /// Mirrors `pyspark.sql.SparkSession.addArtifacts(*path)`.
+ /// Add local files, or `file:` URIs, as artifacts of the session,
mirroring
+ /// `pyspark.sql.SparkSession.addArtifacts(*path)`: a `.jar` is added to
the classpath and
+ /// a `.class` to the classes; any other file is not supported without a
flag of
+ /// [`add_artifacts_with`](Self::add_artifacts_with).
pub fn add_artifacts(&self, paths: &[&str]) -> Result<()> {
- block_on(self.client.add_artifacts(paths, false, false, true))
+ self.add_artifacts_with(paths, false, false, false)
+ }
+
+ /// Add local files as artifacts of the session, mirroring
+ /// `pyspark.sql.SparkSession.addArtifacts(*path, pyfile, archive, file)`:
+ /// - `pyfile`: a `.py`, `.egg`, `.zip` or `.jar` is importable by Python
UDFs;
+ /// - `archive`: a `.zip`, `.jar`, `.tar.gz`, `.tgz` or `.tar` is
extracted, into the
+ /// directory that a `#<name>` fragment of its path names;
+ /// - `file`: any file is added to the files of the session.
+ ///
+ /// At most one of the flags may be true.
+ pub fn add_artifacts_with(
+ &self,
+ paths: &[&str],
+ pyfile: bool,
+ archive: bool,
+ file: bool,
+ ) -> Result<()> {
+ if [pyfile, archive, file].iter().filter(|flag| **flag).count() > 1 {
+ return Err(SparkError::connect_msg(
+ "[INVALID_MULTIPLE_ARGUMENT_CONDITIONS] ['pyfile', 'archive'
and/or 'file'] \
+ cannot be True together.",
+ ));
+ }
+ block_on(self.client.add_artifacts(paths, pyfile, archive, file))
}
/// Add a single local file as an artifact to the session.
///
/// Mirrors `pyspark.sql.SparkSession.addArtifact(path)`.
pub fn add_artifact(&self, path: &str) -> Result<()> {
- block_on(self.client.add_artifacts(&[path], false, false, true))
+ self.add_artifacts(&[path])
}
/// Copy a local file to the driver's filesystem at `dest_path`.
diff --git a/crates/spark-connect/tests/e2e_coverage.rs
b/crates/spark-connect/tests/e2e_coverage.rs
index 801449d..fd4a2be 100644
--- a/crates/spark-connect/tests/e2e_coverage.rs
+++ b/crates/spark-connect/tests/e2e_coverage.rs
@@ -1030,9 +1030,17 @@ fn artifact_and_tag_surface() {
// A larger file to exercise the multi-chunk path (well over one gRPC
chunk).
std::fs::write(&f2, vec![b'x'; 512 * 1024]).unwrap();
- s.add_artifact(f1.to_str().unwrap()).expect("add_artifact");
- s.add_artifacts(&[f1.to_str().unwrap(), f2.to_str().unwrap()])
- .expect("add_artifacts");
+ // As PySpark, a file that is neither a jar nor a class needs the `file`
flag.
+ assert!(s.add_artifact(f1.to_str().unwrap()).is_err());
+ s.add_artifacts_with(&[f1.to_str().unwrap()], false, false, true)
+ .expect("add_artifact");
+ s.add_artifacts_with(
+ &[f1.to_str().unwrap(), f2.to_str().unwrap()],
+ false,
+ false,
+ true,
+ )
+ .expect("add_artifacts");
// copyFromLocalToFs: exercises the client-side named-artifact upload
// (forward_to_fs/...). A single-node server rejects a *local*
destination, but the
// client upload code we're covering runs before that, so tolerate the
server error.
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index 459a65e..3f0f4b7 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -464,7 +464,8 @@ def _arrow_input_service(
session_id = None
def save(name, data):
- if not (allow_file_artifacts and name.startswith(("files/",
"forward_to_fs/"))):
+ named = ("files/", "forward_to_fs/", "jars/", "classes/",
"pyfiles/", "archives/")
+ if not (allow_file_artifacts and name.startswith(named)):
assert name == "cache/" + hashlib.sha256(data).hexdigest()
assert name not in state["cache"], "cached artifacts should not be
uploaded again"
state["cache"][name] = bytes(data)
@@ -649,7 +650,7 @@ def test_file_upload_attribution_on_every_frame(
protocol, user_id=user_id, user_agent=user_agent,
allow_file_artifacts=True
) as (spark, state):
if upload == "file":
- spark.addArtifacts(str(small), str(large))
+ spark.addArtifacts(str(small), str(large), file=True)
artifact_names = ["files/small.txt", "files/large.txt"]
else:
destinations = ["/wire/small.txt", "/wire/large.txt"]
@@ -889,3 +890,21 @@ def test_non_arrow_input_does_not_import_pyarrow(protocol,
monkeypatch, data):
spark.createDataFrame(data)
assert imported == []
assert state["config_requests"] == []
+
+
+def test_add_artifacts_names_each_artifact_as_pyspark(protocol, tmp_path):
+ for name in ["a.jar", "A.class", "m.py", "b.zip", "n.txt"]:
+ (tmp_path / name).write_bytes(name.encode())
+ with _arrow_input_service(protocol, allow_file_artifacts=True) as (spark,
state):
+ # As PySpark's ArtifactManager names them, by the flags and the file
extension.
+ spark.addArtifacts(str(tmp_path / "a.jar"), str(tmp_path / "A.class"))
+ spark.addArtifacts(str(tmp_path / "m.py"), pyfile=True)
+ spark.addArtifacts(str(tmp_path / "b.zip") + "#alias", archive=True)
+ spark.addArtifact(str(tmp_path / "n.txt"), file=True)
+ assert state["cache"] == {
+ "jars/a.jar": b"a.jar",
+ "classes/A.class": b"A.class",
+ "pyfiles/m.py": b"m.py",
+ "archives/b.zip#alias": b"b.zip",
+ "files/n.txt": b"n.txt",
+ }
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 9734e93..4adf970 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -241,6 +241,38 @@ def test_create_dataframe_from_arrow_table(session,
schema, expected):
assert _rows(df) == ARROW_TABLE_ROWS
+def test_add_artifacts_reach_udfs_as_pyspark(session, tmp_path):
+ import uuid
+ import zipfile
+
+ tag = uuid.uuid4().hex[:8]
+ module, note, bundle = f"mod_{tag}", f"note_{tag}.txt", f"bundle_{tag}.zip"
+ (tmp_path / f"{module}.py").write_text("def f():\n return 'from the
module'\n")
+ (tmp_path / note).write_text("file content")
+ with zipfile.ZipFile(tmp_path / bundle, "w") as archive:
+ archive.writestr("inner.txt", "inside")
+ session.addArtifacts(str(tmp_path / f"{module}.py"), pyfile=True)
+ session.addArtifacts(str(tmp_path / note), file=True)
+ session.addArtifacts(str(tmp_path / bundle) + f"#alias_{tag}",
archive=True)
+
+ @F.udf("string")
+ def read(_):
+ import importlib
+ import os
+
+ imported = importlib.import_module(module)
+ # The directory of the session's files, which holds the module. Not
SparkFiles: a
+ # worker started with SPARK_SKIP_CONNECT_COMPAT_TESTS, as in CI, does
not set it up.
+ root = os.path.dirname(imported.__file__)
+ return "|".join([
+ imported.f(),
+ open(os.path.join(root, note)).read(),
+ open(os.path.join(root, f"alias_{tag}", "inner.txt")).read(),
+ ])
+
+ assert session.range(1).select(read("id")).first()[0] == "from the
module|file content|inside"
+
+
def test_create_dataframe_from_empty_arrow_table(session):
empty = pa.table({"a": pa.array([], pa.int64()), "b": pa.array([],
pa.string())})
df = session.createDataFrame(empty, "a long, b string")
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index df2d96f..2c0a9d0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -558,6 +558,25 @@ def test_pandas_udf():
assert callable(p) or p is not None
+def test_add_artifacts_raises_for_paths_that_pyspark_does_not_upload(tmp_path):
+ from pyspark.errors import PySparkRuntimeError, PySparkValueError
+
+ spark = _offline_session()
+ note = tmp_path / "note.txt"
+ note.write_text("x")
+ # Raised before any upload, as in PySpark.
+ for path, operation in [(str(note), f"{note} file format"),
("s3://b/x.jar", "s3 scheme")]:
+ with pytest.raises(PySparkRuntimeError) as error:
+ spark.addArtifacts(path)
+ assert error.value.getCondition() == "UNSUPPORTED_OPERATION"
+ assert error.value.getMessageParameters() == {"operation": operation}
+ with pytest.raises(PySparkValueError) as error:
+ spark.addArtifacts(str(note), pyfile=True, file=True)
+ assert error.value.getCondition() == "INVALID_MULTIPLE_ARGUMENT_CONDITIONS"
+ assert error.value.getMessageParameters() == {
+ "arg_names": "'pyfile', 'archive' and/or 'file'", "condition": "True
together"}
+
+
def test_dropin_module_imports():
# The thin re-export shims import cleanly.
import pyspark.sql.dataframe # noqa: F401
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]