This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new d2d3c4e  [SPARK-60956][CONNECT] Upload the jars, Python files and 
archives of addArtifacts as PySpark does
d2d3c4e is described below

commit d2d3c4e301d240a16b6ebb791e592bfc62e72418
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 13:13:07 2026 +0900

    [SPARK-60956][CONNECT] Upload the jars, Python files and archives of 
addArtifacts as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60956](https://issues.apache.org/jira/browse/SPARK-60956): 
`SparkSession.addArtifacts(*path, pyfile=False, archive=False, file=False)` now 
uploads each path as PySpark's `ArtifactManager._parse_artifacts` names it, by 
the flags and its file extension:
    
    - With `pyfile`, a `.py`, `.egg`, `.zip` or `.jar` goes to `pyfiles/`, 
which Python UDFs import from.
    - With `archive`, a `.zip`, `.jar`, `.tar.gz`, `.tgz` or `.tar` goes to 
`archives/`. A `#<name>` fragment of the path names the directory that the 
archive is extracted to.
    - With `file`, any file goes to `files/`.
    - Otherwise a `.class` goes to `classes/` and a `.jar` to `jars/`, the 
classpath. Any other file raises `UNSUPPORTED_OPERATION` (`<path> file format`).
    
    As in PySpark:
    - A path can be a `file:` URI. A URI of another scheme raises 
`UNSUPPORTED_OPERATION` (`<scheme> scheme`).
    - More than one flag raises `INVALID_MULTIPLE_ARGUMENT_CONDITIONS`.
    - `addArtifact` is the same method.
    
    In the Rust crates:
    - `spark_connect_core::artifact::parse_artifact` names a path.
    - `SparkSession::add_artifacts` takes PySpark's defaults.
    - The new `SparkSession::add_artifacts_with(paths, pyfile, archive, file)` 
takes the flags.
    
    ### Why are the changes needed?
    
    The Python client accepted the flags and dropped them, and the Rust session 
uploaded every path to `files/`:
    
    - A jar was not added to the classpath.
    - An archive with a fragment failed to upload.
    - `addArtifacts("note.txt")` succeeded, where PySpark raises.
    - More than one flag was accepted.
    - `addArtifacts("s3://b/x.jar")` failed with an unclassified error.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, `addArtifacts` behaves as in PySpark 4.3.0.
    
    In the Rust API, `SparkSession::add_artifacts` and `add_artifact` no longer 
upload a file that is neither a jar nor a class; `add_artifacts_with(paths, 
false, false, true)` uploads one to `files/`.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, a probe of six cases gives the same 
results as official PySpark 4.3.0:
      - the errors for a text file without a flag, an `s3:` URI and two flags;
      - a UDF that imports a `pyfile=True` module, reads a `file=True` file 
with `SparkFiles`, and reads from a `#alias` archive.
    
      Before this change, four of them differed.
    - New unit tests check the names and errors of `parse_artifact` for each 
kind, fragments, queries, `file:` URIs and relative paths.
    - A new mock-server test checks the name of each upload.
    - A new offline test checks the errors.
    - A new live test checks the UDF reads, and it also passes with official 
PySpark 4.3.0.
    - The e2e coverage test of artifacts passes the `file` flag for its text 
files.
    - The lib tests and every integration test, e2e included, against the local 
server (643 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (615 passed, 8 live cases skipped), and the live drop-in and Arrow 
suites (58 passed) and live cases (10 passed). Clippy reports no diagnostics in 
the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #176 from HyukjinKwon/SPARK-60956-artifacts.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs           |  59 +++++++--
 crates/spark-connect-core/src/artifact.rs  | 204 +++++++++++++++++++++++++++++
 crates/spark-connect-core/src/client.rs    |  53 ++------
 crates/spark-connect/src/session.rs        |  35 ++++-
 crates/spark-connect/tests/e2e_coverage.rs |  14 +-
 python/tests/test_arrow_handoff.py         |  23 +++-
 python/tests/test_dropin_e2e.py            |  32 +++++
 python/tests/test_dropin_offline.py        |  19 +++
 8 files changed, 374 insertions(+), 65 deletions(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 7a4d00c..4f13fcb 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1156,9 +1156,10 @@ impl PySparkSession {
         self.session.clear_tags()
     }
 
-    // Mirrors reference `SparkSession.addArtifacts(*path, pyfile=False, 
archive=False,
-    // file=False)`. The pyfile/archive/file flags are accepted for signature 
parity;
-    // artifact classification is handled by the Rust transport / server.
+    /// Add local files, or `file:` URIs, as artifacts of the session, as 
PySpark's
+    /// `addArtifacts(*path, pyfile=False, archive=False, file=False)`: each 
path is named by
+    /// the flags and its extension, and the errors for a path that is not 
supported are
+    /// PySpark's (see `spark_connect_core::artifact::parse_artifact`).
     #[pyo3(name = "addArtifacts", signature = (*path, pyfile=false, 
archive=false, file=false))]
     fn add_artifacts(
         &self,
@@ -1168,11 +1169,53 @@ impl PySparkSession {
         archive: bool,
         file: bool,
     ) -> PyResult<()> {
-        let _ = (pyfile, archive, file);
+        use spark_connect_core::artifact::{parse_artifact, 
UnsupportedArtifact};
+
+        if [pyfile, archive, file].iter().filter(|flag| **flag).count() > 1 {
+            return Err(crate::column::raise_pyspark(
+                py,
+                "PySparkValueError",
+                "INVALID_MULTIPLE_ARGUMENT_CONDITIONS",
+                &[
+                    ("arg_names", "'pyfile', 'archive' and/or 'file'"),
+                    ("condition", "True together"),
+                ],
+            ));
+        }
+        for p in &path {
+            if let Err(unsupported) = parse_artifact(p, pyfile, archive, file) 
{
+                return Err(match unsupported {
+                    UnsupportedArtifact::Format(local) => 
crate::column::raise_pyspark(
+                        py,
+                        "PySparkRuntimeError",
+                        "UNSUPPORTED_OPERATION",
+                        &[("operation", &format!("{local} file format"))],
+                    ),
+                    UnsupportedArtifact::Scheme(scheme) => 
crate::column::raise_pyspark(
+                        py,
+                        "PySparkRuntimeError",
+                        "UNSUPPORTED_OPERATION",
+                        &[("operation", &format!("{scheme} scheme"))],
+                    ),
+                    UnsupportedArtifact::HashInArchivePath => 
crate::column::raise_pyspark(
+                        py,
+                        "PySparkValueError",
+                        "VALUE_ALLOWED",
+                        &[("arg_name", "artifact path"), ("disallowed_value", 
"#")],
+                    ),
+                });
+            }
+        }
         let refs: Vec<&str> = path.iter().map(|s| s.as_str()).collect();
-        py.detach(|| self.session.add_artifacts(&refs)).to_pyerr()
+        py.detach(|| {
+            self.session
+                .add_artifacts_with(&refs, pyfile, archive, file)
+                .map_err(Box::new)
+        })
+        .to_pyerr()
     }
 
+    /// `addArtifact` is `addArtifacts`, as in PySpark.
     #[pyo3(name = "addArtifact", signature = (*path, pyfile=false, 
archive=false, file=false))]
     fn add_artifact(
         &self,
@@ -1182,11 +1225,7 @@ impl PySparkSession {
         archive: bool,
         file: bool,
     ) -> PyResult<()> {
-        // pyfile/archive/file are accepted for PySpark signature parity; 
artifact
-        // classification is handled by the Rust transport / server.
-        let _ = (pyfile, archive, file);
-        let refs: Vec<&str> = path.iter().map(|s| s.as_str()).collect();
-        py.detach(|| self.session.add_artifacts(&refs)).to_pyerr()
+        self.add_artifacts(py, path, pyfile, archive, file)
     }
 
     #[pyo3(name = "newSession")]
diff --git a/crates/spark-connect-core/src/artifact.rs 
b/crates/spark-connect-core/src/artifact.rs
index ace2bba..d254571 100644
--- a/crates/spark-connect-core/src/artifact.rs
+++ b/crates/spark-connect-core/src/artifact.rs
@@ -305,8 +305,212 @@ fn build_chunked_artifact_requests(
     Ok(requests)
 }
 
+/// A path of `addArtifacts` that PySpark's `ArtifactManager` does not upload.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum UnsupportedArtifact {
+    /// A file of a format that is not uploaded with the flags given, by its 
local path.
+    Format(String),
+    /// A URI of a scheme other than `file`.
+    Scheme(String),
+    /// The local path of an archive with a fragment has a `#`, which the 
fragment would
+    /// be taken from.
+    HashInArchivePath,
+}
+
+impl std::fmt::Display for UnsupportedArtifact {
+    /// The message of PySpark's error for the path.
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            UnsupportedArtifact::Format(path) => {
+                write!(
+                    f,
+                    "[UNSUPPORTED_OPERATION] {path} file format is not 
supported."
+                )
+            }
+            UnsupportedArtifact::Scheme(scheme) => {
+                write!(
+                    f,
+                    "[UNSUPPORTED_OPERATION] {scheme} scheme is not supported."
+                )
+            }
+            UnsupportedArtifact::HashInArchivePath => write!(
+                f,
+                "[VALUE_ALLOWED] Value for `artifact path` does not allow #."
+            ),
+        }
+    }
+}
+
+/// `%XX` escapes decoded, as `url2pathname` does on POSIX.
+fn unquote(text: &str) -> String {
+    let bytes = text.as_bytes();
+    let mut out = Vec::with_capacity(bytes.len());
+    let mut i = 0;
+    while i < bytes.len() {
+        let hex = |b: u8| (b as char).to_digit(16);
+        if bytes[i] == b'%' && i + 2 < bytes.len() {
+            if let (Some(high), Some(low)) = (hex(bytes[i + 1]), hex(bytes[i + 
2])) {
+                out.push((high * 16 + low) as u8);
+                i += 3;
+                continue;
+            }
+        }
+        out.push(bytes[i]);
+        i += 1;
+    }
+    String::from_utf8_lossy(&out).into_owned()
+}
+
+/// The name that a path of `addArtifacts` is uploaded as, and the local file 
to upload, as
+/// PySpark's `ArtifactManager._parse_artifacts` parses a local path or a 
`file:` URI:
+///
+/// - With `pyfile`, a `.py`, `.egg`, `.zip` or `.jar` is `pyfiles/<name>`.
+/// - With `archive`, a `.zip`, `.jar`, `.tar.gz`, `.tgz` or `.tar` is 
`archives/<name>`, or
+///   `archives/<name>#<fragment>` for a path with a fragment, which names the 
directory
+///   that the archive is extracted to.
+/// - With `file`, any file is `files/<name>`.
+/// - Otherwise a `.class` is `classes/<name>`, a `.jar` is `jars/<name>`, and 
any other file
+///   is not supported.
+pub fn parse_artifact(
+    path_or_uri: &str,
+    pyfile: bool,
+    archive: bool,
+    file: bool,
+) -> std::result::Result<(String, std::path::PathBuf), UnsupportedArtifact> {
+    // As `urlparse`: a scheme of a letter and then letters, digits, `+`, `-` 
or `.` before
+    // `:`, a netloc after `//`, and the fragment after `#` and the query 
after `?`.
+    let (scheme, rest) = match path_or_uri.find(':') {
+        Some(i)
+            if path_or_uri[..i].starts_with(|c: char| c.is_ascii_alphabetic())
+                && path_or_uri[..i]
+                    .chars()
+                    .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c)) =>
+        {
+            (path_or_uri[..i].to_ascii_lowercase(), &path_or_uri[i + 1..])
+        }
+        _ => (String::new(), path_or_uri),
+    };
+    let rest = match rest.strip_prefix("//") {
+        Some(after) => &after[after.find(['/', '?', 
'#']).unwrap_or(after.len())..],
+        None => rest,
+    };
+    let (rest, fragment) = rest.split_once('#').unwrap_or((rest, ""));
+    let path = rest.split_once('?').map_or(rest, |(path, _)| path);
+    let local = match scheme.as_str() {
+        "" => {
+            let path = std::path::PathBuf::from(unquote(path));
+            if path.is_absolute() {
+                path
+            } else {
+                std::env::current_dir().unwrap_or_default().join(path)
+            }
+        }
+        "file" => std::path::PathBuf::from(unquote(path)),
+        _ => return Err(UnsupportedArtifact::Scheme(scheme)),
+    };
+    let local_path = local.to_string_lossy().into_owned();
+    let name = local
+        .file_name()
+        .map(|name| name.to_string_lossy().into_owned())
+        .unwrap_or_default();
+    let ends = |suffixes: &[&str]| suffixes.iter().any(|suffix| 
name.ends_with(suffix));
+    let artifact = if pyfile && ends(&[".py", ".egg", ".zip", ".jar"]) {
+        format!("pyfiles/{name}")
+    } else if archive && ends(&[".zip", ".jar", ".tar.gz", ".tgz", ".tar"]) {
+        if fragment.is_empty() {
+            format!("archives/{name}")
+        } else if local_path.contains('#') {
+            return Err(UnsupportedArtifact::HashInArchivePath);
+        } else {
+            format!("archives/{name}#{fragment}")
+        }
+    } else if file {
+        format!("files/{name}")
+    } else if ends(&[".class"]) {
+        format!("classes/{name}")
+    } else if ends(&[".jar"]) {
+        format!("jars/{name}")
+    } else {
+        return Err(UnsupportedArtifact::Format(local_path));
+    };
+    Ok((artifact, local))
+}
+
 #[cfg(test)]
 mod tests {
+    use super::{parse_artifact, UnsupportedArtifact};
+
+    #[test]
+    fn artifacts_are_named_as_pyspark_parses_their_paths() {
+        use UnsupportedArtifact::*;
+        let named = |name: &str| Ok(name.to_string());
+        let cases = [
+            ("/d/a.jar", (false, false, false), named("jars/a.jar")),
+            (
+                "/d/A.class",
+                (false, false, false),
+                named("classes/A.class"),
+            ),
+            ("/d/m.py", (true, false, false), named("pyfiles/m.py")),
+            ("/d/m.zip", (true, false, false), named("pyfiles/m.zip")),
+            (
+                "/d/b.tar.gz",
+                (false, true, false),
+                named("archives/b.tar.gz"),
+            ),
+            // The fragment of an archive names the directory that it is 
extracted to.
+            (
+                "/d/b.zip#alias",
+                (false, true, false),
+                named("archives/b.zip#alias"),
+            ),
+            ("/d/n.txt", (false, false, true), named("files/n.txt")),
+            // Another path drops its fragment, and its query, as `urlparse` 
takes them.
+            ("/d/a.jar#x", (false, false, false), named("jars/a.jar")),
+            (
+                "file:/d/a.jar?q=1",
+                (false, false, false),
+                named("jars/a.jar"),
+            ),
+            (
+                "file:///d/a%20b.jar",
+                (false, false, false),
+                named("jars/a b.jar"),
+            ),
+            (
+                "/d/m.py",
+                (false, false, false),
+                Err(Format("/d/m.py".to_string())),
+            ),
+            (
+                "/d/m.py",
+                (false, true, false),
+                Err(Format("/d/m.py".to_string())),
+            ),
+            (
+                "s3://bucket/a.jar",
+                (false, false, false),
+                Err(Scheme("s3".to_string())),
+            ),
+            (
+                "/d/b%23c.zip#alias",
+                (false, true, false),
+                Err(HashInArchivePath),
+            ),
+        ];
+        for (path, (pyfile, archive, file), expected) in cases {
+            let parsed = parse_artifact(path, pyfile, archive, 
file).map(|(name, _)| name);
+            assert_eq!(parsed, expected, "{path}");
+        }
+    }
+
+    #[test]
+    fn a_relative_artifact_path_is_in_the_current_directory() {
+        let (name, local) = parse_artifact("a.jar", false, false, 
false).unwrap();
+        assert_eq!(name, "jars/a.jar");
+        assert_eq!(local, std::env::current_dir().unwrap().join("a.jar"));
+    }
+
     use super::*;
 
     #[test]
diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index 60fc68b..347c7a3 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -812,54 +812,17 @@ impl SparkConnectClient {
     pub async fn add_artifacts(
         &self,
         paths: &[&str],
-        _pyfile: bool,
-        _archive: bool,
-        _file: bool,
+        pyfile: bool,
+        archive: bool,
+        file: bool,
     ) -> Result<()> {
-        // Build list of named artifacts from file paths
+        // Each path is named as PySpark's ArtifactManager names it, by the 
flags and its file
+        // extension (see `parse_artifact`).
         let mut artifacts = Vec::new();
         for path in paths {
-            // Extract just the filename as the artifact name
-            let path_obj = std::path::Path::new(path);
-            let file_name = path_obj
-                .file_name()
-                .and_then(|n| n.to_str())
-                .ok_or_else(|| {
-                    SparkError::connect_msg(format!("Invalid artifact path: 
{}", path))
-                })?;
-
-            // Determine artifact prefix based on file extension
-            let artifact_name = if _pyfile
-                && (file_name.ends_with(".py")
-                    || file_name.ends_with(".zip")
-                    || file_name.ends_with(".egg")
-                    || file_name.ends_with(".jar"))
-            {
-                format!("pyfiles/{}", file_name)
-            } else if _archive
-                && (file_name.ends_with(".zip")
-                    || file_name.ends_with(".jar")
-                    || file_name.ends_with(".tar.gz")
-                    || file_name.ends_with(".tgz")
-                    || file_name.ends_with(".tar"))
-            {
-                format!("archives/{}", file_name)
-            } else if _file {
-                format!("files/{}", file_name)
-            } else if file_name.ends_with(".jar") {
-                format!("jars/{}", file_name)
-            } else {
-                return Err(SparkError::connect_msg(format!(
-                    "Unsupported artifact type: {}",
-                    path
-                )));
-            };
-
-            // Create artifact from file
-            artifacts.push(NamedArtifact::new(
-                artifact_name,
-                Box::new(FileArtifact::new(path)),
-            ));
+            let (name, local) = crate::artifact::parse_artifact(path, pyfile, 
archive, file)
+                .map_err(|e| SparkError::connect_msg(e.to_string()))?;
+            artifacts.push(NamedArtifact::new(name, 
Box::new(FileArtifact::new(local))));
         }
 
         if artifacts.is_empty() {
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 000e9d2..67e8870 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -569,18 +569,43 @@ impl SparkSession {
         block_on(self.client.interrupt_operation(operation_id))
     }
 
-    /// Add local files as artifacts to the session (e.g. `.py`, `.jar`, 
`.zip`).
-    ///
-    /// Mirrors `pyspark.sql.SparkSession.addArtifacts(*path)`.
+    /// Add local files, or `file:` URIs, as artifacts of the session, 
mirroring
+    /// `pyspark.sql.SparkSession.addArtifacts(*path)`: a `.jar` is added to 
the classpath and
+    /// a `.class` to the classes; any other file is not supported without a 
flag of
+    /// [`add_artifacts_with`](Self::add_artifacts_with).
     pub fn add_artifacts(&self, paths: &[&str]) -> Result<()> {
-        block_on(self.client.add_artifacts(paths, false, false, true))
+        self.add_artifacts_with(paths, false, false, false)
+    }
+
+    /// Add local files as artifacts of the session, mirroring
+    /// `pyspark.sql.SparkSession.addArtifacts(*path, pyfile, archive, file)`:
+    /// - `pyfile`: a `.py`, `.egg`, `.zip` or `.jar` is importable by Python 
UDFs;
+    /// - `archive`: a `.zip`, `.jar`, `.tar.gz`, `.tgz` or `.tar` is 
extracted, into the
+    ///   directory that a `#<name>` fragment of its path names;
+    /// - `file`: any file is added to the files of the session.
+    ///
+    /// At most one of the flags may be true.
+    pub fn add_artifacts_with(
+        &self,
+        paths: &[&str],
+        pyfile: bool,
+        archive: bool,
+        file: bool,
+    ) -> Result<()> {
+        if [pyfile, archive, file].iter().filter(|flag| **flag).count() > 1 {
+            return Err(SparkError::connect_msg(
+                "[INVALID_MULTIPLE_ARGUMENT_CONDITIONS] ['pyfile', 'archive' 
and/or 'file'] \
+                 cannot be True together.",
+            ));
+        }
+        block_on(self.client.add_artifacts(paths, pyfile, archive, file))
     }
 
     /// Add a single local file as an artifact to the session.
     ///
     /// Mirrors `pyspark.sql.SparkSession.addArtifact(path)`.
     pub fn add_artifact(&self, path: &str) -> Result<()> {
-        block_on(self.client.add_artifacts(&[path], false, false, true))
+        self.add_artifacts(&[path])
     }
 
     /// Copy a local file to the driver's filesystem at `dest_path`.
diff --git a/crates/spark-connect/tests/e2e_coverage.rs 
b/crates/spark-connect/tests/e2e_coverage.rs
index 801449d..fd4a2be 100644
--- a/crates/spark-connect/tests/e2e_coverage.rs
+++ b/crates/spark-connect/tests/e2e_coverage.rs
@@ -1030,9 +1030,17 @@ fn artifact_and_tag_surface() {
     // A larger file to exercise the multi-chunk path (well over one gRPC 
chunk).
     std::fs::write(&f2, vec![b'x'; 512 * 1024]).unwrap();
 
-    s.add_artifact(f1.to_str().unwrap()).expect("add_artifact");
-    s.add_artifacts(&[f1.to_str().unwrap(), f2.to_str().unwrap()])
-        .expect("add_artifacts");
+    // As PySpark, a file that is neither a jar nor a class needs the `file` 
flag.
+    assert!(s.add_artifact(f1.to_str().unwrap()).is_err());
+    s.add_artifacts_with(&[f1.to_str().unwrap()], false, false, true)
+        .expect("add_artifact");
+    s.add_artifacts_with(
+        &[f1.to_str().unwrap(), f2.to_str().unwrap()],
+        false,
+        false,
+        true,
+    )
+    .expect("add_artifacts");
     // copyFromLocalToFs: exercises the client-side named-artifact upload
     // (forward_to_fs/...). A single-node server rejects a *local* 
destination, but the
     // client upload code we're covering runs before that, so tolerate the 
server error.
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index 459a65e..3f0f4b7 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -464,7 +464,8 @@ def _arrow_input_service(
         session_id = None
 
         def save(name, data):
-            if not (allow_file_artifacts and name.startswith(("files/", 
"forward_to_fs/"))):
+            named = ("files/", "forward_to_fs/", "jars/", "classes/", 
"pyfiles/", "archives/")
+            if not (allow_file_artifacts and name.startswith(named)):
                 assert name == "cache/" + hashlib.sha256(data).hexdigest()
             assert name not in state["cache"], "cached artifacts should not be 
uploaded again"
             state["cache"][name] = bytes(data)
@@ -649,7 +650,7 @@ def test_file_upload_attribution_on_every_frame(
         protocol, user_id=user_id, user_agent=user_agent, 
allow_file_artifacts=True
     ) as (spark, state):
         if upload == "file":
-            spark.addArtifacts(str(small), str(large))
+            spark.addArtifacts(str(small), str(large), file=True)
             artifact_names = ["files/small.txt", "files/large.txt"]
         else:
             destinations = ["/wire/small.txt", "/wire/large.txt"]
@@ -889,3 +890,21 @@ def test_non_arrow_input_does_not_import_pyarrow(protocol, 
monkeypatch, data):
         spark.createDataFrame(data)
         assert imported == []
         assert state["config_requests"] == []
+
+
+def test_add_artifacts_names_each_artifact_as_pyspark(protocol, tmp_path):
+    for name in ["a.jar", "A.class", "m.py", "b.zip", "n.txt"]:
+        (tmp_path / name).write_bytes(name.encode())
+    with _arrow_input_service(protocol, allow_file_artifacts=True) as (spark, 
state):
+        # As PySpark's ArtifactManager names them, by the flags and the file 
extension.
+        spark.addArtifacts(str(tmp_path / "a.jar"), str(tmp_path / "A.class"))
+        spark.addArtifacts(str(tmp_path / "m.py"), pyfile=True)
+        spark.addArtifacts(str(tmp_path / "b.zip") + "#alias", archive=True)
+        spark.addArtifact(str(tmp_path / "n.txt"), file=True)
+        assert state["cache"] == {
+            "jars/a.jar": b"a.jar",
+            "classes/A.class": b"A.class",
+            "pyfiles/m.py": b"m.py",
+            "archives/b.zip#alias": b"b.zip",
+            "files/n.txt": b"n.txt",
+        }
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 9734e93..4adf970 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -241,6 +241,38 @@ def test_create_dataframe_from_arrow_table(session, 
schema, expected):
     assert _rows(df) == ARROW_TABLE_ROWS
 
 
+def test_add_artifacts_reach_udfs_as_pyspark(session, tmp_path):
+    import uuid
+    import zipfile
+
+    tag = uuid.uuid4().hex[:8]
+    module, note, bundle = f"mod_{tag}", f"note_{tag}.txt", f"bundle_{tag}.zip"
+    (tmp_path / f"{module}.py").write_text("def f():\n    return 'from the 
module'\n")
+    (tmp_path / note).write_text("file content")
+    with zipfile.ZipFile(tmp_path / bundle, "w") as archive:
+        archive.writestr("inner.txt", "inside")
+    session.addArtifacts(str(tmp_path / f"{module}.py"), pyfile=True)
+    session.addArtifacts(str(tmp_path / note), file=True)
+    session.addArtifacts(str(tmp_path / bundle) + f"#alias_{tag}", 
archive=True)
+
+    @F.udf("string")
+    def read(_):
+        import importlib
+        import os
+
+        imported = importlib.import_module(module)
+        # The directory of the session's files, which holds the module. Not 
SparkFiles: a
+        # worker started with SPARK_SKIP_CONNECT_COMPAT_TESTS, as in CI, does 
not set it up.
+        root = os.path.dirname(imported.__file__)
+        return "|".join([
+            imported.f(),
+            open(os.path.join(root, note)).read(),
+            open(os.path.join(root, f"alias_{tag}", "inner.txt")).read(),
+        ])
+
+    assert session.range(1).select(read("id")).first()[0] == "from the 
module|file content|inside"
+
+
 def test_create_dataframe_from_empty_arrow_table(session):
     empty = pa.table({"a": pa.array([], pa.int64()), "b": pa.array([], 
pa.string())})
     df = session.createDataFrame(empty, "a long, b string")
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index df2d96f..2c0a9d0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -558,6 +558,25 @@ def test_pandas_udf():
     assert callable(p) or p is not None
 
 
+def test_add_artifacts_raises_for_paths_that_pyspark_does_not_upload(tmp_path):
+    from pyspark.errors import PySparkRuntimeError, PySparkValueError
+
+    spark = _offline_session()
+    note = tmp_path / "note.txt"
+    note.write_text("x")
+    # Raised before any upload, as in PySpark.
+    for path, operation in [(str(note), f"{note} file format"), 
("s3://b/x.jar", "s3 scheme")]:
+        with pytest.raises(PySparkRuntimeError) as error:
+            spark.addArtifacts(path)
+        assert error.value.getCondition() == "UNSUPPORTED_OPERATION"
+        assert error.value.getMessageParameters() == {"operation": operation}
+    with pytest.raises(PySparkValueError) as error:
+        spark.addArtifacts(str(note), pyfile=True, file=True)
+    assert error.value.getCondition() == "INVALID_MULTIPLE_ARGUMENT_CONDITIONS"
+    assert error.value.getMessageParameters() == {
+        "arg_names": "'pyfile', 'archive' and/or 'file'", "condition": "True 
together"}
+
+
 def test_dropin_module_imports():
     # The thin re-export shims import cleanly.
     import pyspark.sql.dataframe  # noqa: F401


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to