HyukjinKwon opened a new pull request, #261: URL: https://github.com/apache/spark-connect-rust/pull/261
### What changes were proposed in this pull request? - Artifact uploads stream their chunks: `ArtifactFrames` (`crates/spark-connect-core/src/artifact.rs`) reads one 32 KiB chunk per frame as the frames are sent, with the same framing as before (a batch for small artifacts, `BeginChunk` and `Chunk` frames otherwise), and each attempt of the retry policy reads the file again from the start (`upload_artifact` in `client.rs`), instead of building every frame up front and cloning them for each attempt. - `addArtifacts` of the Python client raises PySpark's `PySparkRuntimeError [UNSUPPORTED_OPERATION] ... file format is not supported.` for a class file, opens local paths as PySpark does (so a missing path or a directory raises `FileNotFoundError` or `IsADirectoryError` with its text), and puts a pyfile zip, egg or jar on `sys.path` and invalidates the import caches, as PySpark does (`crates/pyspark-rs/src/session.rs`). The Rust API still uploads class files, as its docs say. ### Why are the changes needed? [SPARK-61070](https://issues.apache.org/jira/browse/SPARK-61070): measured as the peak RSS of `addArtifacts` of a 300 MB jar to an in-process server: 496 MB on the current master, 141 to 174 MB with this change, 173 MB with PySpark 4.3.0. And `spark.addArtifacts("A.class")` uploaded the file where PySpark raises. ### Does this PR introduce _any_ user-facing change? Yes. Uploads take less memory, and the Python client's `addArtifacts` raises PySpark's errors for class files and for missing paths. ### How was this patch tested? - `crates/spark-connect-core/tests/artifact_uploads.rs`, against a loopback AddArtifacts service: chunked and batched frames, class file names, a retried upload sent again from the start, CRC mismatches and permission errors not retried, and a missing file failing before any attempt; and unit tests of the frames (lazy reads, framing, read failures, the same frames on each attempt). - New offline tests in `python/tests/test_dropin_offline.py` for the errors of `addArtifacts` and the `sys.path` order; `test_add_artifacts_names_each_artifact_as_pyspark` uploads its class file with `file=True`, as PySpark requires. - 112 combinations of paths (jar, py, zip, egg, tar.gz, fragments, `%23`, `file:` URIs, a directory, missing paths, remote schemes, an empty path) and flags give the same outcome, exception and message as PySpark 4.3.0's `_parse_artifacts`. - The core tests and the build of every integration test of `apache-spark-connect`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
