dongjoon-hyun opened a new pull request, #472: URL: https://github.com/apache/spark-connect-swift/pull/472
### What changes were proposed in this pull request? This PR aims to support cache artifact upload in `SparkConnectClient` by adding a new file, `SparkConnectClient+Artifact.swift`, with the following internal APIs. - `artifactExists`: checks whether an artifact exists in the server-side session via the `ArtifactStatus` RPC. - `cacheArtifact`: caches the given data as a `cache/<sha256>` artifact and returns its SHA-256 hash. The upload is skipped if the artifact already exists. Data up to 32KiB is uploaded as a single-chunk batch, and larger data as a chunked artifact stream, like PySpark's `ArtifactManager.cache_artifact`. The existing `addArtifact` is moved to the new file to group all artifact operations together. ### Why are the changes needed? This is an infrastructure for supporting large `createDataFrame` data via `CachedLocalRelation`. ### Does this PR introduce _any_ user-facing change? No, these are internal APIs. ### How was this patch tested? Pass the CIs with a new test suite, `SparkConnectClientArtifactTests`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
