dongjoon-hyun opened a new pull request, #473: URL: https://github.com/apache/spark-connect-swift/pull/473
### What changes were proposed in this pull request? This PR supports large local data in `SparkSession.createDataFrame` via `CachedLocalRelation`. When the serialized Arrow IPC stream is equal to or larger than `spark.sql.session.localRelationCacheThreshold` (1MiB by default), the serialized `LocalRelation` is uploaded to the server as a `cache/<sha256>` artifact and the plan references it with a `CachedLocalRelation` message, instead of inlining the data into the plan. This follows the PySpark client behavior. `CachedLocalRelation` (supported since Spark 3.5) is used instead of the newer `ChunkedCachedLocalRelation` (added in Spark 4.1.0 by SPARK-53917) in order to support all Spark Connect servers this library targets, including 4.0.x. ### Why are the changes needed? Previously, `createDataFrame` inlined the data into the plan, so it was limited by the gRPC message size limit and threw `LocalRelationTooLarge` for larger data. With this PR, larger data is transparently uploaded via the artifact API (SPARK-58528) and re-uploading identical data is skipped based on its SHA-256 hash. ### Does this PR introduce _any_ user-facing change? Yes. `createDataFrame` now succeeds for data whose serialized size exceeds the gRPC message size limit. The API signature is unchanged. ### How was this patch tested? Pass the CIs with a new test case (`CreateDataFrameTests.largeData`) which creates a DataFrame with 10,000 rows (~2MiB serialized, above the default 1MiB threshold), verifies `count`/`sum`/`filter` round trips against a live Spark Connect server, and verifies that a second `createDataFrame` call with the same data skips the upload. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
