HyukjinKwon opened a new pull request, #253: URL: https://github.com/apache/spark-connect-rust/pull/253
### What changes were proposed in this pull request? `createDataFrame` reads the server's local-relation settings for a relation of at least 32 MiB of Arrow data, whichever path encoded it, and then sends it as the Arrow importer does: inline below `spark.sql.session.localRelationCacheThreshold`, and uploaded as cache artifacts at it, as PySpark does. This covers local rows (`SparkSession::create_dataframe`), the pandas row conversion, `create_dataframe_from_columns`, and the Arrow IPC that the float64 and string pandas fast paths (#207, #208) encode (`create_dataframe_from_inline_arrow_ipc`, which now returns a `Result`). Smaller relations are still sent inline without the configuration RPC. ### Why are the changes needed? [SPARK-61062](https://issues.apache.org/jira/browse/SPARK-61062): these paths sent every relation inline, so one beyond the server's gRPC message size limit (128 MiB by default) failed. Against a local Spark 4.3.0 server: | Input | Current master | This change | PySpark 4.3.0 | | --- | --- | --- | --- | | pandas, 15,000 strings of 10 KB (about 150 MB) | `h2 protocol error` | 15,000 rows, 2.8 s | 15,000 rows, 2.4 s | | pandas, 100,000 rows of 200 `float64` columns (160 MB) | `h2 protocol error` | the same sum as PySpark | the same sum | | local rows, 80,000 rows of a 2 KB string and a long (170 MB) | `h2 protocol error` | the same count and sum, 2.7 s | the same, 3.5 s | ### Does this PR introduce _any_ user-facing change? Yes. Large local relations are uploaded at the cache threshold instead of failing, as in PySpark. ### How was this patch tested? - New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow input service: relations of local rows, of the pandas row conversion, and of the float64 and string fast paths above 32 MiB read the cache threshold and are uploaded at it, with all their rows; a small relation stays inline without reading it. - The float64 and string fast-path tests (301 passed), and the lib tests of `apache-spark-connect` (461 passed). - The reproductions above against a local Spark 4.3.0 server. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
