dongjoon-hyun opened a new pull request, #597: URL: https://github.com/apache/spark-connect-swift/pull/597
### What changes were proposed in this pull request? This PR aims to align the gRPC message size limit and the Arrow result handling with the Scala and Python Spark Connect clients. 1. Set the default `grpc_max_message_size` to 128 MiB (`SparkConnectClient.DEFAULT_GRPC_MAX_MESSAGE_SIZE`). `SparkConnectClient.maxMessageSize` is no longer optional, and the `ServiceConfig` always applies the limit to both request and response messages. 2. Send `ResultChunkingOptions(allow_arrow_batch_chunking = true)` in every `ExecutePlanRequest` so that the server can split large Arrow batches into chunks. 3. Assemble the chunks in `DataFrame.execute()` using `chunk_index` and `num_chunks_in_batch` before parsing the Arrow IPC stream. An out-of-order chunk fails with `InvalidArrowData`. 4. Count `row_count` only for the first chunk (`chunk_index == 0`) in `executeAndCount()`, because the server sets the same `row_count` on all chunks of a batch. 5. Add the `ResultChunkingOptions` type alias. `preferred_arrow_chunk_size` is not exposed in this PR. ### Why are the changes needed? Without `grpc_max_message_size`, the client is limited to the gRPC default of 4 MiB, so a response larger than 4 MiB fails with `Received message larger than max`. The Scala client and the Python client both default to 128 MiB. Python has used 128 MiB since [SPARK-42816](https://github.com/apache/spark/commit/da19e0de05b) (2023-03-21, v3.5.0). Even with a larger limit, a single Arrow batch can exceed it. Result chunking, enabled by default in the Scala and Python clients, lets the server split such batches. ### Does this PR introduce _any_ user-facing change? Yes. - When `grpc_max_message_size` is not specified, the limit changes from 4 MiB to 128 MiB. Responses larger than 4 MiB now succeed instead of failing. - Large Arrow batches are now received in chunks and transparently assembled. ### How was this patch tested? - Updated `parameterWithoutGrpcMaxMessageSize` to expect the 128 MiB default in `maxMessageSize` and in the `ServiceConfig`. - Added `resultChunking` to `SparkConnectClientTests` to verify the request option. - Confirmed that `swift build` and the test target compile. - The integration tests were not run locally because no Spark Connect server was available. They are expected to run in CI. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
