byronwang123 opened a new issue, #7635: URL: https://github.com/apache/kyuubi/issues/7635
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the feature Arrow-based query results are currently transferred as uncompressed Arrow IPC bytes. The result fetch dominates query time on bandwidth-limited links (60-97% of total on a cross-subnet cluster), so compressing the IPC body would cut both bytes and wall-clock. This carries on #3877 (deferred in 2022 waiting for an upstream approach); Spark 4.1.0 has since added spark.sql.execution.arrow.compression.codec / .zstd.level, so the upstream-aligned way now exists. ### Motivation On TPC-DS SF10 (nntest cluster): cs_numeric 1.17GB -> 247MB, fetch 19.3s -> 7.5s; cs_full 4.67GB -> 850MB, fetch 205.8s -> 94.1s. ### Describe the solution Reuse the upstream Spark config keys (getConfString on Spark < 4.1), compress the Arrow IPC body with zstd (default none, zero behavior change), and let the JDBC client decompress via CompressionCodec.Factory from the batch body compression metadata -- no Kyuubi-specific wire protocol or config key. ### Additional context _No response_ ### Are you willing to submit PR? - [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve. - [ ] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
