Rich-T-kid commented on PR #10044: URL: https://github.com/apache/arrow-rs/pull/10044#issuecomment-4623736500
i'm suspecting the issue has to do with ` let (client, server) = tokio::io::duplex(1024 * 1024);` per the [docs](https://docs.rs/tokio/latest/tokio/io/fn.duplex.html) "_The max_buf_size argument is the maximum amount of bytes that can be written to a side before the write returns Poll::Pending_." The two regression cases both involve large variable-length data where the encoded payload can exceed 1MB: roundtrip/variable/8192x8 — 8 columns × 8192 rows ~1.96MB roundtrip/variable/65536x1 — 65536 rows, large values buffer This also shows up in the regression cases, `roundtrip/variable/8192x8 1.00 1251.7±29.66µs 1798.6 MB/sec 1.53 1914.9±122.75µs 1175.7 MB/sec` `roundtrip/variable/65536x1 1.00 1285.3±51.72µs 1750.7 MB/sec 1.49 1910.5±128.29µs 1177.8 MB/sec` throughput falls flat. taking a look at the other benchmark results this seems consistant, `roundtrip/fixed/65536x8 1.00 2.2±0.03ms 1821.4 MB/sec 1.19 2.6±0.16ms 1532.6 MB/sec` throughput shrinks and as such causes more blocking to happen. Even in the event where this isn't the reason for the slow down I think 1MB is still to small for realistic max throughput. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
