matanper opened a new issue, #68891: URL: https://github.com/apache/doris/issues/68891
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description `ArrowStreamReader` reads Stream Load bodies with `arrow::ipc::IpcReadOptions::Defaults()` (`be/src/format/arrow/arrow_stream_reader.cpp`, master too), and that leaves `use_threads = true`. For compressed IPC streams (ZSTD/LZ4 record batches), Arrow then decompresses every record batch's buffers on Arrow's single process-wide CPU thread pool, not on the scanner thread doing the read. Under many concurrent Arrow loads, scanner threads mostly wait on that one pool's mutex and condition variable. Off-CPU stacks show around 75 scanner threads per BE parked in `pthread_cond_signal` / `arrow::internal::ThreadPool::SpawnReal`, while the BE stays well below CPU saturation. ### Measurements 4 BEs (192 vCPU each, arm64), cloud mode, many concurrent Stream Loads, ZSTD-compressed Arrow record batches: - default `IpcReadOptions` (global pool): plateau of about 5.7M rows/s total, BEs at about 64% CPU; - the same data sent uncompressed: about 7.1M rows/s; - ZSTD, with decompression inline on the scanner thread (`use_threads = false`): 8.15–8.45M rows/s. ### Solution Read Stream Load bodies with `use_threads = false` (optionally behind a BE config), so decompression runs on the scanner thread that already owns the load. Load concurrency comes from the scanner pool, so Arrow's internal pool adds only contention. `arrow_flight_batch_reader.cpp` uses the same pattern and may benefit too. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
