XiaoHongbo-Hope opened a new pull request, #11278: URL: https://github.com/apache/paimon/pull/11278
### Purpose CPU-count-based split concurrency leaves filtered native BLOB reads waiting on object-storage I/O. Use `min(split count, 32)` automatically for Data Evolution reads that project BLOB payloads, have a filter and no LIMIT. Explicit `parallelism` and the existing `read.parallelism` table option still take precedence. Other reads and Python fallback retain their defaults. Ordering, LIMIT handling and Core's total BLOB concurrency cap are unchanged. Random 1,000-image reads from a 94,513-image (~35 GB) copy, four warmed repetitions per cell (median seconds): | vCPU | CPU-count default | Split 16 | Split 32 | |---:|---:|---:|---:| | 2 | 2.82 | 3.05 | 1.82 | | 4 | 2.41 | 2.56 | 1.10 | | 8 | 2.50 | 2.40 | 0.78 | | 16 | 2.30 | 2.30 | 0.68 | Benchmarks used Python `df565c64` + #11269, Rust `11fe259e`, BLOB concurrency 16 and CRC enabled. These are query latencies including planning and Python conversion, not training throughput. The table has at most 32 splits; this does not establish a universal optimum. Ordered-group scheduling is a separate optimization. ### Tests - Native dispatch and Python reader tests: **157 passed**; covers both Arrow APIs, explicit/table overrides, projection, LIMIT, descriptor reads and fallback. - flake8 and `git diff --check` passed. - 288 matrix measurements plus 48 ordering diagnostics validated exact keys and image SHA256. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
