XiaoHongbo-Hope opened a new pull request, #11278:
URL: https://github.com/apache/paimon/pull/11278

   ### Purpose
   
   CPU-count-based split concurrency leaves filtered native BLOB reads waiting 
on object-storage I/O. Use `min(split count, 32)` automatically for Data 
Evolution reads that project BLOB payloads, have a filter and no LIMIT.
   
   Explicit `parallelism` and the existing `read.parallelism` table option 
still take precedence. Other reads and Python fallback retain their defaults. 
Ordering, LIMIT handling and Core's total BLOB concurrency cap are unchanged.
   
   Random 1,000-image reads from a 94,513-image (~35 GB) copy, four warmed 
repetitions per cell (median seconds):
   
   | vCPU | CPU-count default | Split 16 | Split 32 |
   |---:|---:|---:|---:|
   | 2 | 2.82 | 3.05 | 1.82 |
   | 4 | 2.41 | 2.56 | 1.10 |
   | 8 | 2.50 | 2.40 | 0.78 |
   | 16 | 2.30 | 2.30 | 0.68 |
   
   Benchmarks used Python `df565c64` + #11269, Rust `11fe259e`, BLOB 
concurrency 16 and CRC enabled. These are query latencies including planning 
and Python conversion, not training throughput. The table has at most 32 
splits; this does not establish a universal optimum. Ordered-group scheduling 
is a separate optimization.
   
   ### Tests
   
   - Native dispatch and Python reader tests: **157 passed**; covers both Arrow 
APIs, explicit/table overrides, projection, LIMIT, descriptor reads and 
fallback.
   - flake8 and `git diff --check` passed.
   - 288 matrix measurements plus 48 ordering diagnostics validated exact keys 
and image SHA256.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to