yihua opened a new issue, #889: URL: https://github.com/apache/hudi-rs/issues/889
### Feature Description `Table::read` reads its file slices with `try_join_all` on the task that awaits it. The reads overlap their I/O, but all Parquet decoding runs on that one thread, and the Python binding blocks on that task from the Python thread, so a full-table read uses a single core however many file slices it has. The default `hoodie.read.file.slice.read.concurrency` of 4 also caps a `Table` read at four slices, although unlike a DataFusion partition it is the whole scan. Proposed: read each file slice on its own runtime task, keeping input order and the scan memory budget, and default a `Table` read's concurrency to the available cores when the config is unset. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
