XiaoHongbo-Hope opened a new pull request, #9486: URL: https://github.com/apache/paimon/pull/9486
### Purpose Avoid materializing every non-streaming Torch row as Python objects. For row-tracking tables, the map-style dataset keeps a compact row-ID index and implements batched `__getitems__`. Each DataLoader batch is read through row-ID ranges, preserving requested order and duplicate indices. Predicates or deletion vectors build only the row-ID index eagerly; unfiltered snapshot reads derive it from split metadata without opening data files. Tables without row tracking retain an Arrow table instead of calling `to_pylist()` for the full result. ### Tests - Lazy construction from row-tracking split metadata - Batched random access, duplicates, negative indices and multi-worker shuffle - Projection with `_ROW_ID` - Predicate and limit semantics - Deletion vectors - Existing BLOB and Torch tests `paimon-python/pypaimon/tests/torch_read_test.py`: 48 passed License, flake8, Python 3.6 syntax compilation and `git diff --check` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
