YannByron opened a new pull request, #9580: URL: https://github.com/apache/paimon/pull/9580
## Summary Add a generic, snapshot-pinned `ContiguousWindowDataset` for exposing fixed-size Paimon row windows to PyTorch without loading projected rows or BLOB payloads into memory up front. This is the contiguous-window portion split from #9466 following review feedback. The ACT benchmark workflow will be submitted separately. ## Changes - Add row-ID-backed window indexing with deterministic group/order semantics, configurable stride, and drop/pad/error tail policies. - Read projected values lazily from the pinned snapshot, including scalar and `MAP<..., BLOB>` payloads and anchor-only columns. - Coalesce overlapping plural reads while keeping logical samples independent for transforms and adapters. - Expose the Dataset through `ScanQuery.to_contiguous_window_dataset` and document its PyTorch usage and contracts. - Cover snapshot pinning, BLOB handling, batching, mutable values, padding, pickling, and multi-worker DataLoader behavior. ## Testing - [x] `PYTHONPATH=. /opt/homebrew/bin/python3.13 -m pytest pypaimon/tests/contiguous_window_dataset_test.py -q` (16 passed) - [x] `PYTHONPATH=. /opt/homebrew/bin/python3.13 -m pytest pypaimon/tests/multimodal_table_test.py -k 'scan_read_blobs or scan_read_and_stream_map_blobs' -q` (6 passed, 74 deselected) - [x] `/opt/homebrew/bin/python3.13 -m flake8 --config=dev/cfg.ini pypaimon/multimodal/query.py pypaimon/multimodal/window_dataset.py pypaimon/tests/contiguous_window_dataset_test.py` - [x] `python3 dev/check_license_header.py` - [x] `/opt/homebrew/bin/python3.13 -m compileall -q pypaimon/multimodal` - [x] `git diff --check origin/master..HEAD` ## Notes No ACT benchmark implementation, packaging, tests, or benchmark documentation is included in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
