XiaoHongbo-Hope opened a new pull request, #9365: URL: https://github.com/apache/paimon/pull/9365
### Purpose PyPaimon Torch streaming currently converts every row to a Python dictionary before PyTorch batches it. This adds an opt-in batch-first path so training pipelines can consume Arrow `RecordBatch` objects or dictionaries of Torch tensors without per-row Python conversion. - Keep `batch_format="row"` as the fully compatible default. - Add streaming `batch_format="pyarrow"` and `batch_format="torch"` modes. - Preserve native reader batches by default, or combine/slice them with an optional `batch_size`. - Convert non-null numeric, boolean, and numeric fixed-size-list columns by default, with `to_tensor_fn` for strings, BLOBs, nulls, and application-specific conversion. - Retain split sharding and bounded threaded prefetch; use `DataLoader(batch_size=None)` to avoid a second batching step. - Document the batch-first API and buffer-sharing semantics. ### Tests - `PYTHONPATH=paimon-python pytest paimon-python/pypaimon/tests/torch_read_test.py -q` (21 passed) - Project-configured flake8 on the changed Python files - Python 3.6 `py_compile` on production files - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
