steFaiz commented on PR #8064:
URL: https://github.com/apache/paimon/pull/8064#issuecomment-4601514705
I introduced 3-layers shuffle in pytorch integration. including:
1. File meta layer chunk-shuffle. This relies on random-access-optimized
data format.
2. Interleaving several chunks
3. A buffer for shuffle
The usage is simple:
```python
from torch.utils.data import DataLoader
seed = 42
# do chunk-shuffle in planning. This is optional
table_scan = read_builder.new_scan().with_chunk_shuffle(
seed=seed,
chunk_size=1000,
)
table_read = read_builder.new_read()
splits = table_scan.plan().splits()
dataset = table_read.to_torch(
splits,
streaming=True,
shuffle=True,
seed=seed,
# buffer shuffle
buffer_size=1000,
# interleave splits
max_buffer_input_splits=10,
)
dataloader = DataLoader(
dataset,
batch_size=32,
num_workers=2,
shuffle=False,
)
```
I refer to the HuggingFace Iterable Dataset:
https://github.com/huggingface/datasets/blob/main/src/datasets/iterable_dataset.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
