JingsongLi commented on code in PR #10405:
URL: https://github.com/apache/paimon/pull/10405#discussion_r4203048987
##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -703,6 +725,30 @@ def __repr__(self):
self.reader.__class__.__name__, self.__class__.__name__, 1)
+class _LeRobotDataLoaderAdapter:
+
+ def __init__(self, dataset, share_memory):
+ self.dataset = dataset
+ self.share_memory = share_memory
+
+ def __len__(self):
+ return len(self.dataset)
+
+ def __getitem__(self, index):
+ return self.dataset[index]
+
+ def __getitems__(self, indices):
+ return self.dataset.reader._get_batch(indices, self.share_memory)
+
+ def __getattr__(self, name):
+ if name.startswith("__") and name.endswith("__"):
+ raise AttributeError(name)
+ dataset = self.__dict__.get("dataset")
+ if dataset is None:
+ raise AttributeError(name)
+ return getattr(dataset, name)
Review Comment:
[P2] Forward public dataset property assignments to the wrapped dataset
This adapter forwards attribute reads, but assignments in `worker_init_fn`,
such as `get_worker_info().dataset.image_transforms = transform` or
`get_worker_info().dataset.return_uint8 = True`, only create attributes on the
adapter. The reader keeps its original configuration, so preprocessing changes
are silently ignored when switching from `DataLoader(dataset, ...)` to
`dataset.to_dataloader(...)`.
I reproduced this with a real `PaimonDatasetReader` and
`multiprocessing_context="spawn"`: setting `return_uint8=True` produces `uint8`
pixels with value 128 through the ordinary loader, but the new loader still
produces normalized `float32` pixels with value approximately 0.502. Assigning
`image_transforms` in the worker is also ignored.
Please forward the setters for these two public properties to the wrapped
dataset and cover these assignments with a `worker_init_fn` regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
