JingsongLi commented on code in PR #11056:
URL: https://github.com/apache/paimon/pull/11056#discussion_r4522563375


##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -1002,9 +1096,30 @@ def _episode_dataset(table):
 
     projection = [
         name for name in _target_schema(table).names
-        if not name.startswith("stats/")
+        if not name.startswith("stats/") and name != "source_episode_ipc"
     ]
-    data = _read_arrow(table, projection).sort_by("episode_index")
+    data = _read_arrow(table, projection)
+    if datasets is not None:
+        data = data.sort_by([
+            ("dataset_index", "ascending"),
+            ("episode_index", "ascending"),
+        ])
+        expected = [
+            (dataset["dataset_index"], local)
+            for dataset in datasets
+            for local in range(dataset["episode_to_index"]
+                               - dataset["episode_from_index"])
+        ]
+        actual = list(zip(data["dataset_index"].to_pylist(),
+                          data["episode_index"].to_pylist()))
+        if actual != expected:
+            raise ValueError("LeRobot source Episode keys are not contiguous.")
+        data = data.set_column(
+            data.schema.get_field_index("episode_index"),
+            data.schema.field("episode_index"),

Review Comment:
   **[P1] Type the in-memory Episode ordinal independently of the source ID**
   
   I reproduced this with two otherwise standard five-frame FileIO sources, 
each using `int32` for `meta/episodes` `episode_index` and `int64` for the 
Frames control column. Single-source import and training work. The merged 
import commits, but constructing `PaimonLeRobotDataset` raises `ArrowInvalid: 
Field type did not match data type` here. `set_column` receives the stored 
`int32` Field and the new `int64` ordinal array. Could the in-memory field use 
`int64` independently of the persisted source ID, with a test that opens and 
reads the second source?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to