Akash3121 commented on code in PR #11082:
URL: https://github.com/apache/paimon/pull/11082#discussion_r4531386068


##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -523,24 +526,29 @@ def _merged_episodes(plans, schema):
                 for key, value in original.items():
                     if key.startswith("stats/"):
                         feature, statistic = key[6:].rsplit("/", 1)
-                        # Rebased control values have new statistics. Their
-                        # original quantiles cannot describe the merged IDs.
-                        if feature not in _CONTROLS or not 
statistic.startswith("q"):
-                            stats.setdefault(feature, {})[statistic] = value
+                        stats.setdefault(feature, {})[statistic] = value
                 row[_DATASET_INDEX_COLUMN] = plan[_DATASET_INDEX_COLUMN]
-                episode = row["episode_index"]
-                if "subtask_index" not in plan["control_stats"][episode]:
-                    if "subtask_index" in stats:
-                        stats["subtask_index"] = {
-                            metric: None for metric in stats["subtask_index"]}
-                for name, values in plan["control_stats"][episode].items():
-                    stats[name] = values
                 row["stats"] = json.dumps(
                     _json_value(stats), ensure_ascii=False, separators=(",", 
":"))
                 records.append(row)
             yield pa.Table.from_pylist(records, schema=schema)
 
 
+def _store_read_control_stats(metadata, plans):
+    rows = metadata["datasets_table"].to_pylist()
+    for plan, row in zip(plans, rows):
+        stats = {}
+        for name in _CONTROLS:
+            parts = [values[name] for values in plan["control_stats"].values()
+                     if name in values]

Review Comment:
   Could we make the `episode_index` entry use the combined reader’s 
coordinates before storing these statistics? `_transform_frames` intentionally 
keeps frame `episode_index` values source-local, while 
`_SourceLocalReader.read_indices` adds each source’s Episode offset. With the 
existing two-source fixture, the reader returns `[0,0,1,1,1,2,2,3,3,3]`, but 
`meta.stats["episode_index"]["mean"]` is `0.6` instead of `1.6` (and its 
min/max/std are likewise source-local). Computing the Episode statistic column 
with `plan["episode_offset"]` while leaving the stored frame column unchanged 
fixes both local and Ray imports. Could we also update the regression assertion 
to compare against the reader-visible Episode IDs?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to