Akash3121 commented on code in PR #11082:
URL: https://github.com/apache/paimon/pull/11082#discussion_r4531386068
##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -523,24 +526,29 @@ def _merged_episodes(plans, schema):
for key, value in original.items():
if key.startswith("stats/"):
feature, statistic = key[6:].rsplit("/", 1)
- # Rebased control values have new statistics. Their
- # original quantiles cannot describe the merged IDs.
- if feature not in _CONTROLS or not
statistic.startswith("q"):
- stats.setdefault(feature, {})[statistic] = value
+ stats.setdefault(feature, {})[statistic] = value
row[_DATASET_INDEX_COLUMN] = plan[_DATASET_INDEX_COLUMN]
- episode = row["episode_index"]
- if "subtask_index" not in plan["control_stats"][episode]:
- if "subtask_index" in stats:
- stats["subtask_index"] = {
- metric: None for metric in stats["subtask_index"]}
- for name, values in plan["control_stats"][episode].items():
- stats[name] = values
row["stats"] = json.dumps(
_json_value(stats), ensure_ascii=False, separators=(",",
":"))
records.append(row)
yield pa.Table.from_pylist(records, schema=schema)
+def _store_read_control_stats(metadata, plans):
+ rows = metadata["datasets_table"].to_pylist()
+ for plan, row in zip(plans, rows):
+ stats = {}
+ for name in _CONTROLS:
+ parts = [values[name] for values in plan["control_stats"].values()
+ if name in values]
Review Comment:
Could we make the `episode_index` entry use the combined reader’s
coordinates before storing these statistics? `_transform_frames` intentionally
keeps frame `episode_index` values source-local, while
`_SourceLocalReader.read_indices` adds each source’s Episode offset. With the
existing two-source fixture, the reader returns `[0,0,1,1,1,2,2,3,3,3]`, but
`meta.stats["episode_index"]["mean"]` is `0.6` instead of `1.6` (and its
min/max/std are likewise source-local). Computing the Episode statistic column
with `plan["episode_offset"]` while leaving the stored frame column unchanged
fixes both local and Ray imports. Could we also update the regression assertion
to compare against the reader-visible Episode IDs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]