JingsongLi commented on code in PR #11082:
URL: https://github.com/apache/paimon/pull/11082#discussion_r4561541575
##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -1228,8 +1229,17 @@ def _merged_source_stats(sources, episodes_table):
for name in names if name not in _CONTROLS}
columns = _target_schema(episodes_table).names
if "stats" in columns:
- stats_rows = _read_arrow(episodes_table,
["stats"])["stats"].to_pylist()
- decoded_rows = [json.loads(raw) for raw in stats_rows if raw]
+ legacy_rows = None
+ if any(not source.get("read_control_stats") for source in sources):
+ legacy_rows = _read_arrow(episodes_table, ["dataset_index",
"stats"]).to_pylist()
+ decoded_rows = []
+ for source in sources:
+ raw = source.get("read_control_stats")
+ if raw:
+ decoded_rows.append(json.loads(raw))
Review Comment:
[P2] Distinguish internal statistics from existing user text columns
`read_control_stats` was a valid public `extra_dataset_fields` name before
this change, but the reader now interprets any column with that name as
internal statistics. Using the exact parent writer, I imported a source with
`extra_dataset_fields={'read_control_stats': 'notes.txt'}` containing ordinary
text; the parent can read the resulting group, while this head's
`PaimonLeRobotDataset` raises `JSONDecodeError` here. The same import with
field name `notes` remains readable. This breaks the stated existing-group
compatibility even without appending or changing any files. Use distinguishable
internal metadata/versioning (and preserve historical user columns), rather
than selecting this representation solely by an unreserved column name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]