JingsongLi commented on code in PR #11082:
URL: https://github.com/apache/paimon/pull/11082#discussion_r4561541575


##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -1228,8 +1229,17 @@ def _merged_source_stats(sources, episodes_table):
               for name in names if name not in _CONTROLS}
     columns = _target_schema(episodes_table).names
     if "stats" in columns:
-        stats_rows = _read_arrow(episodes_table, 
["stats"])["stats"].to_pylist()
-        decoded_rows = [json.loads(raw) for raw in stats_rows if raw]
+        legacy_rows = None
+        if any(not source.get("read_control_stats") for source in sources):
+            legacy_rows = _read_arrow(episodes_table, ["dataset_index", 
"stats"]).to_pylist()
+        decoded_rows = []
+        for source in sources:
+            raw = source.get("read_control_stats")
+            if raw:
+                decoded_rows.append(json.loads(raw))

Review Comment:
   [P2] Distinguish internal statistics from existing user text columns
   
   `read_control_stats` was a valid public `extra_dataset_fields` name before 
this change, but the reader now interprets any column with that name as 
internal statistics. Using the exact parent writer, I imported a source with 
`extra_dataset_fields={'read_control_stats': 'notes.txt'}` containing ordinary 
text; the parent can read the resulting group, while this head's 
`PaimonLeRobotDataset` raises `JSONDecodeError` here. The same import with 
field name `notes` remains readable. This breaks the stated existing-group 
compatibility even without appending or changing any files. Use distinguishable 
internal metadata/versioning (and preserve historical user columns), rather 
than selecting this representation solely by an unreserved column name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to