XiaoHongbo-Hope opened a new pull request, #11074: URL: https://github.com/apache/paimon/pull/11074
Merged LeRobot imports currently keep each source's original task, subtask, stats, and Episode metadata in Arrow IPC binary columns. These columns cannot be inspected as JSON in SQL. This change writes JSON payloads and source schemas instead, while keeping common training columns typed and preserving source-specific extension fields for future export. New table groups use `tasks_json`, `subtasks_json`, `stats_json`, and `episodes_schema_json` in `__datasets`, plus `source_episode_json` in `__episodes`. The training reader excludes these archive fields and still reads existing table groups with IPC columns. This PR is independent of #11070. Validation: 144 LeRobot aggregate tests passed; flake8 and `git diff --check` passed. Two customer sample sources (25/30 FPS) imported into a local table group, and their task, subtask, and Episode Parquet data round-tripped through JSON with schemas and metadata intact. The updated reader also opened a table group written by the prior IPC layout. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
