Zouxxyy opened a new pull request, #11078: URL: https://github.com/apache/paimon/pull/11078
### Purpose LeRobot imports retain standard metadata but omit additional source text documents. Add `extra_dataset_fields`, mapping new `__datasets` column names to relative file paths or globs in every source. A file becomes `STRING`; a glob becomes `MAP<STRING, STRING>`, including when it matches one file. Keys are relative to the static directory prefix before the glob, without the final suffix: `config/model_a.json` matched by `config/*.json` is accessed as `config['model_a']`. Preserve UTF-8 text verbatim for JSON, TXT, Markdown, YAML/YML, JSONL/NDJSON, and CSV/TSV. Validate file matches, suffixes, encoding, source containment, column names, and duplicate keys before creating target tables. Training readers project only required source metadata, skipping extra text columns. Rename `dataset_names` to `dataset_name`: accept a scalar label for one source or an ordered sequence for multiple sources. Default to automatic source FPS handling; existing explicit `dataset` and `source` modes remain available. Names or extra fields also create `__datasets` for a single source. Documentation and fixtures use generic `config` and `notes` fields. ### Tests - Six LeRobot test modules: 366 passed, 134 subtests passed, 9 skipped, and 1 deselected. Coverage includes serial/Ray imports, source selection, mixed FPS, all supported text suffixes, and local/FileIO/Hub text resolution (remote backends use test doubles). - Eight skips require the native reader runtime; one requires a working TorchCodec runtime. - The deselected `LeRobotValidationTest.test_video_batches_include_delta_frames_and_preserve_backends` fails four decoder subtests both on this branch and independently on unchanged master (`8f6f39ff17ff8c56d01e087bc33114e9b6926a1e`). This PR leaves that existing video decoding issue outside its scope. - Repository-configured flake8 and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
