Zouxxyy opened a new pull request, #11078:
URL: https://github.com/apache/paimon/pull/11078

   ### Purpose
   
   LeRobot imports retain standard metadata but omit additional source text 
documents. Add `extra_dataset_fields`, mapping new `__datasets` column names to 
relative file paths or globs in every source. A file becomes `STRING`; a glob 
becomes `MAP<STRING, STRING>`, including when it matches one file. Keys are 
relative to the static directory prefix before the glob, without the final 
suffix: `config/model_a.json` matched by `config/*.json` is accessed as 
`config['model_a']`.
   
   Preserve UTF-8 text verbatim for JSON, TXT, Markdown, YAML/YML, 
JSONL/NDJSON, and CSV/TSV. Validate file matches, suffixes, encoding, source 
containment, column names, and duplicate keys before creating target tables. 
Training readers project only required source metadata, skipping extra text 
columns.
   
   Rename `dataset_names` to `dataset_name`: accept a scalar label for one 
source or an ordered sequence for multiple sources. Default to automatic source 
FPS handling; existing explicit `dataset` and `source` modes remain available. 
Names or extra fields also create `__datasets` for a single source. 
Documentation and fixtures use generic `config` and `notes` fields.
   
   ### Tests
   
   - Six LeRobot test modules: 366 passed, 134 subtests passed, 9 skipped, and 
1 deselected. Coverage includes serial/Ray imports, source selection, mixed 
FPS, all supported text suffixes, and local/FileIO/Hub text resolution (remote 
backends use test doubles).
   - Eight skips require the native reader runtime; one requires a working 
TorchCodec runtime.
   - The deselected 
`LeRobotValidationTest.test_video_batches_include_delta_frames_and_preserve_backends`
 fails four decoder subtests both on this branch and independently on unchanged 
master (`8f6f39ff17ff8c56d01e087bc33114e9b6926a1e`). This PR leaves that 
existing video decoding issue outside its scope.
   - Repository-configured flake8 and `git diff --check` passed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to