XiaoHongbo-Hope opened a new pull request, #9446:
URL: https://github.com/apache/paimon/pull/9446
### Purpose
Add a one-time PyPaimon importer:
`LeRobot Dataset v3 -> Arrow batches -> Paimon table`
This follows `load_from_hdf5()` and reuses the existing connection, table
creation, strict append validation, BLOB, batch write, and snapshot commit
paths. It does not add dynamic training conversion, Dataset Version, Ray, Torch
DataLoader, or Paimon-to-LeRobot export.
Public API:
```python
result = conn.load_from_lerobot(
"robot_data",
"/path/to/lerobot_dataset",
batch_size=1024,
)
```
- Local LeRobot v3 directories are supported first; a non-path string may be
a Hugging Face `repo_id`.
- A missing table is created from `meta/info.json`; an existing table uses
the same strict schema validation and append behavior as HDF5 import.
- Each frame becomes one Paimon row. Batches never cross Episode boundaries,
and one call produces one commit/snapshot.
- Standard feature names and index fields are preserved; `feature_mapping`
and `transform` are optional.
Feature mapping:
| LeRobot feature | Paimon/Arrow representation |
| --- | --- |
| Numeric or string scalar | Corresponding scalar type |
| One-dimensional numeric tensor | Fixed-length vector |
| Higher-rank numeric tensor | Nested array with a fixed-length innermost
vector |
| Image or video frame | BLOB |
Images retain their compressed source bytes. Video is decoded through the
official LeRobot API and each frame is stored once as an independent PNG BLOB,
so the complete MP4 is not copied into every row.
LeRobot remains optional through `pypaimon[lerobot]`. The first version
supports LeRobot Dataset v3 and rejects v2.x, `uint64`, depth video, and
unsupported language event structures with explicit errors.
### Tests
- Official LeRobot API creates a two-Episode v3 dataset entirely in a
temporary directory, without network access.
- Covers metadata-derived schema, scalar/vector/action/observation/timestamp
values, Episode and frame order, task text, row count, snapshot, strict append
validation, missing optional dependency, and image/video BLOB round trips.
- `multimodal_lerobot_test.py`: 7 passed
- `multimodal_hdf5_test.py`: 24 passed, 30 subtests passed
- `multimodal_table_test.py`: 67 passed
- `./paimon-python/dev/lint-python.sh -i license,flake8`: passed
- Python 3.6 syntax compilation for the shared modules: passed
- `git diff --check`: passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
