XiaoHongbo-Hope opened a new pull request, #9536:
URL: https://github.com/apache/paimon/pull/9536
### Purpose
Multimodal training data is often stored as independently sampled streams. A
caller currently has to collect those streams and implement timestamp
matching,
episode isolation, snapshot pinning, and payload lookup itself.
This draft proposes a typed, bounded temporal-alignment API for
`MultimodalTable.scan()`:
```python
steps = align(
actions.scan().select(["episode_id", "event_time", "action"]),
on="event_time",
by="episode_id",
camera=nearest(images.scan().select("image"), tolerance=20),
state=backward(states.scan().select("state"), tolerance=50),
)
for batch in steps.to_arrow_batch_reader(batch_size=128):
train(batch)
```
The implementation:
- requires an explicit group boundary so matches never cross episodes/clips;
- supports exact, backward, forward, and nearest matching with inclusive
tolerance and deterministic earlier-frame tie breaking;
- pins every input scan to its current snapshot;
- plans only group keys, timestamps, and `_ROW_ID`, then fetches selected
payload rows in batches;
- keeps BLOB values as descriptors so existing image/video readers can fetch
or
decode only selected samples;
- emits validity, matched timestamp, and signed delta columns for auditing;
- exposes `ScanQuery.to_arrow_batch_reader()` as the high-level streaming
entry
point.
This is deliberately a bounded MVP. It does not add interpolation, training
windows, watermark/streaming semantics, or a customer-specific JSON spec. The
metadata index is currently coordinator-local. Since this is a draft,
feedback
on the API shape and whether the alignment executor belongs in PyPaimon is
especially welcome.
### Tests
```text
python -m pytest \
pypaimon/tests/multimodal_temporal_test.py \
pypaimon/tests/multimodal_table_test.py -q
81 passed
```
```text
python -m flake8 \
pypaimon/multimodal/temporal.py \
pypaimon/tests/multimodal_temporal_test.py
```
A broader local run reached 4306 passed / 102 skipped. Remaining failures
were
in suites requiring unavailable Vortex, Mosaic, DuckDB, Java golden data, or
existing Ray/Pandas dtype expectations; no temporal or multimodal test
failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
