JingsongLi opened a new pull request, #10960:
URL: https://github.com/apache/paimon/pull/10960
### Purpose
Search queries currently require callers to collect row IDs and build a
separate pinned scan before training. Add `to_torch()` and
`to_contiguous_window_dataset()` to vector, full-text, hybrid, and batch vector
queries on data-evolution tables.
Search, post-filtering, and lazy training reads share one captured snapshot.
Search filters and top-k select window anchors while preserving their history
and future frames; `context_filter` independently restricts the context. Batch
queries retain separate datasets in input-query order. Keep the captured
snapshot metadata so checkpointing and resuming work when a tag retains an
expired snapshot.
Training conversions preserve BLOB descriptors/lazy payload reads and
existing stride, padding, transforms, and adapters. Score output and score
ordering are explicitly unsupported. Document the APIs and add regression
coverage.
### Tests
- Targeted regression suite: **267 passed, 9 skipped, 121 subtests passed**
across search training, contiguous windows/planning/resume, search
snapshots/metadata, Torch reads, full-text filtering, and query authorization.
- After installing the optional TorchData dependency, reran search training
and window resume: **64 passed**, including retained-tag checkpoint recovery
and spawned DataLoader workers.
- Ray dispatch and snapshot propagation are covered with mocked search
scheduling; no live Ray cluster was run.
- `flake8` on all changed Python files and `git diff --check` passed.
```shell
cd paimon-python
python -m pytest -q \
pypaimon/tests/search_training_test.py \
pypaimon/tests/contiguous_window_dataset_test.py \
pypaimon/tests/contiguous_window_plan_test.py \
pypaimon/tests/contiguous_window_resume_test.py \
pypaimon/tests/search_snapshot_test.py \
pypaimon/tests/search_result_metadata_test.py \
pypaimon/tests/torch_read_test.py \
pypaimon/tests/full_text_scalar_filter_test.py \
pypaimon/tests/search_query_auth_reject_test.py
python -m pytest -q \
pypaimon/tests/contiguous_window_resume_test.py \
pypaimon/tests/search_training_test.py -ra
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]