JingsongLi opened a new pull request, #10960:
URL: https://github.com/apache/paimon/pull/10960

   ### Purpose
   
   Search queries currently require callers to collect row IDs and build a 
separate pinned scan before training. Add `to_torch()` and 
`to_contiguous_window_dataset()` to vector, full-text, hybrid, and batch vector 
queries on data-evolution tables.
   
   Search, post-filtering, and lazy training reads share one captured snapshot. 
Search filters and top-k select window anchors while preserving their history 
and future frames; `context_filter` independently restricts the context. Batch 
queries retain separate datasets in input-query order. Keep the captured 
snapshot metadata so checkpointing and resuming work when a tag retains an 
expired snapshot.
   
   Training conversions preserve BLOB descriptors/lazy payload reads and 
existing stride, padding, transforms, and adapters. Score output and score 
ordering are explicitly unsupported. Document the APIs and add regression 
coverage.
   
   ### Tests
   
   - Targeted regression suite: **267 passed, 9 skipped, 121 subtests passed** 
across search training, contiguous windows/planning/resume, search 
snapshots/metadata, Torch reads, full-text filtering, and query authorization.
   - After installing the optional TorchData dependency, reran search training 
and window resume: **64 passed**, including retained-tag checkpoint recovery 
and spawned DataLoader workers.
   - Ray dispatch and snapshot propagation are covered with mocked search 
scheduling; no live Ray cluster was run.
   - `flake8` on all changed Python files and `git diff --check` passed.
   
   ```shell
   cd paimon-python
   python -m pytest -q \
     pypaimon/tests/search_training_test.py \
     pypaimon/tests/contiguous_window_dataset_test.py \
     pypaimon/tests/contiguous_window_plan_test.py \
     pypaimon/tests/contiguous_window_resume_test.py \
     pypaimon/tests/search_snapshot_test.py \
     pypaimon/tests/search_result_metadata_test.py \
     pypaimon/tests/torch_read_test.py \
     pypaimon/tests/full_text_scalar_filter_test.py \
     pypaimon/tests/search_query_auth_reject_test.py
   
   python -m pytest -q \
     pypaimon/tests/contiguous_window_resume_test.py \
     pypaimon/tests/search_training_test.py -ra
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to