Gabriel39 opened a new pull request, #68775:
URL: https://github.com/apache/doris/pull/68775
### What problem does this PR solve?
Related issue: #68692 (M2); parent tracker: #66340.
Related PR: #68698 (merged dependency baseline).
Lance index reads currently enter the BE cache during foreground queries.
This adds synchronous SQL to load a named logical index into the same shared
Lance session before queries run:
```sql
WARM UP INDEX vector_idx ON lake_catalog.demo.items;
WARM UP INDEX vector_idx ON lake_catalog.demo.items WITH COMPUTE GROUP
search_group;
```
The FE checks ADMIN and table SELECT privileges, plus compute-group USAGE in
cloud mode. It resolves the normal catalog access path (including vended
credentials), pins one dataset version and eligible BE set, and dispatches at
most eight RPCs at a time. Success requires every target to acknowledge the
pinned snapshot. The result contains table, index, version, backend count, and
elapsed milliseconds.
The BE uses the existing bounded heavy-work pool and allows one concurrent
prewarm operation per shared session. It opens the pinned dataset through
`LanceSessionManager` and delegates to the upstream synchronous C API. Missing
indexes, storage failures, unavailable or older BEs, and missing
acknowledgements fail explicitly; provider error details and storage
credentials are excluded from prewarm errors.
The statement observes the SQL timeout and KILL/connection cancellation
while waiting. Cancelling a caller does not guarantee immediate interruption of
running SDK IO or undo successful cache fills. Cached entries remain evictable.
Explicit compute-group selection is supported in cloud mode; otherwise the
current resource-tag execution context is used.
### Release note
Add `WARM UP INDEX ... ON ... [WITH COMPUTE GROUP ...]` for synchronous
Lance index prewarm on the eligible query backends.
### Validation
- Passed 12 focused FE tests covering parsing, permissions, default/explicit
compute groups, snapshot/index resolution, target selection, bounded fan-out,
acknowledgements, errors, timeout, and cancellation.
- FE Checkstyle: passed with zero violations.
- clang-format 16: passed for all five affected C++ source/header files.
- Affected C++ translation units passed syntax compilation against the
pinned Lance-C headers.
- Real pinned Lance-C SDK smoke test passed: scalar-index prewarm, cache
hits through a separately opened shared-session handle, and historical-snapshot
prewarm after dropping the current index.
- Added BE manager tests for invalid inputs, sanitized errors,
shared-session cache reuse, and fixed snapshots. Full BE unit-test execution
was blocked locally because the available third-party bundle is missing
`libsimdutf.a`; these tests are not reported as passing.
- Added external regression coverage for indexed query equivalence,
synchronous results, retry, explicit cloud groups, permission denial, and
one-target failure injection. Groovy syntax validation passed; the external
suite has not been run against a cluster locally.
M3's supported-format/catalog matrix, broader lifecycle/performance
verification, and bilingual website documentation remain tracked in #68692.
This PR does not close that issue.
### Check List (For Author)
- Test
- [x] Regression test (added; cluster execution pending)
- [x] Unit Test (FE passed; BE execution blocked as described above)
- [x] Manual test (SDK smoke test described above)
- Behavior changed:
- [x] Yes: new synchronous administrative SQL command.
- Does this need documentation?
- [x] Yes: tracked separately in #68692 M3.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
