Gabriel39 opened a new pull request, #70089:
URL: https://github.com/apache/doris/pull/70089
### What problem does this PR solve?
Operation-owned Lance scan and decoder tasks use SDK runtimes independently
of Doris scanner execution. This PR adds optional host executor injection to
the pinned Lance Rust SDK and Lance-C bindings, and uses it from
`LanceTableReader`.
The versioned C ABI provides thread-bound executor scopes, readiness
callbacks, host-driven queues and lifetime callbacks. Doris keeps one executor
per reader and cooperatively runs its child tasks on the calling scanner thread
while waiting. This avoids submitting children to the same saturated scanner
pool and waiting for them. Exported Arrow stream callbacks use the same
cooperative wait path.
`enable_lance_host_executor` defaults to `false`. Tokio still handles I/O
readiness, timers, join notifications and shared background services. This is
the first synchronous reader integration; it does not provide complete
scheduling of third-party operators, Rust allocation accounting, query memory
enforcement, or preemption of running CPU closures. Single-split CPU
parallelism can decrease, so enabling it requires workload measurements.
The source patches preserve the existing Foyer and prewarm APIs. Patch
fingerprints refresh stale C/Rust source caches, and the normal build entry
point refreshes an older installed Lance-C ABI. Design and lifecycle notes are
in `be/docs/lance-host-executor.md`.
### Validation
- Passed 88 Lance-C unit tests, including host execution, callback lifetime
and existing stream panic/cancellation tests.
- Passed the new C API integration test covering scans, exported Arrow
streams, filters, take, vector search, full-text search and early close.
- Passed 15 focused scheduling/lifetime tests against the actual executor
modules in a standalone harness.
- Built the native SDK and passed C++ ABI smoke tests for both scanner-next
and exported Arrow streams: 4 fixture rows and 17 host polls each.
- Passed `cargo check --locked --offline --all-targets` from freshly patched
sources and `cargo clippy --lib -- -D warnings`.
- Passed fresh extraction, idempotence, Foyer-only upgrade, stale C/Rust
fingerprint and failed-patch marker tests.
- Compiled the changed Doris reader, configuration and BE test translation
units with the ASAN build configuration; passed clang-format 16 and shell
syntax checks.
- Full BE unit-test execution is pending: the local third-party installation
lacks `libgoogle_cloud_cpp_oauth2.a`, so the monolithic test binary could not
be built.
### Release note
Adds an opt-in cooperative host executor for Lance reads, disabled by
default.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (native C++ scanner and Arrow stream ABI smoke tests)
- Behavior changed:
- [x] Yes. When explicitly enabled, operation-owned Lance tasks run
cooperatively on the scanner caller.
- Does this need documentation?
- [x] Yes. Developer design, lifecycle and validation notes are included.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
