u70b3 opened a new pull request, #68668: URL: https://github.com/apache/doris/pull/68668
### What problem does this PR solve? Issue Number: #66497 Related PR: #67978 (PR3D — this PR stacks on its head; only the last 8 commits belong to this PR) Problem Summary: mutations dispatched by PR3D currently land on a NOT_IMPLEMENTED stub. This PR delivers the execution side of slice 4 — the first of the four §9.7 enablement prerequisites, a **verified hard worker memory/PID/runtime boundary** — plus the full dispatch → isolated execution → typed report chain for CREATE/REPLACE/DROP of IVF_PQ indexes on object storage. Design: v5.1 ([comment 5301314544](https://github.com/apache/doris/issues/66497#issuecomment-5301314544)); roadmap: [comment 5595163744](https://github.com/apache/doris/issues/66497#issuecomment-5595163744). The `enable_lance_index_mutation` gate stays off. What lands: 1. **Termination contracts (IDL + FE, additive)** — `reportLanceIndexJobTermination` RPC and `TLanceIndexTerminationProof.NEVER_LAUNCHED = 3` for invocations without a trusted result code (kill / wall-clock timeout / OOM / panic), complementing PR3D's FE-internal `NEVER_ENQUEUED` for proven-never-enqueued dispatches. Building on PR3D's `NOT_ENQUEUED` classification, dispatch payload-bound violations (the only FE-local failures that can still occur after PR3D moved dispatch preparation before the durable RUNNING mark — a violated bound never heals, so staying PENDING would retry forever) now converge `NOT_COMMITTED` with the internal `NEVER_LAUNCHED` slot release in **one durable transition**. Two-tier taxonomy: `NOT_ENQUEUED` (PR3D, FE-proven never reached the backend) vs `NEVER_LAUNCHED` (this PR, wire value 3, BE-supervisor-proven worker never exec'd; also used by the BE for post-enqueue pre-fork rejections). Dispatch fields 17/18 carry the admission-time snapshot of the mut able `lance_index_max_num_partitions` / `lance_index_max_num_sub_vectors` bounds; the worker replays exactly the bounds admission enforced and safely rejects a dispatch that lacks them. 2. **Worker execution library** (`be/src/lance/index_worker*.cpp`) — length-capped pipe protocol (dispatch ≤512 KiB / handshake ≤4 KiB / result ≤8 KiB, cap before allocation, thrift-compact decode depth ≤16), pinned dataset open (never version 0) with version assertion, independent contract-v1 recomputation and ordered slot comparison, IVF_PQ mirror validation (float16/float32 only; uint8/int8 refused even though Lance supports them), `external_metadata_advanced` produced from a read-only `lance_dataset_latest_version` observation on the pinned handle, one-shot FFI with saved-code-before-message discipline (unknown codes send no result frame), `file://` rejected before any FFI (identity scheme still open). 3. **Cgroup-isolated supervisor** (`be/src/lance/index_job_supervisor*.cpp`) — startup preflight (delegation / no-internal-process / group creation / limit write-read-back / dummy-child migration; any failure rejects submissions — no in-process fallback), per-invocation child cgroup (`memory.max` + `memory.swap.max=0` + `pids.max` + `memory.oom.group=1`), fork barrier (the child arms PDEATHSIG and rechecks its parent before blocking; the parent migrates it into the cgroup, reads back membership, and only then releases — a failure before release kills the child, provably un-exec'd, with the `NEVER_LAUNCHED` proof), pidfd + wall-clock event loop with parallel bounded stdout/stderr drain, SIGTERM → grace → SIGKILL escalation, and `CHILD_REAPED` emitted only with exact-child-reaped plus `populated=0` evidence (immune to the CDC process-wide reaper). rlimit hygiene: DUMPABLE=0, CORE=0, NOFILE=1024, RLIMIT_AS as a 32 GiB address-space fuse (not a memory bound), CPU = 4× wall clock. The deadline budget is recomputed at enqueue and dequeue against `lance_index_worker_report_margin_seconds` (default 400 s — the 3-attempt callback worst case including the client-internal reopen retry) plus the TERM grace; there is no positive clamp past the FE deadline. 4. **BE consumption** — `BackendService::submit_lance_index_job` replaces the stub with guarded bounded enqueue (heartbeat / isolation-verified / payload bounds / duplicate invocation id → idempotent OK / queue-full → clean ERROR; never synchronous execution), `MasterServerClient::report_lance_index_job[_termination]` with the finish_task 3-retry discipline, a `doris_be --lance-worker` thin early branch (self-exec with a controlled two-directory `LD_LIBRARY_PATH` allowlist, validated on the packaged artifact), and metrics (queue depth / in-flight workers / callback counters). DROP jobs now persist a recomputable schema contract at admission (same builder and same pinned snapshot as CREATE), so the worker runs a single revalidation path for all three mutations. Reviewer-visible narrowing: a DROP whose index→column binding cannot be resolved to exactly one top-level field at admission fails closed; legacy no-contract DROP records replay to null and are safely rejected by the new worker; `SHOW LANCE INDEX JOBS` renders the contract column as `1` for new DROP jobs. ★ Design readings asked of reviewers: 1. **Worker shape** — `doris_be --lance-worker` subcommand (same-build worker, zero new packaging). Self-exec needs a controlled `LD_LIBRARY_PATH` (the binary has no RPATH; libjvm/libomp are required); the launcher builds an absolute allowlist from the validated BE install dir and JAVA_HOME only and never inherits an external search path. 2. **field-id binding = manifest-DFS zip, empirically closed** — over lance-c 0.1.9's entire reachable schema-evolution surface (33 drop-probe ground truths + 35 pinned-version dumps covering append/drop/rename/nested/restore/overwrite), DFS preorder skipping the FixedSizeList item child equals the stats ID order; the counterexample shape (a nested child with an id greater than a later top-level column) is unreachable through every write/alter API (nested-child addition is rejected with code 6/1). Guardrails: `stats_count == DFS nodes − FSL columns`, strictly ascending IDs, dictionary-encoded columns rejected, fragments >4096 fail closed. 3. **DROP contract** — admission-persisted recomputable contract; no empty-contract exception to v5.1 §4.2. 4. **`external_metadata_advanced`** — produced by the worker on the pinned handle via read-only `latest_version`; `latest > admitted` ⇒ true. A pinned-open NOT_FOUND is attributed `STALE_ADMISSION` only when a read-only latest open observes a strictly newer version; otherwise `RESOURCE_REJECTED` with the saved typed error. The observation handle is never mutated and its messages are never forwarded. 5. **typed credential classification** — lance-c 0.1.9 produces no typed expiry evidence, so `CREDENTIAL_EXPIRED` is never emitted by this PR; credential failures converge `RESOURCE_REJECTED`; no result code is ever inferred from message text. 6. **deadline budget** — worker wall clock = `min(BE limit, FE remaining − report_margin(400 s) − term_grace(10 s))`; insufficient budget before fork → full async rejection envelope with `NEVER_LAUNCHED`, never a forced short run. 7. **protocol constants** — dispatch 512 KiB / handshake 4 KiB / result 8 KiB, ≤64 storage options, key ≤256 B, value ≤4096 B, enforced at FE-send, BE-enqueue and worker-parse; frozen against a maximal legal fixture test. Not in this PR: `file://` mutation (rejected pre-FFI), scalar indexes and the gate flip (slice 5), UNKNOWN watcher/cancel/retry, K8s delegation (preflight rejects there — a systemd unit with `Delegate=yes memory pids` is the recommended deployment shape; documentation lands with slice 5). The IVF_PQ tracer-bullet e2e and the fault-injection suites follow in the stacked PR4B. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [x] Yes. <!-- Explain the behavior change --> Behavior change: dispatched mutations execute in an isolated worker instead of hitting the NOT_IMPLEMENTED stub; FE-local pre-send failures converge NOT_COMMITTED (was UNKNOWN); DROP admissions persist a schema contract (was null). All behind the still-off `enable_lance_index_mutation` gate. - Does this need documentation? - [ ] No. - [x] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> Documentation lands with slice 5 (worker deployment/delegation requirements, new BE configs `lance_index_worker_*`). ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
