u70b3 opened a new pull request, #67630: URL: https://github.com/apache/doris/pull/67630
> Target: branch-4.1. Stacks on #67201 (PR3A, DDL surface) and #67235 (PR3B, durable job infrastructure). Implements the FE admission half of design v5.1 (issue #66497) slice 3: gate config, one-pinned-snapshot authoritative read, schema contract v1, IF preflight, durable job creation with JobId result, SHOW LANCE INDEX JOBS/JOB, and the catalog DDL guard. **Mutation stays disabled by default** (`enable_lance_index_mutation = false`); this PR does not make index mutation usable — admitted jobs sit PENDING until dispatch (next slice) and FORCE_RELEASE (slice 3E) land. ## What lands 1. **Gate + quotas + static bounds** (`Config.java`, fe-common): `enable_lance_index_mutation` (mutable, masterOnly, EXPERIMENTAL, default false), `lance_index_job_max_unresolved_per_table/catalog/global` (8/64/256, positive-validated), `lance_index_max_num_partitions`/`lance_index_max_num_sub_vectors` (4096/256, positive-validated, masterOnly=false). Validators extend `ConfigBase.DefaultConfHandler` so ADMIN SET both validates and assigns. 2. **Catalog DDL guard** (§4.4): `ALTER CATALOG` SET that changes the value of a target-identity key (`lance.catalog.type`, `warehouse`, `lance.namespace.parent/delimiter/root_database`, key match case-insensitive, same-value rewrites pass) and `DROP CATALOG` are rejected with a neutral message while the catalog has unresolved Lance index jobs. Credential-only ALTERs, renames and replay are never guarded. 3. **Admission snapshot read** (§3.1): one `Dataset.open` yields version + schema fields + logical + physical indexes (no `countRows()`/`getIndexStatistics()`, no MetaCache schema). Duplicate physical names in one snapshot fail closed at construction (a stale post-REPLACE entry may legally coexist until VACUUM; ambiguous is never guessed). The snapshot is pure Java values — no Dataset/allocator/native reference. 4. **Schema contract v1 builder**: reads the LanceField tree directly (LanceTypeConverter erases fslist dimension/float16-32/timestamp tz), canonical `normalizedType` vocabulary pinned in `LanceSchemaContractBuilder` (the slice-4 Rust golden fixtures align to it). No supportability judgment at admission; the only failure is the indexed field missing. 5. **Admission** (§2.2/§4.1/§4.2): reserved `__lance_` prefix rejected three ways; ambiguous case-only collisions fail closed; IF preflight = requested-algorithm equality + physical-family corroboration + single normalized column + per-property whitelist comparison; REPLACE/DROP persist the stored display name of a unique case-insensitive match; exactly one id allocated after all preflights, immediately before the durable `createJob` transfer whose fence/quota rejections pass through verbatim; non-positive quota that slipped in via fe.conf (callbacks only fire on ADMIN SET) is asserted with 5102 before any allocation. Admitted statements answer a single-column `JobId` result set (one row; zero rows for an IF no-op), on direct and forwarded links alike. 6. **Job SQL** (§2.3): `SHOW LANCE INDEX JOBS [FROM [ctl.]db] [WHERE TableName = "t" [AND State = "PENDING"]]` and `SHOW LANCE INDEX JOB <id>`, both FORWARD_NO_SYNC. Row-level authorization: orphan and half-orphan targets (catalog gone, or persisted db/table no longer resolves) are visible to global ADMIN only — non-ADMIN listings omit them entirely, not even a count; a missing job and an unauthorized job share the same fixed 5103 "Lance index job not found: <id>" response. Locator, provider, normalized names, propertiesJson and contract contents are never shown; null result/dispatch fields render empty (no worker exists yet). 7. **Error codes**: 5102 `ERR_LANCE_INDEX_MUTATION_DISABLED` (gate-off rejection and quota backstop), 5103 `ERR_LANCE_INDEX_JOB_NOT_FOUND`. ## Property comparison semantics (pinned here; the plan's N1 left value form open) Per §2.2 "compared only when the snapshot exposes a stable value; no additional canonicalization is a prerequisite": - `metric` ↔ `metric_type`: both sides case-fold (`toLowerCase(ROOT)`) — Lance stores `"L2"`, the validated request vocabulary is lowercase `"l2"`; without folding a same-definition IF NOT EXISTS could never no-op. - `num_sub_vectors` / `num_bits`: parsed-long numeric equality (`"4"` ≡ `4`). - Snapshot side key absent → property skipped; request side absent → never compared; snapshot value present but unparsable/non-primitive → malformed data, fail closed (§3.4). - `num_partitions` is never compared. ## Operational notes (read before enabling the gate) - The gate and the three quotas are masterOnly: `ADMIN SET FRONTEND CONFIG` lands on the master node's local custom conf and does not fan out; after a master switch/restart the values fall back to fe.conf/defaults (fail-closed direction). The two static bounds are masterOnly=false — use `ADMIN SET ALL FRONTEND CONFIG` (or set on the master) for cluster-wide effect. - The callback only fires on the ADMIN SET path; fe.conf values bypass it, so admission re-asserts quota positivity (this is the only defense for file-loaded zeros). - Enabling before dispatch (next slice) and FORCE_RELEASE (3E) land creates PENDING jobs that cannot be resolved, and freezes identity-property changes and DROP CATALOG for the affected catalog until then. The gate's description carries this warning. - FORWARD_NO_SYNC job SQL on a follower may lag a just-admitted job (replay lag); the admission response already returns the JobId, so this is cosmetic. - Lock ordering: the catalog guard takes the job-manager read lock inside the CatalogMgr write lock, and `createJob` holds the manager write lock across the journal write — a single order (catalog → manager → journal), no deadlock; journal stalls affect catalog-DDL liveness the same way existing "editlog inside DDL lock" does. Accepted, same shape as the baseline. - The catalog guard's old-value lookup is deliberately case-sensitive on the incoming key's spelling: a case-variant rewrite of the same value is over-blocked (fail-closed), never under-blocked. ## Deliberately deferred / not done (with owners) - Dispatcher, BE selection, thrift, worker (slice 3D/4); FORCE_RELEASE + RESOLVE SQL and job-retention GC (3E) — admitted jobs stay durable with no GC until 3E (the regression suite uses per-run index/catalog names so reruns cannot collide with leftover fences). - Possible-live slot configuration (3D); gate default flip (§9.7 evidence gates G1–G5); REST catalog mutation (still 5101); ALTER TABLE ADD/DROP INDEX and BUILD INDEX (still rejected). - `local/file://` operator assertion rides with 3D (it lives in §5.2 "before dispatch"; 3C has no dispatch path, and PENDING is strictly non-invoking). - WHERE on job SQL supports EqualTo+AND only (no Like — ShowCopyCommand precedent supports it; left for later). - `describeIndices` keeps PR1's per-name criteria form: the no-arg bulk describe materializes internal-index details the SDK does not support. - `LancePhysicalIndexEntry` (PR2) unchanged — the snapshot carries the physical index type itself. - `Long.parseLong` overflow on `SHOW LANCE INDEX JOB <id>` surfaces as a generic parse error, same as `SHOW PARTITION <id>` and all existing precedents. ## Tests - FE UT (JUnit5+Mockito, no FE startup, no JNI): config validator (assign-and-validate, defaults), catalog guard (5 identity keys × value-change, same-value pass, case-variant keys, credential pass, drop/rename/replay, REST no-op, neutral message), job-manager queries (ordering, copy isolation, terminal-released not unresolved), contract builder (full ArrowType canonical table, decimal/timestamp/float16/int widths, fixed_size_list literal, missing field), families (normalization incl. Turkish-İ, family matrix, case-collision), snapshot (bounds, duplicate physical name fail-closed, no-native-handle reflection), admission (§2.2 full matrix incl. same-name different-algorithm rejection, stored-name persistence, quota edge `count+1==limit`, quota ≤0 5102, propertiesJson normalization, contract persistence, four-nothing failure paths), AlterTableCommand rewiring (gate-off 5102 three forms, JobId result set, no-op zero rows, alter=true regression, internal-table zero impact, proxy branch, single-validate invariant), job SQL parser + commands (auth filtering, half-orphan ADMIN-only, 5103 identical text, null-result rendering, forwarded-identity authorization, FORWARD_NO_SYNC). - Regression (`external_table_p0/lance`): `test_lance_index_ddl.groovy` updated to the 5102 wording; new `test_lance_index_admission.groovy` (gate on in try, restored in finally; admitted CREATE returns JobId and shows PENDING; same-name fence conflict; IF NOT EXISTS no-op/mismatch against the preloaded `embedding_ivf_pq_f32`; DROP IF EXISTS no-op/admit; quota=1 rejection; non-privileged invisibility; REST still rejected). Needs the external MinIO/REST docker env — runs in pipeline, not locally. - The admitted-DROP case targets the preloaded index because a name whose CREATE is still PENDING is authoritatively absent (§2.2) and same-name fenced (§6.4); the regression catalog carries a per-run suffix because fence/quota keys include the persisted catalog id and DROP CATALOG is guarded while jobs are unresolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
