eldenmoon opened a new pull request, #69761: URL: https://github.com/apache/doris/pull/69761
### What problem does this PR solve? Issue Number: None Related PR: https://github.com/eldenmoon/incubator-doris/pull/18 Problem Summary: Provide the minimal VARIANT whole-column token search implementation. A row is one docs-only SNII T1 document; object and array leaves are analyzed independently, keys are omitted, and MATCH_ALL can be satisfied across leaves in the same row. SQL NULL remains distinct from an empty document. The supported entry points are direct-root MATCH / MATCH_ANY / MATCH_ALL on VARIANT V2, with one built-in English values index per column on a DUPLICATE KEY table. Exact-value indexes, path-bound candidates, CAST reverse probes, OTHER markers, multiple analyzers, MOW, phrases, scoring, and ADD/BUILD/DROP of this index are outside this version. Ordinary path predicates continue through their existing indexes or scalar execution. SEARCH excludes the values index in both generic and field-pattern lookup; it reports a missing-index error when no applicable path index exists. Import and rebuild both index finalized storage values. This includes dynamic promotions and DOC buckets; the latter use the existing V1 storage-cell decoder rather than interpreting cells as JSONB. Legacy JSON-text index input is rejected. Root scalar execution shares the analyzer bound from the values index. A derived root slot without unambiguous source/index metadata (including UNION ALL output) is rejected instead of using a default analyzer. Ordinary object subpaths retain full JSON-to-STRING keyword semantics. Root and ordinary path indexes have independent analyzer deduplication scopes, so the same English analyzer may be used in both. TOKEN encoding stays byte-compatible (`0x07` plus raw token bytes), and shared NULL, escaping, residual and memory infrastructure is retained. Compaction plans the data layout before fixing index maintenance. It checks source type metadata against the proposed extracted-column conversions, allowing unchanged types and proven signed-integer widening. Unproven boolean/numeric, floating, sparse, declared-path, or root-scalar conversions rebuild from final V2 values. Schema extension operates on copied schemas; the current V2-only writer rebuilds from the complete logical value. Eligible homogeneous full/cumulative compaction uses the existing SNII streamed T1 merge, rowid validation and abort behavior. Source metadata is aggregated once for all VARIANT columns and reused by every root safety proof and both schema plans; native-disabled compaction skips root safety analysis. Each root index records native/rebuild mode and rebuild reason. Native compaction assumes valid source postings and does not repair previously incorrect postings. Direct-load V2 writers consume the caller-filtered index list, preserving skipped index writes instead of reloading every schema index. Empty writers follow the current V2-only lifecycle and can finish an empty values index without appending rows. Current master forbids setting skip_write_index_on_load=true through SQL; the filtered-writer contract is covered directly by BE tests. No complete production SQL trigger is claimed for the historical zero-row V1-selection defect. The branch is reconciled with master's shared TextIndexReader/SNII engine, V2-only writer and compaction-path metadata APIs. Existing ordinary array-phrase expectations follow current master. Obsolete exact/candidate tests and the experimental benchmark are removed. No old benchmark numbers are claimed for this implementation. ### Release note Add minimal whole-column VARIANT token search using one English values index on DUPLICATE KEY tables, with safe native index compaction or final-value rebuild. ### Check List (For Author) - [x] Regression test - [x] Unit Test - Test: Validated commit: `61ab69fa942bc135c5cb317574a55a41e3f5e29d` (merged master: `d8bbb991656a1b87a4792f120ee80470b0b36ca0`). ASAN builds via `build.sh --be` and `build.sh --fe` succeeded. 121 distinct BE unit tests passed; 2 NestedGroup tests skipped because the module is disabled. 188 FE unit tests passed. Restoring the F1/F3/K3 defect logic makes all three focused FE tests fail (zero test errors), and restoring the fixes passes. The complete `variant_p0/v2/with_index` directory passes 6 suites with zero failures/skips, both generating expectations and comparing normally. SQL checks lower_case=false, ambiguous UNION ALL binding, ordinary object keyword behavior and root/path coexistence. Compaction instrumentation confirms one aggregation per source rowset across three root-indexed columns, and zero proof aggregation when native merge is disabled. No RELEASE performance benchmark was run. - Behavior changed: Yes; adds direct-root token MATCH on a values index. Ambiguous derived roots are rejected, and ordinary object subpaths retain their existing keyword behavior. - Does this need documentation: Yes; the supported contract and limitations are described above. A public documentation PR is not included. ### Review checkpoints - Scope and reuse: retain the existing SNII T1 streamed compaction engine, analyzer, NULL tracking and final-value conversion helpers; remove unsupported exact/candidate branches. - Correctness and coverage: cover ordinary and DOC storage, empty documents versus SQL NULL, index/scalar parity, ordinary path coexistence, native full/cumulative compaction, unsafe conversion rebuild and non-vertical schema ownership. - Compatibility and lifecycle: token bytes remain unchanged; enforce the minimal creation contract and existing lifecycle restrictions. This does not migrate or repair historical invalid postings. - Ownership, failure and observability: clone shared schemas before mutation; preserve resource preflight, rowid checks and atomic abort behavior; log native/rebuild decisions and reasons. - Style and analysis: clang-format 16 and header hygiene pass. Source whitespace checks pass; regression outputs retain the runner's canonical final blank line. Clang-tidy checked all 40 changed C++ files against the ASAN compilation database: zero diagnostics on changed lines, using the repository's diagnostic-start-line policy. Five function-level complexity diagnostics outside the changed lines are recorded separately. An explicit empty member initializer avoids compiler errors in designated aggregate callers and has a justified readability-redundant-member-init suppression. - Performance: no RELEASE benchmark was run; previous experimental benchmark numbers have been removed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
