Gabriel39 opened a new pull request, #69672:
URL: https://github.com/apache/doris/pull/69672
### What problem does this PR solve?
ES scans can return incorrect SQL results in two cases:
1. **Indexed values differ from document source values.** For a keyword
field with `ignore_above=4`, a source value of `elephant` is readable but
absent from the index. Pushing `field = 'elephant'` down as a term query loses
a matching row before Doris can filter it. Normalizers, null substitutions and
copied/runtime values can also produce false positives or incorrect null
filtering. The existing docvalue eligibility check does not establish predicate
equivalence, and explicit `doc_values=true` can bypass the `ignore_above` read
fallback.
2. **Parallel scans replace an alias with its backing index.** This bypasses
alias filters. For a two-index filtered alias in the regression fixture,
disabling parallel scroll returns four rows, while enabling it returns all
eight backing documents. Reusing the original multi-index alias separately for
each shard would risk repeated scans when backing indices share shard IDs.
Derive source-value safety from every backing mapping. Keep affected SQL
predicates local without sending a narrowing remote predicate, and read the
affected fields from `_source`. Cover keyword subfields, `ignore_above`,
normalizers, null substitutions, disabled indexing, explicit
`ignore_malformed`, copy targets, runtime overrides and source filtering. Keep
Boolean residual tracking and the existing early-LIMIT gate intact; explicit
`esquery` retains native ES semantics.
When shard discovery resolves the requested name to backing indices, issue
one scroll against the original target so Elasticsearch applies its alias
filter and search routing. Physical indices retain shard-parallel scans. This
conservative fix trades pushdown/docvalue optimization on affected fields and
scan parallelism on aliases for correct results.
Add `test_es_source_semantics` with generated output. It compares source
predicates against a Doris internal-table baseline and alias results against
native ES queries, across both scanner versions, docvalues on/off and parallel
scroll on/off. Cover multi-index aliases with different filters and shared
shard IDs, routed/filtered aliases, empty results, full row counts, pagination,
nested Boolean filters and LIMIT.
### Release note
Fix incorrect Elasticsearch query results caused by source/index mapping
differences and filtered aliases in parallel scans. Alias reads now use one
scroll against the original alias.
### Check List (For Author)
- Test
- [x] Four new FE test methods fail against the unmodified
implementation before the fix.
- [x] ES connector suite: 135 reported tests, 134 passed and one
optional live ES 9 compatibility class skipped; no failures or errors.
- [x] FE package build using `build.sh --fe` and Checkstyle for the ES
module and reactor dependencies.
- [x] Real Elasticsearch 8.11.3 SQL regression: eight configurations,
288 generated result checks and 208 independent comparisons.
- [x] Unmodified connector reproduces missing keyword rows,
normalized/null-value mismatches and alias over-reads against the same live
fixtures; the new SQL suite fails on the missing keyword row.
- [x] Regression output generated by `run-regression-test.sh`, then
verified by a separate run without output generation.
- Behavior changed:
- [x] Yes: unsafe mappings retain local filters/source reads; alias
targets use one coordinated scroll.
- Does this need documentation?
- [x] No new SQL syntax or catalog/session properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
