924060929 commented on code in PR #69672:
URL: https://github.com/apache/doris/pull/69672#discussion_r4396702288
##########
fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsScanPlanProvider.java:
##########
@@ -251,26 +254,54 @@ private void serializeFieldContexts(EsMetadataState
state, Map<String, String> n
}
private EsQueryDslResult buildQueryDsl(Optional<ConnectorExpression>
filter,
- EsMetadataState state) {
+ EsMetadataState state, int docValuesMode) {
if (!filter.isPresent()) {
return new EsQueryDslResult("{\"match_all\":{}}",
Collections.emptyList());
}
Map<String, String> fieldsContext = Collections.emptyMap();
List<String> needCompatDateFields = Collections.emptyList();
Map<String, String> column2typeMap = Collections.emptyMap();
+ Set<String> unsafeFields = new HashSet<>();
if (state.getFieldContext() != null) {
if (props.isKeywordSniff()) {
fieldsContext =
state.getFieldContext().getFetchFieldsContext();
}
needCompatDateFields =
state.getFieldContext().getNeedCompatDateFields();
column2typeMap = state.getFieldContext().getColumn2typeMap();
+
unsafeFields.addAll(state.getFieldContext().getSourceValueUnsafeFields());
+ if (!props.isKeywordSniff()) {
+ // Without keyword sniff a term targets analyzed text, even
when a safe raw subfield
+ // exists. Do not apply the unused raw path's safety to the
actual text query.
+
unsafeFields.addAll(state.getFieldContext().getFetchFieldsContext().keySet());
+ }
+ }
+
+ if (docValuesMode == 0) {
+ collectSourceStringFields(filter.get(), unsafeFields);
}
return EsQueryDslBuilder.buildQueryDslWithResult(
filter.get(), fieldsContext, column2typeMap,
- props.isLikePushDown(), needCompatDateFields);
+ props.isLikePushDown(), needCompatDateFields, unsafeFields);
+ }
+
+ private static void collectSourceStringFields(ConnectorExpression
expression, Set<String> fields) {
+ if (expression instanceof ConnectorColumnRef) {
+ ConnectorColumnRef column = (ConnectorColumnRef) expression;
+ String type = column.getType().getTypeName();
+ // Source arrays are serialized as STRING, whereas ES predicates
match their elements.
+ // _id comes from hit metadata and cannot acquire an array
representation in source mode.
+ if (!"_id".equals(column.getColumnName())
+ && ("STRING".equalsIgnoreCase(type) ||
"VARCHAR".equalsIgnoreCase(type)
+ || "CHAR".equalsIgnoreCase(type))) {
Review Comment:
[P2] Preserve numeric predicates after normalizer-triggered source fallback
Projecting a normalized keyword now switches the entire scan from docvalues
to `_source`, but this final-mode check protects only string predicates. A
sibling `scaled_float` maps to DOUBLE and keeps its fully pushed predicate, so
the scan can return a value that fails the SQL `WHERE` condition. I reproduced
this against real Elasticsearch 8.11.3, comparing the merge-base connector
(`af9dad5b0a50`) with this head (`74dc1718b9f4`) while keeping the FE core, BE,
ES data and settings fixed.
**Reproduction**
1. Create this ES index and insert one document:
```http
PUT pr69672_normalizer_repro
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"analysis": {
"normalizer": {
"lowercase_value": {"type": "custom", "filter": ["lowercase"]}
}
}
},
"mappings": {
"properties": {
"a": {"type": "keyword", "normalizer": "lowercase_value"},
"b": {"type": "scaled_float", "scaling_factor": 4}
}
}
}
PUT pr69672_normalizer_repro/_doc/1?refresh=true
{"a": "ok", "b": 2.34}
```
2. Create a Doris ES catalog with docvalue scanning enabled, then run:
```sql
CREATE CATALOG es_normalizer_repro PROPERTIES (
"type" = "es",
"hosts" = "http://<es-host>:9200",
"nodes_discovery" = "false",
"enable_docvalue_scan" = "true"
);
SELECT a, b
FROM es_normalizer_repro.default_db.pr69672_normalizer_repro
WHERE b = 2.25;
```
**Actual results**
| Query | Merge base | This PR head |
| --- | --- | --- |
| `SELECT a,b ... WHERE b=2.25` | `(ok, 2.25)` | `(ok, 2.34)` |
| `SELECT b ... WHERE b=2.25` | `(2.25)` | `(2.25)` |
These results are identical across all four combinations of
`enable_file_scanner_v2=false/true` and `enable_es_parallel_scroll=false/true`.
The value 2.25 is exactly representable in binary, avoiding the floating-point
tail of the earlier factor-10 example. The sample's `a="ok"` is already
lowercase, so normalizing `a` does not itself change the sample value.
The captured ES requests explain the regression. Before this PR:
```json
{"query":{"term":{"b":2.25}},"stored_fields":"_none_","docvalue_fields":["a","b"],"size":8160}
```
At this head:
```json
{"query":{"term":{"b":2.25}},"_source":["a","b"],"size":8160}
```
`changesSourceValue` now detects `a`'s normalizer and removes its docvalue
path, forcing source reads for both columns. However, `b` is absent from the
unsafe-field set, so `term(b,2.25)` remains pushed and its SQL residual is
removed. ES matches the rounded index value `round(2.34 * 4) / 4 = 2.25`; Doris
then returns the source value 2.34, violating `b=2.25`. The projection-only
control above confirms that reading `a` triggers the change.
Please include value-changing numeric mappings in the final-read-mode
equivalence check, retain the SQL residual and avoid a narrowing remote
predicate when equivalence is absent, and add this normalizer/scaled_float
regression with the pure-docvalue projection control.
For clarity, this corrects the attribution in the [earlier scaled_float
comment](https://github.com/apache/doris/pull/69672#discussion_r4387133671):
its `ignore_above` example is pre-existing, because ES omits default
`doc_values=true` from the mapping response and the old connector already falls
back to `_source`. The normalizer case above is the runtime-confirmed
regression introduced by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
