github-actions[bot] commented on code in PR #69672:
URL: https://github.com/apache/doris/pull/69672#discussion_r4387133671
##########
fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsScanPlanProvider.java:
##########
@@ -251,26 +254,54 @@ private void serializeFieldContexts(EsMetadataState
state, Map<String, String> n
}
private EsQueryDslResult buildQueryDsl(Optional<ConnectorExpression>
filter,
- EsMetadataState state) {
+ EsMetadataState state, int docValuesMode) {
if (!filter.isPresent()) {
return new EsQueryDslResult("{\"match_all\":{}}",
Collections.emptyList());
}
Map<String, String> fieldsContext = Collections.emptyMap();
List<String> needCompatDateFields = Collections.emptyList();
Map<String, String> column2typeMap = Collections.emptyMap();
+ Set<String> unsafeFields = new HashSet<>();
if (state.getFieldContext() != null) {
if (props.isKeywordSniff()) {
fieldsContext =
state.getFieldContext().getFetchFieldsContext();
}
needCompatDateFields =
state.getFieldContext().getNeedCompatDateFields();
column2typeMap = state.getFieldContext().getColumn2typeMap();
+
unsafeFields.addAll(state.getFieldContext().getSourceValueUnsafeFields());
+ if (!props.isKeywordSniff()) {
+ // Without keyword sniff a term targets analyzed text, even
when a safe raw subfield
+ // exists. Do not apply the unused raw path's safety to the
actual text query.
+
unsafeFields.addAll(state.getFieldContext().getFetchFieldsContext().keySet());
+ }
+ }
+
+ if (docValuesMode == 0) {
+ collectSourceStringFields(filter.get(), unsafeFields);
}
return EsQueryDslBuilder.buildQueryDslWithResult(
filter.get(), fieldsContext, column2typeMap,
- props.isLikePushDown(), needCompatDateFields);
+ props.isLikePushDown(), needCompatDateFields, unsafeFields);
+ }
+
+ private static void collectSourceStringFields(ConnectorExpression
expression, Set<String> fields) {
+ if (expression instanceof ConnectorColumnRef) {
+ ConnectorColumnRef column = (ConnectorColumnRef) expression;
+ String type = column.getType().getTypeName();
+ // Source arrays are serialized as STRING, whereas ES predicates
match their elements.
+ // _id comes from hit metadata and cannot acquire an array
representation in source mode.
+ if (!"_id".equals(column.getColumnName())
+ && ("STRING".equalsIgnoreCase(type) ||
"VARCHAR".equalsIgnoreCase(type)
+ || "CHAR".equalsIgnoreCase(type))) {
Review Comment:
[P2] Account for scaled numeric values when falling back to source
This final-mode check only protects string slots, but `scaled_float` maps to
Doris DOUBLE and can also differ from `_source`. Map `a` as `keyword` with
`ignore_above=4,doc_values=true` and `b` as `scaled_float` with
`scaling_factor=10`; insert `{"a":"ok","b":2.34}`. With docvalue scanning
enabled, `SELECT a,b FROM idx WHERE b = 2.3` previously read `b=2.3`. This PR
removes `a`'s docvalue path and makes both readers read source `b=2.34`, yet
leaves `b`'s term query pushed and prunes its SQL residual, returning a row
that fails the equality. ES documents [this
rounding](https://www.elastic.co/guide/en/elasticsearch/reference/8.19/number.html#scaled-float-params).
This is a scalar numeric instance in the new type filter, distinct from the
existing STRING-array thread. Include numeric mappings that change the indexed
representation in source-mode safety, avoid a narrowing remote predicate, and
add a scaled_float regression with a pure-docvalue control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
