Gabriel39 commented on code in PR #69672:
URL: https://github.com/apache/doris/pull/69672#discussion_r4398417925
##########
fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsMappingUtils.java:
##########
@@ -153,9 +153,73 @@ public static EsFieldContext
resolveFieldContext(List<String> columnNames,
resolveKeywordFields(fieldContext, fieldObject, colName,
fieldType);
resolveDocValuesFields(fieldContext, fieldObject, colName,
fieldType);
}
+ resolveSourceValueSafety(fieldContext, indexMapping, mappingType);
return fieldContext;
}
+ private static void resolveSourceValueSafety(EsFieldContext context,
String indexMapping, String mappingType) {
+ try {
+ // An alias can span mappings. Every backing index must preserve
source semantics before
+ // a predicate may remove rows or doc_values may replace the value
returned to Doris.
+ for (JsonNode index : MAPPER.readTree(indexMapping)) {
+ ObjectNode root = getRootSchema((ObjectNode)
index.get("mappings"), mappingType, new ArrayList<>());
Review Comment:
Rechecked the complete thread, including the configuration-only follow-up.
Fixed in 789ceff51d1bfcaff7de1da02f9d8d18d715846e.
Root parsing now distinguishes typeless scalar/array settings and recognized
root objects from legacy typed wrappers. This handles enabled:false,
date/numeric detection, dynamic date formats, source settings and runtime-only
roots, while preserving selected-schema array annotations and explicit
missing-schema diagnostics. Legacy type names resembling scalar settings are
also covered.
The previous connector reproduced the BooleanNode/ObjectNode failure against
live ES 8.11.3. Tests cover both backing orders; the live suite additionally
stores source-only documents in enabled:false/dynamic:false backings and
verifies both full alias reads and predicates retain those documents. All 272
new stored checks and 272 independent SQL comparisons passed across both
readers, docvalue modes and parallel modes, followed by a separate
output-verification run.
##########
fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsMappingUtils.java:
##########
@@ -153,9 +162,101 @@ public static EsFieldContext
resolveFieldContext(List<String> columnNames,
resolveKeywordFields(fieldContext, fieldObject, colName,
fieldType);
resolveDocValuesFields(fieldContext, fieldObject, colName,
fieldType);
}
+ resolveSourceValueSafety(fieldContext, indexMapping, mappingType);
return fieldContext;
}
+ private static void resolveSourceValueSafety(EsFieldContext context,
String indexMapping, String mappingType) {
+ try {
+ // An alias can span mappings. Every backing index must preserve
source semantics before
+ // a predicate may remove rows or doc_values may replace the value
returned to Doris.
+ List<ObjectNode> roots = new ArrayList<>();
+ Set<String> columns = new
HashSet<>(context.getColumn2typeMap().keySet());
+ for (JsonNode index : MAPPER.readTree(indexMapping)) {
+ ObjectNode root = getRootSchema((ObjectNode)
index.get("mappings"), mappingType,
+ new ArrayList<>(), true);
+ roots.add(root);
+ }
+ // Only the first mapping's schema is exposed by Doris, including
unprojected filter columns.
+
roots.get(0).path("properties").fieldNames().forEachRemaining(columns::add);
+ Set<String> inexactKeywords = new HashSet<>();
+ for (ObjectNode root : roots) {
+ JsonNode properties = root.path("properties");
+ JsonNode source = root.path("_source");
+ boolean filteredSource =
!source.path("enabled").asBoolean(true)
+ || source.has("includes") || source.has("excludes");
+ Set<String> copyTargets = new HashSet<>();
+ collectCopyTargets(properties, copyTargets);
+ for (String column : columns) {
+ JsonNode field = properties.path(column);
+ String indexedPath =
context.getFetchFieldsContext().get(column);
+ JsonNode indexedField = field;
+ if (indexedPath != null) {
+ indexedField =
field.path("fields").path(indexedPath.substring(column.length() + 1));
+ }
+ boolean inexact = filteredSource ||
root.path("runtime").has(column)
+ || copyTargets.contains(column) ||
changesSourceValueExceptLength(field)
+ || changesSourceValueExceptLength(indexedField);
+ for (JsonNode subfield : field.path("fields")) {
+ if ("keyword".equals(subfield.path("type").asText())
+ && changesSourceValueExceptLength(subfield)) {
+ inexact = true;
+ }
+ }
+ if (inexact || field.has("ignore_above") ||
indexedField.has("ignore_above")
+ || hasUnsafeKeywordSubfield(field)) {
+ context.getSourceValueUnsafeFields().add(column);
+ }
+ if (!inexact &&
"keyword".equals(indexedField.path("type").asText())) {
+ // Equality/IN with bounded literals stay exact:
ignored longer values cannot
+ // equal those literals. Intersect the bound over
every backing index.
+ context.getKeywordIgnoreAboveLimits().merge(column,
+
indexedField.path("ignore_above").asInt(Integer.MAX_VALUE), Math::min);
+ } else {
+ inexactKeywords.add(column);
+ }
+ }
+ }
+
inexactKeywords.forEach(context.getKeywordIgnoreAboveLimits()::remove);
+ } catch (IOException e) {
+ throw new DorisConnectorException("Failed to parse ES mapping: " +
e.getMessage());
+ }
+ // Local residuals must read the original value, including with
explicit doc_values=true.
+
context.getSourceValueUnsafeFields().forEach(context.getDocValueFieldsContext()::remove);
+ }
+
+ private static boolean hasUnsafeKeywordSubfield(JsonNode field) {
+ for (JsonNode subfield : field.path("fields")) {
+ if ("keyword".equals(subfield.path("type").asText()) &&
changesSourceValue(subfield)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static boolean changesSourceValue(JsonNode field) {
+ return field.has("ignore_above") ||
changesSourceValueExceptLength(field);
+ }
+
+ private static boolean changesSourceValueExceptLength(JsonNode field) {
+ return field.isMissingNode() || field.has("normalizer")
+ || field.has("null_value") ||
field.path("ignore_malformed").asBoolean(false)
+ || !field.path("index").asBoolean(true);
Review Comment:
Rechecked the complete thread, including the numeric extension. Fixed in
789ceff51d1bfcaff7de1da02f9d8d18d715846e.
Searchability now includes supported numeric, date, boolean and IP docvalues
as well as keyword, with explicit/default docvalues controls. Source/read-value
equivalence remains a separate check: this does not bypass source
string/rounded-number restrictions, and fields with both index and docvalues
disabled still retain local predicates.
I reproduced the previous head rejecting the requested BIGINT/native OR on
live ES 8.11.3. New FE coverage spans every numeric family, date, boolean and
IP with native OR/NOT and negative controls. Runtime OR/NOT comparisons use an
independent internal-table oracle for BIGINT/date/boolean across both readers,
docvalue modes and parallel settings. The NULL rows exposed a related
inequality issue after NOT rewriting; inequalities now require field existence
instead of letting ES must_not match missing values. NULL rows remain in the
test oracle.
Validation: 152 FE tests passed, one optional external class skipped,
Checkstyle passed; all 1,148 stored regression checks and 1,052 independent
comparisons passed. The 88 expected native planning errors are preserved.
##########
fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsScanPlanProvider.java:
##########
@@ -251,26 +254,54 @@ private void serializeFieldContexts(EsMetadataState
state, Map<String, String> n
}
private EsQueryDslResult buildQueryDsl(Optional<ConnectorExpression>
filter,
- EsMetadataState state) {
+ EsMetadataState state, int docValuesMode) {
if (!filter.isPresent()) {
return new EsQueryDslResult("{\"match_all\":{}}",
Collections.emptyList());
}
Map<String, String> fieldsContext = Collections.emptyMap();
List<String> needCompatDateFields = Collections.emptyList();
Map<String, String> column2typeMap = Collections.emptyMap();
+ Set<String> unsafeFields = new HashSet<>();
if (state.getFieldContext() != null) {
if (props.isKeywordSniff()) {
fieldsContext =
state.getFieldContext().getFetchFieldsContext();
}
needCompatDateFields =
state.getFieldContext().getNeedCompatDateFields();
column2typeMap = state.getFieldContext().getColumn2typeMap();
+
unsafeFields.addAll(state.getFieldContext().getSourceValueUnsafeFields());
+ if (!props.isKeywordSniff()) {
+ // Without keyword sniff a term targets analyzed text, even
when a safe raw subfield
+ // exists. Do not apply the unused raw path's safety to the
actual text query.
+
unsafeFields.addAll(state.getFieldContext().getFetchFieldsContext().keySet());
+ }
+ }
+
+ if (docValuesMode == 0) {
+ collectSourceStringFields(filter.get(), unsafeFields);
}
return EsQueryDslBuilder.buildQueryDslWithResult(
filter.get(), fieldsContext, column2typeMap,
- props.isLikePushDown(), needCompatDateFields);
+ props.isLikePushDown(), needCompatDateFields, unsafeFields);
+ }
+
+ private static void collectSourceStringFields(ConnectorExpression
expression, Set<String> fields) {
+ if (expression instanceof ConnectorColumnRef) {
+ ConnectorColumnRef column = (ConnectorColumnRef) expression;
+ String type = column.getType().getTypeName();
+ // Source arrays are serialized as STRING, whereas ES predicates
match their elements.
+ // _id comes from hit metadata and cannot acquire an array
representation in source mode.
+ if (!"_id".equals(column.getColumnName())
+ && ("STRING".equalsIgnoreCase(type) ||
"VARCHAR".equalsIgnoreCase(type)
+ || "CHAR".equalsIgnoreCase(type))) {
Review Comment:
Rechecked and closing this older numeric thread. Commit
0a792c9461252986de54efcf38bd0d7d845c0612 already applies a source-mode-only
guard for scaled_float and half_float after the final scan mode is selected,
retaining the SQL residual without a narrowing numeric query while preserving
pure numeric docvalue scans.
The later normalizer reproduction correctly distinguishes the PR-introduced
regression from the earlier ignore_above trigger, which already fell back to
source at the merge base. The same final-mode guard protects either source
trigger. Fresh validation on the current update passed all 264 numeric stored
checks, 264 independent SQL comparisons and 12 expected native planning-error
checks across 12 configurations, including the pure-docvalue controls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
