Doris-Breakwater commented on issue #70429: URL: https://github.com/apache/doris/issues/70429#issuecomment-6761346008
Initial triage: **FE/Nereids type coercion involving Iceberg nested required fields.** The reported error is consistent with an explicit nullability safety check. It does not, by itself, establish a VARIANT conversion defect or invalid input data. No labels are currently assigned. **Verified code mechanism** I inspected the public `4.1.5-release` tag, commit `917b20be83878e98f94e3ebf7efccc80346e3e50`, without modifying the repository. This is source inspection, not an end-to-end reproduction on the reported deployment. 1. The only difference in the reported types is `ticketRefs[].id`: nullable `TEXT` in the source versus `TEXT NOT NULL` in the target. Iceberg struct conversion preserves nested field optionality using `x.isOptional()`, so changing field 22 to required explains that target type. See [IcebergUtils](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java#L803-L812). 2. The explicit casts in the supplied SQL declare `STRUCT<id:STRING>`, whose field is nullable. `LogicalPlanBuilder.visitComplexColType` constructs such fields with `nullable=true`; therefore, a non-NULL sample value does not make that declared field required. The nested-field grammar also does not accept simply adding `NOT NULL` inside this cast type. See [type construction](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java#L5531-L5536) and [grammar](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4#L1934-L1940). 3. `CheckCast` recursively checks arrays and structs and returns false when a source struct field is nullable but its target is required. This check applies in both strict and non-strict cast modes. `TypeCoercionUtils.checkCanCastTo` emits the exact reported error wording. See [recursive cast checks](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/check/CheckCast.java#L400-L430) and [error construction](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java#L576-L585). 4. Iceberg MERGE adds target-type casts to both UPDATE assignments and INSERT values. This provides a code path for the source/target mismatch to surface during MERGE analysis. See [UPDATE projection](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergMergeCommand.java#L297-L300) and [INSERT projection](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergMergeCommand.java#L367-L374). Thus, valid payload values and a rejected cast can coexist: the analyzer checks the expression's declared type, without proving every nested value non-NULL. The outer CASE checks the presence of `issuelinks`; it does not establish non-nullability of each nested `id`. **What remains unverified / information needed** - Exact FE/BE build versions and git hashes, including any downstream patches. - A complete minimal script: source DDL and `payload_raw` type, one sanitized input row, full MERGE including ON/WHEN clauses, and target Iceberg schema/table properties. Include schema ID and whether requiredness was present at creation or changed later; omit credentials. - Whether the USING SELECT succeeds independently, whether an equivalent INSERT SELECT fails, and whether UPDATE-only and INSERT-only MERGE cases behave differently. Please provide the complete error, query ID, and corresponding FE log/stack trace. Include EXPLAIN output if available, or its error if analysis fails. An execution profile is not needed for this initial analysis failure. **Suggested next steps** For maintainers, reproduce against two disposable Iceberg tables that differ only in field 22's requiredness. Start with a constant ARRAY/STRUCT source to remove JSON/VARIANT extraction from the experiment, then add back the supplied expression and test INSERT and each MERGE action separately. This will distinguish a general nested-field assignment limitation from a MERGE-specific problem. A candidate SQL experiment is to construct named structs with an explicitly non-null leaf expression, avoiding the casts that redeclare `id` nullable. This is **not a verified workaround**. Wrapping the whole array in `non_nullable` does not change its nested field types, and any leaf assertion must reject actual NULLs rather than substitute values. Do not relax the Iceberg schema solely to suppress the error unless optional IDs are intended. If broader write support is needed, evaluate it with explicit runtime enforcement of required descendants. Regression coverage should include valid IDs, missing/null IDs, empty lists, nullable list elements, and absent optional parents; an absent optional parent must remain legal. Do not simply remove the cast guard. No patch or fixed release is established by this investigation. Breakwater-GitHub-Analysis-Slot: slot_a85fe2b6d5f1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
