randomdev2027 opened a new issue, #70429: URL: https://github.com/apache/doris/issues/70429
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.5 ### What's Wrong? We use Apache Doris to transform complex JSON-Payloads (OpenAPI-Specification) using the VARIANT-type and write them back to S3 (with Apache Polaris and Apache Iceberg). With an Iceberg-Field definition like `{ "id": 17, "name": "issuelinks", "required": false, "type": { "type": "list", "element-id": 18, "element-required": false, "element": { "type": "struct", "fields": [ {"id": 19, "name": "relationshipType", "required": false, "type": "string"}, { "id": 20, "name": "ticketRefs", "required": false, "type": { "type": "list", "element-id": 21, "element-required": false, "element": { "type": "struct", "fields": [ {"id": 22, "name": "id", "required": false, "type": "string"} ] } } } ] } }, "doc": "some description" }` we are able to use "MERGE INTO" using a Iceberg-Catalog, but if we change field id number 22 to required true, we face the following failure: `can not cast from origin type ARRAY<STRUCT<relationshiptype:TEXT,ticketrefs:ARRAY<STRUCT<id:TEXT>>>> to target type=ARRAY<STRUCT<relationshiptype:TEXT,ticketrefs:ARRAY<STRUCT<id:TEXT NOT NULL>>>>` Our payload is valid and it look similar to `"issuelinks" : [ { "relationshipType" : "clones", "ticketRefs" : [ { "id" : "COJPDMCEC" } ] } ]` Our MERGE INTO looks like (excerpt): ` MERGE INTO ${DORIS_CATALOG}.${ICEBERG_OUTPUT_NAMESPACE}.${ICEBERG_OUTPUT_TABLE} target USING ( SELECT ... CASE WHEN json_extract(s.payload_raw, '$.daten.ticket.issuelinks') IS NULL THEN NULL ELSE CAST( array_map( x0 -> STRUCT( json_extract_string(x0, '$.relationshipType'), CAST( array_map( x1 -> STRUCT( json_extract_string(x1, '$.id') ), CAST(json_extract(x0, '$.ticketRefs') AS ARRAY<STRING>) ) AS ARRAY<STRUCT<id:STRING>> ) ), CAST(json_extract(s.payload_raw, '$.daten.ticket.issuelinks') AS ARRAY<STRING>) ) AS ARRAY<STRUCT< relationshipType:STRING, ticketRefs:ARRAY<STRUCT<id:STRING>> >> ) END AS issuelinks FROM... ` ### What You Expected? "MERGE INTO" with required fields should work ### How to Reproduce? Please see Bug description ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
