uthanuja opened a new issue, #16246: URL: https://github.com/apache/iceberg/issues/16246
### Apache Iceberg version 1.5.2 ### Query engine Spark ### Please describe the bug 🐞 When performing a MERGE INTO operation on an Iceberg table where a nullable parent struct contains non-nullable child fields, the struct's non-nullable children silently become null — even though only other columns were updated. No error is raised. Schema example: Type: struct<Type: string NOT NULL, TypeAttr: struct<...>> (NULLABLE) The parent field `Type` is nullable, but its child field `Type.Type` is non-nullable. What happens: During a MERGE operation that updates other columns (not the struct itself), the nullable parent struct's non-nullable child fields silently become null. No error is raised. Subsequent reads return null for `Type.Type` even though it's marked NOT NULL and was never modified by the MERGE. Expected behavior: Either: 1. Iceberg should preserve the existing struct data when the MERGE does not modify that column, OR 2. Iceberg should raise an error if a non-nullable field would become null (even transitively through a null parent) Steps to reproduce: 1. Create an Iceberg table with a nullable struct column containing a non-nullable child field 2. Insert rows where the struct is populated (child field has a value) 3. Perform a MERGE INTO that updates other columns (not the struct itself) for some rows 4. Read back the table — the struct's non-nullable child field is now null despite not being touched by the MERGE Additional context: - This is not related to parent and child having the same name — tested with different names (Type.Type2) and the same issue occurs. - Full table rewrites do not trigger the issue — only incremental MERGE operations. - The issue may be related to how Iceberg reconciles Parquet file-level nullability with the Iceberg schema during partial updates. Environment: - Iceberg 1.5.2 - Spark 3.3 - Hadoop catalog type - Storage: ABFSS (Azure Data Lake) ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
