singhpk234 commented on PR #5679: URL: https://github.com/apache/iceberg/pull/5679#issuecomment-1233328664
Posting what I found so far : In case of MOR whats happening is due to https://github.com/apache/iceberg/blob/5764c93a59d8fe5d1c8a60c6ec2cc4deaf8f23d4/spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala#L267 nullifies all row attrs that are not part of the row ID. In mergeRow's output, all rowAttrs that are not part of rowID becomes nullbable, becuase delete actions are not actually ignored here for MOR as they are non-empty, unlike what comment suggests (i think this implies for COW, as we return empty in delete action). https://github.com/apache/iceberg/blob/5764c93a59d8fe5d1c8a60c6ec2cc4deaf8f23d4/spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala#L353-L360 because of this WriteDelta's resolution fails due to table.opt suggesting non-null and projection.schema being nullable. https://github.com/apache/iceberg/blob/5764c93a59d8fe5d1c8a60c6ec2cc4deaf8f23d4/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/WriteDelta.scala#L107-L113 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
