[GitHub] [spark] Yaohua628 commented on pull request #38683: [SPARK-41151][SQL][3.3] Keep built-in file `_metadata` column nullable value consistent

Yaohua628 commented on PR #38683: URL: https://github.com/apache/spark/pull/38683#issuecomment-1319636434 > If it has been persisted before (like a table), then it's totally fine to write non-nullable data to a nullable column. The optimizer may also optimize a column from nullable to non-n

[GitHub] [spark] Yaohua628 commented on pull request #38683: [SPARK-41151][SQL][3.3] Keep built-in file `_metadata` column nullable value consistent

Yaohua628 commented on PR #38683: URL: https://github.com/apache/spark/pull/38683#issuecomment-1319593235 > shall we change `FileSourceMetadataAttribute`? I initially thought we could relax this field for some future cases. But yeah, you are right, it seems like it is always not null

[GitHub] [spark] Yaohua628 commented on pull request #38683: [SPARK-41151][SQL][3.3] Keep built-in file `_metadata` column nullable value consistent

Yaohua628 commented on PR #38683: URL: https://github.com/apache/spark/pull/38683#issuecomment-1319529815 > Maybe simpler to apply KnownNullable / KnownNotNull against CreateStruct to enforce desired nullability? Please refer the change in https://github.com/apache/spark/pull/35543.

[GitHub] [spark] Yaohua628 commented on pull request #38683: [SPARK-41151][SQL][3.3] Keep built-in file `_metadata` column nullable value consistent

Yaohua628 commented on PR #38683: URL: https://github.com/apache/spark/pull/38683#issuecomment-1318133748 cc: @HeartSaVioR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.