cashmand commented on PR #45703: URL: https://github.com/apache/spark/pull/45703#issuecomment-2018438251
Hi @cloud-fan and @dongjoon-hyun, I updated the title, and made a few more changes: 1) Refactored the Variant checks into a separate function, similar to what already exists for scalar and struct types. 2) Added a few more test cases. 3) Added a check that the value and metadata fields in parquet are non-nullable. This is also one that is a bit up for debate - without this check, we should fail somewhere downstream with a null pointer exception if we encounter a null value, but it seems clearer to enforce it at the parquet metadata level. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
