cloud-fan commented on PR #37011: URL: https://github.com/apache/spark/pull/37011#issuecomment-1175058395
@ravwojdyla I'm fine to support this use case. We can use the `AssertNotNull` expression to do this. What I was against is the opposite: the dataframe infers some columns as non-nullable, but users want to forcibly mark these columns as nullable, to degrade performance. @dilipbiswal what do you think of this use case? When the column is nullable but the specified schema requires it to be non-nullabl, shall we add runtime null check to make it work? At least this can be under a config. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
