matthewgapp commented on issue #7636: URL: https://github.com/apache/arrow-datafusion/issues/7636#issuecomment-1740171545
@alamb, more of a meta thought, but with #4815, but I'm concerned that all of these "bugs" will be go unnoticed (unless they're caught in the DF application logic like [here](https://github.com/apache/arrow-datafusion/blob/4b2b7dcfc63abfc03b0279abe122c5bdfcca5275/datafusion/physical-plan/src/insert.rs#L286)), potentially creating incorrect behavior. I think it could be helpful to have something like your strict mode (potentially a compiler flag). But I'm still ramping on this ecosystem so not sure who should determine the correctness of a schema and/or when that correctness should be asserted. However, it does **_feel_** like anytime that record batches are stuck together and assumed to be a congruent, continuous thing (like a table or stream), that nullability between these batches should be **_consistent_** (or at least a superset of the nullability of containing thing (table or stream). For example, for the purposes of DF, it seems appropriate that non-nullable batches would always be acceptable in a table that is nullable. The inverse is not true. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
