GitHub user 90degs2infty created a discussion: Drop/filter out rows containing `nulls` at the schema level
Hello everyone! Is there a way to 1. drop rows from a `DataFrame` containing `null`s in a set of selected columns while at the same time 2. informing the `DataFrame`'s schema about the fact that the set of selected columns cannot contain physical `null`s afterwards? I.e. I'd like [`is_nullable`](https://docs.rs/datafusion/latest/datafusion/common/arrow/datatypes/struct.Field.html#method.is_nullable) to return `false` for the respective `Field`s after dropping (similar to what [`fill_null`](https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.fill_null) does). One can easily achieve 1. by passing a suitable `Expr` to [`filter`](https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.filter). However, I don't know how to implement 2. Any thoughts? Thank you very much! 🙏 GitHub link: https://github.com/apache/datafusion/discussions/17034 ---- This is an automatically sent email for github@datafusion.apache.org. To unsubscribe, please send an email to: github-unsubscr...@datafusion.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
