rich7420 opened a new issue, #27075: URL: https://github.com/apache/datafusion/issues/27075
### Is your feature request related to a problem or challenge? Spark's DataFrame.na.drop uses AtLeastNNonNulls to retain rows with at least n values that are neither NULL nor NaN. Comet added a native implementation in apache/datafusion-comet#6180. Making that expression available in datafusion-spark would let downstream engines reuse it. ### Describe the solution you'd like Port the implementation and its tests to datafusion/spark/src/function/predicate as a public SparkAtLeastNNonNullsExpr. Callers would supply a fixed threshold and already-planned physical children. The expression would return a non-null Boolean result and stop evaluating a row's remaining children once the threshold is reached. This would use the same direct-construction approach as SparkFromUtcTimestampExpr. AtLeastNNonNulls is an internal Catalyst predicate with no Spark SQL function name, so consumers would construct the expression directly. Planning and any shared-subexpression hoisting remain the caller's responsibility. Would a standalone PhysicalExpr be appropriate for this crate? The proposed tests cover NULL/NaN counting, scalar and array inputs, dictionary values, bitmap slices and boundaries, outer validity of complex values, and skipped versus required child errors. A rustdoc example would show direct construction. ### Describe alternatives you've considered An ordinary scalar UDF receives already-evaluated arguments, so it cannot preserve conditional evaluation of computed children. A directly constructed PhysicalExpr reuses the existing extension API without introducing lazy-UDF infrastructure. ### Additional context - Source implementation: https://github.com/apache/datafusion-comet/pull/6180 - Related Spark-compatible functions epic: https://github.com/apache/datafusion/issues/15914 - Direct-construction precedent: https://github.com/apache/datafusion/blob/181a87af61cef18139e436ad4ab5726602bf2425/datafusion/spark/src/function/datetime/from_utc_timestamp.rs#L211-L240 ### Willingness to contribute I plan to contribute to this feature independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
