rich7420 opened a new pull request, #27078: URL: https://github.com/apache/datafusion/pull/27078
## Which issue does this PR close? Closes #27075. Related to #15914. ## Rationale for this change Port the native predicate from [apache/datafusion-comet#6180](https://github.com/apache/datafusion-comet/pull/6180) so downstream engines can reuse it for Spark's DataFrame.na.drop. ## What changes are included in this PR? Add SparkAtLeastNNonNullsExpr for direct construction from a fixed threshold and planned physical children. It returns a non-null Boolean, excludes NULL/NaN values, and stops evaluating a row's remaining children once the threshold is reached. Callers remain responsible for shared-subexpression hoisting. The port includes the bitmap counting paths, correctness tests, a usage example and a 31-case Criterion benchmark. ## What is the testing strategy for this PR? - `cargo test --profile ci -p datafusion-spark`: 332 unit tests and 4 doctests passed; 1 existing doctest ignored. - Extended workspace tests with Avro, JSON, backtraces, `extended_tests`, recursive protection and Parquet encryption, excluding the CLI/examples/benchmarks: 13,487 Rust tests passed, 8 ignored; 547 SQLLogicTest files completed. - Workspace Clippy with all targets/features and warnings denied, formatting, license/dependency checks, and Rust/HTML documentation builds passed. - The Display and SQL-formatting assertions fail with the original Debug-based formatting and pass after restoring the fix. - All 31 `release-nonlto` benchmark cases completed with answer assertions. Two independent comparisons with reversed strategy order favored the bitmap counter in all 11 general-n column-input cases. These are expression-only measurements on an M3 Pro. Package tests, extended workspace tests and lint checks were run on base `ea33cce29`. Benchmark comparisons used base `181a87af6` with the same feature diff. No end-to-end Spark benchmark was run. ## Are there any user-facing changes? A new public Rust PhysicalExpr is available for direct construction. There is no Spark SQL function name for this internal Catalyst predicate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
