alamb commented on code in PR #18444:
URL: https://github.com/apache/datafusion/pull/18444#discussion_r2487826185
##########
datafusion/physical-expr/src/expressions/case.rs:
##########
@@ -290,6 +293,84 @@ fn filter_array(
filter.filter(array)
}
+fn merge(
Review Comment:
This looks like an implementation of `zip` to me (rather than `merge`). It
seems like it would be better to use consistent terminology if this is indeed
the case
This looks very similar to the fancy new code that @rluvaton added to arrow
for `zip` with scalars (will be released in arrow 57.1.0):
- https://github.com/apache/arrow-rs/pull/8653
If you agree it is the same, perhaps we can either avoid adding this method
to DataFusion or else we can add a comment that says we can revert to using
just `zip` once https://github.com/apache/arrow-rs/pull/8653 is available
##########
datafusion/physical-expr/src/expressions/case.rs:
##########
@@ -964,24 +1064,38 @@ impl CaseBody {
&self,
batch: &RecordBatch,
when_value: &BooleanArray,
- return_type: &DataType,
) -> Result<ColumnarValue> {
- let then_value = self.when_then_expr[0]
- .1
- .evaluate_selection(batch, when_value)?
- .into_array(batch.num_rows())?;
+ let when_value = match when_value.null_count() {
+ 0 => Cow::Borrowed(when_value),
+ _ => {
+ // `prep_null_mask_filter` is required to ensure null is
treated as false
+ Cow::Owned(prep_null_mask_filter(when_value))
+ }
+ };
+
+ let optimize_filter = batch.num_columns() > 1;
Review Comment:
It might also be worth checking if there are any nested types (e.g.
structarrays) and optimize the filter in that case too -- this is done
elsewhere (maybe in the filter kernel itself 🤔 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
