andygrove opened a new issue, #251:
URL: https://github.com/apache/datafusion-spark/issues/251

   Part of #176 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `map_filter(expr, func)` keeps the entries of the map `expr` for which the 
lambda `(k, v) -> BOOLEAN` returns
   `true`. It drops entries where the lambda returns `false` or NULL. Entries 
keep their original order. The result
   type is the input map type, unchanged. A NULL map gives NULL, and the result 
is nullable if the input is nullable.
   The key variable is non-nullable, and the value variable has the map's 
`valueContainsNull` nullability. The lambda
   must take exactly 2 parameters and return `BOOLEAN`. A non-boolean body, or 
a non-lambda literal such as
   `map_filter(m, 1)`, fails with `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`. 
`expr` must be a `MAP`. Available since
   Spark 3.0.0. No aliases.
   
   ### Examples
   
   ```sql
   SELECT map_filter(map(1, 0, 2, 2, 3, -1), (k, v) -> k > v);
   -- {1:0,3:-1}
   ```
   
   From the `ExpressionDescription`. `HigherOrderFunctionsSuite` adds the 
following cases:
   
   | map | lambda | result |
   |---|---|---|
   | `{1:0, 2:10, 3:-1}` | `k > v` | `{1:0, 3:-1}` |
   | `{1:null, 2:10, 3:null}` | `k > v` | `{}` (NULL predicate drops the entry) 
|
   | `{1:0, 2:10, 3:-1}` | `v IS NULL` | `{}` |
   | `{1:null, 2:10, 3:null}` | `v IS NULL` | `{1:null, 3:null}` |
   | `{"abcdf":5, "abc":10, "":0}` | `length(k) = v` | `{"abcdf":5, "":0}` |
   | `{"abcdf":5, "abc":10, "":null}` | `length(k) = v` | `{"abcdf":5}` |
   | `{1:[0,1,2], 2:null, -3:[-1,0,-2,3]}` | `size(v) + k > 3` | `{1:[0,1,2]}` |
   | NULL map | any | NULL |
   
   ### Edge cases to test
   
   - A NULL map gives NULL. An empty map gives `{}`. A map whose entries are 
all filtered out gives `{}`, not NULL.
   - A NULL predicate result is treated as `false`. Spark checks 
`f.eval(...).asInstanceOf[Boolean]`, and a NULL unboxes
     to `false`.
   - NULL values in the map.
   - In the `size(v) + k > 3` case, `size(NULL)` is `-1` when ANSI is off 
(`spark.sql.legacy.sizeOfNull`, default true)
     and NULL when ANSI is on. The `2:null` entry is dropped either way, 
because `1 > 3` is `false` and a NULL predicate
     also drops the entry. Test both settings.
   - Entry order is preserved, and duplicate keys cannot appear (the input map 
is already deduplicated).
   - Captured outer columns: `map_filter(m, (k, v) -> v > threshold_col)`.
   - Nondeterministic predicates, evaluated once per entry.
   - Errors inside the predicate under ANSI mode (`v / 0 > 1`), raised for 
every evaluated entry.
   - Lambda arity: `k -> ...` and `(k, v, x) -> ...` both fail with 
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`.
     DataFusion's planner accepts lambdas with fewer parameters than offered, 
so the Spark arity check has to be
     enforced explicitly.
   - Key types: integers, strings, binary, structs, and floating point 
(including `NaN` and `-0.0` keys).
   - Sliced `MapArray` input, where offsets do not start at 0.
   
   ### Differences across Spark versions
   
   - None found in `MapFilter` between 3.4 and 4.2. The literal-argument error 
tests were added to the golden file in
     4.1, which only adds tests. ANSI mode is on by default from 4.0.
   
   ### Implementation notes
   
   - Spark: `MapFilter` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
     It builds the result with `ArrayBasedMapData` directly, with no dedup and 
no key normalization.
   - Infrastructure. DataFusion's higher-order function framework 
(`HigherOrderUDFImpl` in
     `datafusion/expr/src/higher_order_function.rs`, and 
`LambdaArgument::evaluate` with `spread_captures` for captured
     columns) is on `branch-56`, which this crate uses. `lambda_parameters` 
should offer `[key (non-null), value]`.
   - DataFusion core has no higher-order map functions. `array_filter` 
(`datafusion/functions-nested/src/array_filter.rs`)
     is the closest model: evaluate the predicate over the flattened keys and 
values of the `MapArray` entries, then
     `filter` the entries struct and rebuild the offsets. Its handling of NULL 
predicates (treated as not matching) and
     of values under NULL rows matches what is needed here.
   - This crate's `crates/spark/src/function/lambda/mod.rs` is an empty 
`ScalarUDF` placeholder. It needs a
     `HigherOrderUDF` list and registration through 
`register_higher_order_function` /
     `SessionStateBuilder::higher_order_functions`. 
`crates/spark/src/function/map/utils.rs` has `MapArray`
     construction helpers.
   - Add `crates/sqllogictest/test_files/lambda/map_filter.slt` and document 
the function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to