andygrove opened a new issue, #251:
URL: https://github.com/apache/datafusion-spark/issues/251
Part of #176 and apache/datafusion-comet#7300.
### What the function does
`map_filter(expr, func)` keeps the entries of the map `expr` for which the
lambda `(k, v) -> BOOLEAN` returns
`true`. It drops entries where the lambda returns `false` or NULL. Entries
keep their original order. The result
type is the input map type, unchanged. A NULL map gives NULL, and the result
is nullable if the input is nullable.
The key variable is non-nullable, and the value variable has the map's
`valueContainsNull` nullability. The lambda
must take exactly 2 parameters and return `BOOLEAN`. A non-boolean body, or
a non-lambda literal such as
`map_filter(m, 1)`, fails with `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`.
`expr` must be a `MAP`. Available since
Spark 3.0.0. No aliases.
### Examples
```sql
SELECT map_filter(map(1, 0, 2, 2, 3, -1), (k, v) -> k > v);
-- {1:0,3:-1}
```
From the `ExpressionDescription`. `HigherOrderFunctionsSuite` adds the
following cases:
| map | lambda | result |
|---|---|---|
| `{1:0, 2:10, 3:-1}` | `k > v` | `{1:0, 3:-1}` |
| `{1:null, 2:10, 3:null}` | `k > v` | `{}` (NULL predicate drops the entry)
|
| `{1:0, 2:10, 3:-1}` | `v IS NULL` | `{}` |
| `{1:null, 2:10, 3:null}` | `v IS NULL` | `{1:null, 3:null}` |
| `{"abcdf":5, "abc":10, "":0}` | `length(k) = v` | `{"abcdf":5, "":0}` |
| `{"abcdf":5, "abc":10, "":null}` | `length(k) = v` | `{"abcdf":5}` |
| `{1:[0,1,2], 2:null, -3:[-1,0,-2,3]}` | `size(v) + k > 3` | `{1:[0,1,2]}` |
| NULL map | any | NULL |
### Edge cases to test
- A NULL map gives NULL. An empty map gives `{}`. A map whose entries are
all filtered out gives `{}`, not NULL.
- A NULL predicate result is treated as `false`. Spark checks
`f.eval(...).asInstanceOf[Boolean]`, and a NULL unboxes
to `false`.
- NULL values in the map.
- In the `size(v) + k > 3` case, `size(NULL)` is `-1` when ANSI is off
(`spark.sql.legacy.sizeOfNull`, default true)
and NULL when ANSI is on. The `2:null` entry is dropped either way,
because `1 > 3` is `false` and a NULL predicate
also drops the entry. Test both settings.
- Entry order is preserved, and duplicate keys cannot appear (the input map
is already deduplicated).
- Captured outer columns: `map_filter(m, (k, v) -> v > threshold_col)`.
- Nondeterministic predicates, evaluated once per entry.
- Errors inside the predicate under ANSI mode (`v / 0 > 1`), raised for
every evaluated entry.
- Lambda arity: `k -> ...` and `(k, v, x) -> ...` both fail with
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`.
DataFusion's planner accepts lambdas with fewer parameters than offered,
so the Spark arity check has to be
enforced explicitly.
- Key types: integers, strings, binary, structs, and floating point
(including `NaN` and `-0.0` keys).
- Sliced `MapArray` input, where offsets do not start at 0.
### Differences across Spark versions
- None found in `MapFilter` between 3.4 and 4.2. The literal-argument error
tests were added to the golden file in
4.1, which only adds tests. ANSI mode is on by default from 4.0.
### Implementation notes
- Spark: `MapFilter` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
It builds the result with `ArrayBasedMapData` directly, with no dedup and
no key normalization.
- Infrastructure. DataFusion's higher-order function framework
(`HigherOrderUDFImpl` in
`datafusion/expr/src/higher_order_function.rs`, and
`LambdaArgument::evaluate` with `spread_captures` for captured
columns) is on `branch-56`, which this crate uses. `lambda_parameters`
should offer `[key (non-null), value]`.
- DataFusion core has no higher-order map functions. `array_filter`
(`datafusion/functions-nested/src/array_filter.rs`)
is the closest model: evaluate the predicate over the flattened keys and
values of the `MapArray` entries, then
`filter` the entries struct and rebuild the offsets. Its handling of NULL
predicates (treated as not matching) and
of values under NULL rows matches what is needed here.
- This crate's `crates/spark/src/function/lambda/mod.rs` is an empty
`ScalarUDF` placeholder. It needs a
`HigherOrderUDF` list and registration through
`register_higher_order_function` /
`SessionStateBuilder::higher_order_functions`.
`crates/spark/src/function/map/utils.rs` has `MapArray`
construction helpers.
- Add `crates/sqllogictest/test_files/lambda/map_filter.slt` and document
the function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]