andygrove opened a new issue, #253:
URL: https://github.com/apache/datafusion-spark/issues/253
Part of #176 and apache/datafusion-comet#7300.
### What the function does
`transform_keys(expr, func)` returns a map whose keys are `func(k, v)` for
each entry of the map `expr`. The values
are unchanged. The result type is `MAP<func.dataType, valueType>` with the
input's `valueContainsNull`. A NULL map
gives NULL, and the result is nullable if the input is. The lambda takes
exactly 2 parameters: `k` (non-null) and
`v` (nullable when `valueContainsNull` is true).
The new key type must be a valid map key, so `checkInputDataTypes` uses
`TypeUtils.checkForMapKeyType`. A key type
that is or contains a `MAP` fails with
`DATATYPE_MISMATCH.INVALID_MAP_KEY_TYPE`. `expr` must be a `MAP`. A
non-lambda `func` is wrapped in a hidden lambda. Available since Spark
3.0.0. No aliases.
The result is built with `ArrayBasedMapBuilder`, which means:
- a NULL produced key fails with `NULL_MAP_KEY`;
- duplicate produced keys follow `spark.sql.mapKeyDedupPolicy`. The default
`EXCEPTION` fails with
`DUPLICATED_MAP_KEY` (`key`, `mapKeyDedupPolicy` parameters). Under
`LAST_WIN` the key keeps the position of its
first occurrence and takes the value of its last occurrence;
- from 4.0, float and double keys are normalized (`-0.0` becomes `0.0`, and
NaN values are unified) unless
`spark.sql.legacy.disableMapKeyNormalization=true`.
### Examples
```sql
SELECT transform_keys(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k,
v) -> k + 1);
-- {2:1,3:2,4:3}
SELECT transform_keys(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k,
v) -> k + v);
-- {2:1,4:2,6:3}
SELECT transform_keys(map('a', 1, 'b', 2), 42);
-- error DUPLICATED_MAP_KEY
SELECT transform_keys(map('a', 1, 'b', 2), map('new', 'key'));
-- error DATATYPE_MISMATCH.INVALID_MAP_KEY_TYPE
SELECT transform_keys(map('a', null), (k, v) -> v);
-- error NULL_MAP_KEY
```
These come from the `ExpressionDescription`,
`higher-order-functions.sql.out` and `DataFrameFunctionsSuite`.
`HigherOrderFunctionsSuite` adds:
| map | lambda | result |
|---|---|---|
| `{1:1, 2:2, 3:3, 4:4}` | `k % 3` | `DUPLICATED_MAP_KEY` (key `1`) |
| `{1:1, 2:2, 3:3, 4:4}` | `k % 3`, `LAST_WIN` | `{1:4, 2:2, 0:3}` |
| `{1:1, 2:null, 3:3}` | `k + 1` | `{2:1, 3:null, 4:3}` |
| `{"a":"xy", "bb":"yz", "ccc":null}` | `length(k) + 1` | `{2:"xy", 3:"yz",
4:null}` |
| `{}` | any | `{}` |
| NULL | any | NULL |
### Edge cases to test
- A NULL map gives NULL. An empty map gives `{}`, and no error is possible.
- A lambda that returns NULL for any entry gives `NULL_MAP_KEY`, even under
`LAST_WIN`.
- Duplicate produced keys under both `spark.sql.mapKeyDedupPolicy` values.
Check the error parameters
(`"key" -> "1"`, `"mapKeyDedupPolicy" ->
"\"spark.sql.mapKeyDedupPolicy\""`). Under `LAST_WIN`, check both the
position and the value: `{1:4, 2:2, 0:3}` above. The datafusion-spark
config is
`datafusion.spark.map_key_dedup_policy`.
- Float and double produced keys, for example `(k, v) -> if(k = 1, 0.0D,
-0.0D)` on a two-entry map. By code
reading, 4.0+ normalization makes `0.0` and `-0.0` collide, so the dedup
policy applies (`DUPLICATED_MAP_KEY` by
default). In 3.4 and 3.5 they are distinct keys. Also test `NaN`.
- Chained calls: `transform_keys(transform_keys(m, (k, v) -> k + 1), (k, v)
-> k + v)` gives `{3:1, 5:2, 7:3, 9:4}`
for `{1:1, 2:2, 3:3, 4:4}`.
- Key-type change (string to int, as in the table above). Struct and array
keys are allowed. Map keys are rejected
at analysis time.
- Lambda arity: `k -> k` fails with
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH` (expected 1, actual 2), and so
does `(k, v, x) -> ...`. DataFusion's planner allows a lambda to declare
fewer parameters than offered, so this check
has to be explicit.
- Captured outer columns, nondeterministic key expressions, and ANSI errors
inside the lambda. With ANSI on,
`k + 1` can overflow and raise `ARITHMETIC_OVERFLOW`.
- A non-map argument (`transform_keys(array(...), ...)`) fails with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`.
### Differences across Spark versions
- 4.0 adds float and double key normalization to `ArrayBasedMapBuilder`
(`spark.sql.legacy.disableMapKeyNormalization`,
default false).
- 3.5 marks `TransformKeys` as `stateful`, because the map builder is
reused. This is internal only.
- ANSI mode is on by default from 4.0.
### Implementation notes
- Spark: `TransformKeys` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
The builder is
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ArrayBasedMapBuilder.scala`.
- Infrastructure. DataFusion's higher-order function framework
(`HigherOrderUDFImpl` in
`datafusion/expr/src/higher_order_function.rs`, and
`LambdaArgument::evaluate` with `spread_captures` for captured
columns, with `config_options` available in `HigherOrderFunctionArgs`) is
on `branch-56`, which this crate uses.
- DataFusion core has no higher-order map functions. Evaluate the lambda
once over the flattened keys and values of
the `MapArray` entries, then rebuild the map with deduplication.
- In this crate, `crates/spark/src/function/map/utils.rs` already implements
Spark's dedup policy
(`map_from_keys_values_offsets_nulls` / `map_deduplicate_keys`, which
raise `[DUPLICATED_MAP_KEY]` or keep the last
value). It is wired to `datafusion.spark.map_key_dedup_policy` and used by
`map_from_arrays`,
`map_from_entries` and `str_to_map`. Reuse it, and check that it also
enforces `NULL_MAP_KEY` and the 4.0+ float
normalization.
- `crates/spark/src/function/lambda/mod.rs` is an empty `ScalarUDF`
placeholder. It needs a `HigherOrderUDF` list and
registration through `register_higher_order_function` /
`SessionStateBuilder::higher_order_functions`.
- Add `crates/sqllogictest/test_files/lambda/transform_keys.slt` and
document the function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]