peterxcli opened a new issue, #20357:
URL: https://github.com/apache/datafusion/issues/20357
### Is your feature request related to a problem or challenge?
neither datafusion-spark nor datafusion-comet expose the map expression
implementation. so when you try to use map expression in spark sql, you will
get:
```sql
Expected only Comet native operators, but found Project.
plan: Project [COMET: map is not supported]
datafusion hasnt had the map function, we should add one there before
supporting it in comet.
```
https://github.com/apache/datafusion-comet/issues/3386
### Describe the solution you'd like
- Implement `ScalarUDFImpl` trait for new Map struct and expose it to
datafusion-spark map module
- use `comparison_coercion` to automatically casting when returning type
- use `make_scalar_function`
- use the `map_from_keys_values_offsets_nulls` to unify spark specific logic
and code reusing
- transform the concat flatted keys and values with reorder_indices and
arrow take
### Describe alternatives you've considered
there are already existing map expression in datafusion nested, but to align
with rules and reuse the code in the
[spark/src/function/map/utils.rs](https://github.com/peterxcli/datafusion/blob/main/datafusion/spark/src/function/map/utils.rs),
I still think it's better to create a new implementation for spark map
expression
https://github.com/apache/datafusion/blob/6746007826ebd3fcb5614bf87183674435bbb134/datafusion/functions-nested/src/map.rs#L240-L301
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
