Daniël Tom created SPARK-51975:
----------------------------------
Summary: Support variant_from_arrays and variant_from_entries
Key: SPARK-51975
URL: https://issues.apache.org/jira/browse/SPARK-51975
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 4.1.0
Reporter: Daniël Tom
Currently, Spark requires a two-step process to create Variant maps from arrays:
# Create MapType using map_from_arrays/map_from_entries
# Convert to Variant using to_variant_object()
A direct variant_from_arrays/variant_from_entries implementation would simplify
to a one-step process, reduces memory usage, improve CPU efficiency.
Particularly benefit workloads with large map structures, high-volume.
Example:
||Function||Description||Example||
|variant_from_entries|Transforms an array of key-value pair entries (structs
with two fields) into a variant object|{code:java}
SELECT variant_from_entries(array(struct(1, 'a'), struct(2, 'b')));
{1 -> a, 2 -> b}{code}|
|variant_from_arrays|Transforms two arrays of
keys and values respectively, into a new variant object.|{code:java}
SELECT map_from_arrays(array(1.0, 3.0), array('2', '4'));
{1.0 -> 2, 3.0 -> 4}{code}|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
