[
https://issues.apache.org/jira/browse/SPARK-51975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniël Tom updated SPARK-51975:
-------------------------------
Description:
Currently, Spark requires a two-step process to create Variant maps from arrays:
# Create MapType using map_from_arrays/map_from_entries
# Convert to Variant using to_variant_object()
A direct variant_from_arrays/variant_from_entries implementation would simplify
to a one-step process, reduces memory usage, improve CPU efficiency.
Particularly benefit workloads with large map structures, high-volume.
Example:
||Function||Description||Example||
|variant_from_entries|Transforms an array of key-value pair entries (structs
with two fields) into a variant object|{code:java}
SELECT variant_from_entries(array(struct(1, 'a'), struct(2, 'b')));
{1 -> a, 2 -> b}{code}|
|variant_from_arrays|Transforms two arrays of
keys and values respectively, into a new variant object.|{code:java}
SELECT variant_from_arrays(array(1.0, 3.0), array('2', '4'));
{1.0 -> 2, 3.0 -> 4}{code}|
was:
Currently, Spark requires a two-step process to create Variant maps from arrays:
# Create MapType using map_from_arrays/map_from_entries
# Convert to Variant using to_variant_object()
A direct variant_from_arrays/variant_from_entries implementation would simplify
to a one-step process, reduces memory usage, improve CPU efficiency.
Particularly benefit workloads with large map structures, high-volume.
Example:
||Function||Description||Example||
|variant_from_entries|Transforms an array of key-value pair entries (structs
with two fields) into a variant object|{code:java}
SELECT variant_from_entries(array(struct(1, 'a'), struct(2, 'b')));
{1 -> a, 2 -> b}{code}|
|variant_from_arrays|Transforms two arrays of
keys and values respectively, into a new variant object.|{code:java}
SELECT map_from_arrays(array(1.0, 3.0), array('2', '4'));
{1.0 -> 2, 3.0 -> 4}{code}|
> Support variant_from_arrays and variant_from_entries
> -----------------------------------------------------
>
> Key: SPARK-51975
> URL: https://issues.apache.org/jira/browse/SPARK-51975
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Daniël Tom
> Priority: Major
>
>
> Currently, Spark requires a two-step process to create Variant maps from
> arrays:
> # Create MapType using map_from_arrays/map_from_entries
> # Convert to Variant using to_variant_object()
> A direct variant_from_arrays/variant_from_entries implementation would
> simplify to a one-step process, reduces memory usage, improve CPU efficiency.
> Particularly benefit workloads with large map structures, high-volume.
> Example:
> ||Function||Description||Example||
> |variant_from_entries|Transforms an array of key-value pair entries (structs
> with two fields) into a variant object|{code:java}
> SELECT variant_from_entries(array(struct(1, 'a'), struct(2, 'b')));
> {1 -> a, 2 -> b}{code}|
> |variant_from_arrays|Transforms two arrays of
> keys and values respectively, into a new variant object.|{code:java}
> SELECT variant_from_arrays(array(1.0, 3.0), array('2', '4'));
> {1.0 -> 2, 3.0 -> 4}{code}|
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
