[
https://issues.apache.org/jira/browse/SPARK-51975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniël Tom updated SPARK-51975:
-------------------------------
Description:
Currently, Spark requires a two-step process to create Variant maps from arrays:
# Create MapType using map_from_arrays/map_from_entries
# Convert to Variant using to_variant_object()
A direct variant_from_arrays/variant_from_entries implementation reduces memory
usage, and improves CPU efficiency. Particularly benefit workloads with large
map structures or high-volume of data.
{*}Example{*}:
||Function||Description||Example||
|variant_from_entries|Transforms an array of key-value pair entries (structs
with two fields) into a variant object|{code:java}
SELECT variant_from_entries(array(struct('1', 'a'), struct('2',
'b')));{code}{code:java}
{"1":"a", "2":"b"}{code}|
|variant_from_arrays|Transforms two arrays of
keys and values respectively, into a new variant object.|{code:java}
SELECT variant_from_arrays(array('1', '2'), array('a', 'b')); {code}{code:java}
{"1":"a", "2":"b"}{code}|
was:
Currently, Spark requires a two-step process to create Variant maps from arrays:
# Create MapType using map_from_arrays/map_from_entries
# Convert to Variant using to_variant_object()
A direct variant_from_arrays/variant_from_entries implementation would simplify
to a one-step process, reduces memory usage, improve CPU efficiency.
Particularly benefit workloads with large map structures, high-volume.
Example:
||Function||Description||Example||
|variant_from_entries|Transforms an array of key-value pair entries (structs
with two fields) into a variant object|{code:java}
SELECT variant_from_entries(array(struct('1', 'a'), struct('2',
'b')));{code}{code:java}
{"1":"a", "2":"b"}{code}|
|variant_from_arrays|Transforms two arrays of
keys and values respectively, into a new variant object.|{code:java}
SELECT variant_from_arrays(array('1', '2'), array('a', 'b')); {code}{code:java}
{"1":"a", "2":"b"}{code}|
> Support variant_from_arrays and variant_from_entries
> -----------------------------------------------------
>
> Key: SPARK-51975
> URL: https://issues.apache.org/jira/browse/SPARK-51975
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Daniël Tom
> Priority: Major
>
>
> Currently, Spark requires a two-step process to create Variant maps from
> arrays:
> # Create MapType using map_from_arrays/map_from_entries
> # Convert to Variant using to_variant_object()
> A direct variant_from_arrays/variant_from_entries implementation reduces
> memory usage, and improves CPU efficiency. Particularly benefit workloads
> with large map structures or high-volume of data.
> {*}Example{*}:
> ||Function||Description||Example||
> |variant_from_entries|Transforms an array of key-value pair entries (structs
> with two fields) into a variant object|{code:java}
> SELECT variant_from_entries(array(struct('1', 'a'), struct('2',
> 'b')));{code}{code:java}
> {"1":"a", "2":"b"}{code}|
> |variant_from_arrays|Transforms two arrays of
> keys and values respectively, into a new variant object.|{code:java}
> SELECT variant_from_arrays(array('1', '2'), array('a', 'b'));
> {code}{code:java}
> {"1":"a", "2":"b"}{code}|
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
