[
https://issues.apache.org/jira/browse/SPARK-40308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-40308.
------------------------------
Fix Version/s: 3.4.0
Resolution: Fixed
Issue resolved by pull request 37763
[https://github.com/apache/spark/pull/37763]
> str_to_map should accept non-foldable delimiter arguments
> ---------------------------------------------------------
>
> Key: SPARK-40308
> URL: https://issues.apache.org/jira/browse/SPARK-40308
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Bruce Robbins
> Assignee: Bruce Robbins
> Priority: Minor
> Fix For: 3.4.0
>
>
> Currently, str_to_map requires the delimiter arguments to be foldable
> expressions. For example, the following doesn't work in Spark SQL:
> {noformat}
> drop table if exists maptbl;
> create table maptbl as select ',' as del1, ':' as del2, 'a:1,b:2,c:3' as str;
> insert into table maptbl select '%' as del1, '-' as del2, 'a-1%b-2%c-3' as
> str;
> select str, str_to_map(str, del1, del2) from maptbl;
> {noformat}
> You get the following error:
> {noformat}
> str_to_map's delimiters must be foldable.; line 1 pos 12;
> {noformat}
> However, the above example SQL statements do work in Hive 2.3.9. There, you
> get:
> {noformat}
> +--------------+----------------------------+
> | str | _c1 |
> +--------------+----------------------------+
> | a:1,b:2,c:3 | {"a":"1","b":"2","c":"3"} |
> | a-1%b-2%c-3 | {"a":"1","b":"2","c":"3"} |
> +--------------+----------------------------+
> 2 rows selected (0.13 seconds)
> {noformat}
> It's unlikely that an input table would have the needed delimiters in
> columns. The use-case is more likely to be something like this, where the
> delimiters are determined based on some other value:
> {noformat}
> select
> str,
> str_to_map(str, ',', if(region = 0, ':', '#')) as m
> from
> maptbl2;
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
