[jira] [Assigned] (SPARK-40308) str_to_map should accept non-foldable delimiter arguments
[ https://issues.apache.org/jira/browse/SPARK-40308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40308: Assignee: Bruce Robbins > str_to_map should accept non-foldable delimiter arguments > - > > Key: SPARK-40308 > URL: https://issues.apache.org/jira/browse/SPARK-40308 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Bruce Robbins >Assignee: Bruce Robbins >Priority: Minor > > Currently, str_to_map requires the delimiter arguments to be foldable > expressions. For example, the following doesn't work in Spark SQL: > {noformat} > drop table if exists maptbl; > create table maptbl as select ',' as del1, ':' as del2, 'a:1,b:2,c:3' as str; > insert into table maptbl select '%' as del1, '-' as del2, 'a-1%b-2%c-3' as > str; > select str, str_to_map(str, del1, del2) from maptbl; > {noformat} > You get the following error: > {noformat} > str_to_map's delimiters must be foldable.; line 1 pos 12; > {noformat} > However, the above example SQL statements do work in Hive 2.3.9. There, you > get: > {noformat} > +--++ > | str |_c1 | > +--++ > | a:1,b:2,c:3 | {"a":"1","b":"2","c":"3"} | > | a-1%b-2%c-3 | {"a":"1","b":"2","c":"3"} | > +--++ > 2 rows selected (0.13 seconds) > {noformat} > It's unlikely that an input table would have the needed delimiters in > columns. The use-case is more likely to be something like this, where the > delimiters are determined based on some other value: > {noformat} > select > str, > str_to_map(str, ',', if(region = 0, ':', '#')) as m > from > maptbl2; > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40308) str_to_map should accept non-foldable delimiter arguments
[ https://issues.apache.org/jira/browse/SPARK-40308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40308: Assignee: Apache Spark > str_to_map should accept non-foldable delimiter arguments > - > > Key: SPARK-40308 > URL: https://issues.apache.org/jira/browse/SPARK-40308 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Bruce Robbins >Assignee: Apache Spark >Priority: Minor > > Currently, str_to_map requires the delimiter arguments to be foldable > expressions. For example, the following doesn't work in Spark SQL: > {noformat} > drop table if exists maptbl; > create table maptbl as select ',' as del1, ':' as del2, 'a:1,b:2,c:3' as str; > insert into table maptbl select '%' as del1, '-' as del2, 'a-1%b-2%c-3' as > str; > select str, str_to_map(str, del1, del2) from maptbl; > {noformat} > You get the following error: > {noformat} > str_to_map's delimiters must be foldable.; line 1 pos 12; > {noformat} > However, the above example SQL statements do work in Hive 2.3.9. There, you > get: > {noformat} > +--++ > | str |_c1 | > +--++ > | a:1,b:2,c:3 | {"a":"1","b":"2","c":"3"} | > | a-1%b-2%c-3 | {"a":"1","b":"2","c":"3"} | > +--++ > 2 rows selected (0.13 seconds) > {noformat} > It's unlikely that an input table would have the needed delimiters in > columns. The use-case is more likely to be something like this, where the > delimiters are determined based on some other value: > {noformat} > select > str, > str_to_map(str, ',', if(region = 0, ':', '#')) as m > from > maptbl2; > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40308) str_to_map should accept non-foldable delimiter arguments
[ https://issues.apache.org/jira/browse/SPARK-40308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40308: Assignee: (was: Apache Spark) > str_to_map should accept non-foldable delimiter arguments > - > > Key: SPARK-40308 > URL: https://issues.apache.org/jira/browse/SPARK-40308 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Bruce Robbins >Priority: Minor > > Currently, str_to_map requires the delimiter arguments to be foldable > expressions. For example, the following doesn't work in Spark SQL: > {noformat} > drop table if exists maptbl; > create table maptbl as select ',' as del1, ':' as del2, 'a:1,b:2,c:3' as str; > insert into table maptbl select '%' as del1, '-' as del2, 'a-1%b-2%c-3' as > str; > select str, str_to_map(str, del1, del2) from maptbl; > {noformat} > You get the following error: > {noformat} > str_to_map's delimiters must be foldable.; line 1 pos 12; > {noformat} > However, the above example SQL statements do work in Hive 2.3.9. There, you > get: > {noformat} > +--++ > | str |_c1 | > +--++ > | a:1,b:2,c:3 | {"a":"1","b":"2","c":"3"} | > | a-1%b-2%c-3 | {"a":"1","b":"2","c":"3"} | > +--++ > 2 rows selected (0.13 seconds) > {noformat} > It's unlikely that an input table would have the needed delimiters in > columns. The use-case is more likely to be something like this, where the > delimiters are determined based on some other value: > {noformat} > select > str, > str_to_map(str, ',', if(region = 0, ':', '#')) as m > from > maptbl2; > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org