[
https://issues.apache.org/jira/browse/SPARK-48937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan reassigned SPARK-48937:
-----------------------------------
Assignee: Uroš Bojanić
> Fix collation support for the StringToMap expression (binary & lowercase
> collation only)
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-48937
> URL: https://issues.apache.org/jira/browse/SPARK-48937
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Uroš Bojanić
> Assignee: Uroš Bojanić
> Priority: Major
> Labels: pull-request-available
>
> Enable collation support for *StringToMap* built-in string function in Spark
> ({*}str_to_map{*}). First confirm what is the expected behaviour for this
> function when given collated strings, and then move on to implementation and
> testing. You will find this expression in the *complexTypeCreator.scala*
> file. However, this experssion is currently implemented as pass-through
> function, which is wrong because it doesn't provide appropriate collation
> awareness for non-default delimiters.
>
> Example 1.
> {code:java}
> SELECT str_to_map('a:1,b:2,c:3', ',', ':');{code}
> This query will give the correct result, regardless of the collation.
> {code:java}
> {"a":"1","b":"2","c":"3"}{code}
>
> Example 2.
> {code:java}
> SELECT str_to_map('ay1xby2xcy3', 'X', 'Y');{code}
> This query will give the *incorrect* result, under UTF8_LCASE collation. The
> correct result should be:
> {code:java}
> {"a":"1","b":"2","c":"3"}{code}
>
> Update the corresponding E2E SQL tests (CollationSQLExpressionsSuite) to
> reflect how this function should be used with collation in SparkSQL, and feel
> free to use your chosen Spark SQL Editor to experiment with the existing
> functions to learn more about how they work. In addition, look into the
> possible use-cases and implementation of similar functions within other other
> open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].
>
> The goal for this Jira ticket is to implement the *StringToMap* expression so
> that it supports UTF8_BINARY and UTF8_LCASE collations (i.e.
> StringTypeBinaryLcase). To understand what changes were introduced in order
> to enable full collation support for other existing functions in Spark, take
> a look at the related Spark PRs and Jira tickets for completed tasks in this
> parent (for example: https://issues.apache.org/jira/browse/SPARK-47414).
>
> Read more about ICU [Collation Concepts|http://example.com/] and
> [Collator|http://example.com/] class. Also, refer to the Unicode Technical
> Standard for string
> [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
