[jira] [Assigned] (SPARK-47617) Add TPC-DS testing infrastructure for collations
[ https://issues.apache.org/jira/browse/SPARK-47617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47617: --- Assignee: Nikola Mandic > Add TPC-DS testing infrastructure for collations > > > Key: SPARK-47617 > URL: https://issues.apache.org/jira/browse/SPARK-47617 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Nikola Mandic >Assignee: Nikola Mandic >Priority: Major > Labels: pull-request-available > > As collation support grows across all SQL features and new collation types > are added, we need to have reliable testing model covering as many standard > SQL capabilities as possible. > We can utilize TPC-DS testing infrastructure already present in Spark. The > idea is to vary TPC-DS table string columns by adding multiple collations > with different ordering rules and case sensitivity, producing new tables. > These tables should yield the same results against predefined TPC-DS queries > for certain batches of collations. For example, when comparing query runs on > table where columns are first collated as UTF8_BINARY and then as > UTF8_BINARY_LCASE, we should be getting same results after converting to > lowercase. > Introduce new query suite which tests the described behavior with available > collations (utf8_binary and unicode) combined with case conversions > (lowercase, uppercase, randomized case for fuzzy testing). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47617) Add TPC-DS testing infrastructure for collations
[ https://issues.apache.org/jira/browse/SPARK-47617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47617: -- Assignee: (was: Apache Spark) > Add TPC-DS testing infrastructure for collations > > > Key: SPARK-47617 > URL: https://issues.apache.org/jira/browse/SPARK-47617 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Nikola Mandic >Priority: Major > Labels: pull-request-available > > As collation support grows across all SQL features and new collation types > are added, we need to have reliable testing model covering as many standard > SQL capabilities as possible. > We can utilize TPC-DS testing infrastructure already present in Spark. The > idea is to vary TPC-DS table string columns by adding multiple collations > with different ordering rules and case sensitivity, producing new tables. > These tables should yield the same results against predefined TPC-DS queries > for certain batches of collations. For example, when comparing query runs on > table where columns are first collated as UTF8_BINARY and then as > UTF8_BINARY_LCASE, we should be getting same results after converting to > lowercase. > Introduce new query suite which tests the described behavior with available > collations (utf8_binary and unicode) combined with case conversions > (lowercase, uppercase, randomized case for fuzzy testing). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47617) Add TPC-DS testing infrastructure for collations
[ https://issues.apache.org/jira/browse/SPARK-47617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47617: -- Assignee: Apache Spark > Add TPC-DS testing infrastructure for collations > > > Key: SPARK-47617 > URL: https://issues.apache.org/jira/browse/SPARK-47617 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Nikola Mandic >Assignee: Apache Spark >Priority: Major > Labels: pull-request-available > > As collation support grows across all SQL features and new collation types > are added, we need to have reliable testing model covering as many standard > SQL capabilities as possible. > We can utilize TPC-DS testing infrastructure already present in Spark. The > idea is to vary TPC-DS table string columns by adding multiple collations > with different ordering rules and case sensitivity, producing new tables. > These tables should yield the same results against predefined TPC-DS queries > for certain batches of collations. For example, when comparing query runs on > table where columns are first collated as UTF8_BINARY and then as > UTF8_BINARY_LCASE, we should be getting same results after converting to > lowercase. > Introduce new query suite which tests the described behavior with available > collations (utf8_binary and unicode) combined with case conversions > (lowercase, uppercase, randomized case for fuzzy testing). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
