[jira] [Resolved] (SPARK-47485) Create column with collations in dataframe API
[ https://issues.apache.org/jira/browse/SPARK-47485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47485. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45569 [https://github.com/apache/spark/pull/45569] > Create column with collations in dataframe API > -- > > Key: SPARK-47485 > URL: https://issues.apache.org/jira/browse/SPARK-47485 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark, SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stefan Kandic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Add ability to create string columns with non default collations in the > dataframe API -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47564) always throw FAILED_READ_FILE error when fail to read files
[ https://issues.apache.org/jira/browse/SPARK-47564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47564: Assignee: Wenchen Fan > always throw FAILED_READ_FILE error when fail to read files > --- > > Key: SPARK-47564 > URL: https://issues.apache.org/jira/browse/SPARK-47564 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47564) always throw FAILED_READ_FILE error when fail to read files
[ https://issues.apache.org/jira/browse/SPARK-47564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47564. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45723 [https://github.com/apache/spark/pull/45723] > always throw FAILED_READ_FILE error when fail to read files > --- > > Key: SPARK-47564 > URL: https://issues.apache.org/jira/browse/SPARK-47564 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47358) StringRepeat (all collations)
[ https://issues.apache.org/jira/browse/SPARK-47358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47358: Assignee: Milan Dankovic > StringRepeat (all collations) > - > > Key: SPARK-47358 > URL: https://issues.apache.org/jira/browse/SPARK-47358 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Uroš Bojanić >Assignee: Milan Dankovic >Priority: Major > Labels: pull-request-available > > Enable collation support for the *StringRepeat* built-in string function in > Spark. First confirm what is the expected behaviour for these functions when > given collated strings, then move on to the implementation that would enable > handling strings of all collation types. Implement the corresponding unit > tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to > reflect how this function should be used with collation in SparkSQL, and feel > free to use your chosen Spark SQL Editor to experiment with the existing > functions to learn more about how they work. In addition, look into the > possible use-cases and implementation of similar functions within other other > open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/]. > > The goal for this Jira ticket is to implement the *StringRepeat* function so > it supports all collation types currently supported in Spark. To understand > what changes were introduced in order to enable full collation support for > other existing functions in Spark, take a look at the Spark PRs and Jira > tickets for completed tasks in this parent (for example: Contains, > StartsWith, EndsWith). > > Read more about ICU [Collation Concepts|http://example.com/] and > [Collator|http://example.com/] class. Also, refer to the Unicode Technical > Standard for > [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback]. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47358) StringRepeat (all collations)
[ https://issues.apache.org/jira/browse/SPARK-47358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47358. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45615 [https://github.com/apache/spark/pull/45615] > StringRepeat (all collations) > - > > Key: SPARK-47358 > URL: https://issues.apache.org/jira/browse/SPARK-47358 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Uroš Bojanić >Assignee: Milan Dankovic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Enable collation support for the *StringRepeat* built-in string function in > Spark. First confirm what is the expected behaviour for these functions when > given collated strings, then move on to the implementation that would enable > handling strings of all collation types. Implement the corresponding unit > tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to > reflect how this function should be used with collation in SparkSQL, and feel > free to use your chosen Spark SQL Editor to experiment with the existing > functions to learn more about how they work. In addition, look into the > possible use-cases and implementation of similar functions within other other > open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/]. > > The goal for this Jira ticket is to implement the *StringRepeat* function so > it supports all collation types currently supported in Spark. To understand > what changes were introduced in order to enable full collation support for > other existing functions in Spark, take a look at the Spark PRs and Jira > tickets for completed tasks in this parent (for example: Contains, > StartsWith, EndsWith). > > Read more about ICU [Collation Concepts|http://example.com/] and > [Collator|http://example.com/] class. Also, refer to the Unicode Technical > Standard for > [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback]. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47256) Assign error classes to FILTER expression errors
[
https://issues.apache.org/jira/browse/SPARK-47256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-47256:
Assignee: David Milicevic
> Assign error classes to FILTER expression errors
>
>
> Key: SPARK-47256
> URL: https://issues.apache.org/jira/browse/SPARK-47256
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Assignee: David Milicevic
>Priority: Minor
> Labels: pull-request-available, starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_102[4-7]*
> defined in {*}common/utils/src/main/resources/error/error-classes.json{*}.
> The name should be short but complete (look at the example in
> error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47256) Assign error classes to FILTER expression errors
[
https://issues.apache.org/jira/browse/SPARK-47256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-47256.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45622
[https://github.com/apache/spark/pull/45622]
> Assign error classes to FILTER expression errors
>
>
> Key: SPARK-47256
> URL: https://issues.apache.org/jira/browse/SPARK-47256
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Assignee: David Milicevic
>Priority: Minor
> Labels: pull-request-available, starter
> Fix For: 4.0.0
>
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_102[4-7]*
> defined in {*}common/utils/src/main/resources/error/error-classes.json{*}.
> The name should be short but complete (look at the example in
> error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47506) Support all file source formats for collated data types
[ https://issues.apache.org/jira/browse/SPARK-47506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47506: Assignee: Stefan Kandic > Support all file source formats for collated data types > --- > > Key: SPARK-47506 > URL: https://issues.apache.org/jira/browse/SPARK-47506 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stefan Kandic >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47506) Support all file source formats for collated data types
[ https://issues.apache.org/jira/browse/SPARK-47506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47506. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45641 [https://github.com/apache/spark/pull/45641] > Support all file source formats for collated data types > --- > > Key: SPARK-47506 > URL: https://issues.apache.org/jira/browse/SPARK-47506 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stefan Kandic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-47429) Rename errorClass to errorCondition
[
https://issues.apache.org/jira/browse/SPARK-47429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829776#comment-17829776
]
Max Gekk commented on SPARK-47429:
--
[~nchammas] I am going to rename `class` to `condition`. Just want to confirm
that you are not working on the task. Let me know, please, I will leave this to
you.
> Rename errorClass to errorCondition
> ---
>
> Key: SPARK-47429
> URL: https://issues.apache.org/jira/browse/SPARK-47429
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Nicholas Chammas
>Priority: Minor
>
> We've agreed on the parent task to rename {{errorClass}} to align it more
> closely with the SQL standard, and take advantage of the opportunity to break
> backwards compatibility offered by the Spark version change from 3.5 to 4.0.
> This is a subtask so the changes are in their own PR and easier to review
> apart from other things.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47483) Add support for aggregation and join operations on arrays of collated strings
[
https://issues.apache.org/jira/browse/SPARK-47483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-47483:
Assignee: Nikola Mandic
> Add support for aggregation and join operations on arrays of collated strings
> -
>
> Key: SPARK-47483
> URL: https://issues.apache.org/jira/browse/SPARK-47483
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Nikola Mandic
>Assignee: Nikola Mandic
>Priority: Major
> Labels: pull-request-available
>
> Example of aggregation sequence:
> {code:java}
> create table t(a array) using parquet;
> insert into t(a) values(array('a' collate utf8_binary_lcase));
> insert into t(a) values(array('A' collate utf8_binary_lcase));
> select distinct a from t; {code}
> Example of join sequence:
> {code:java}
> create table l(a array) using parquet;
> create table r(a array) using parquet;
> insert into l(a) values(array('a' collate utf8_binary_lcase));
> insert into r(a) values(array('A' collate utf8_binary_lcase));
> select * from l join r where l.a = r.a; {code}
> Both runs should yield one row since the arrays are considered equal.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47483) Add support for aggregation and join operations on arrays of collated strings
[
https://issues.apache.org/jira/browse/SPARK-47483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-47483.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45611
[https://github.com/apache/spark/pull/45611]
> Add support for aggregation and join operations on arrays of collated strings
> -
>
> Key: SPARK-47483
> URL: https://issues.apache.org/jira/browse/SPARK-47483
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Nikola Mandic
>Assignee: Nikola Mandic
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Example of aggregation sequence:
> {code:java}
> create table t(a array) using parquet;
> insert into t(a) values(array('a' collate utf8_binary_lcase));
> insert into t(a) values(array('A' collate utf8_binary_lcase));
> select distinct a from t; {code}
> Example of join sequence:
> {code:java}
> create table l(a array) using parquet;
> create table r(a array) using parquet;
> insert into l(a) values(array('a' collate utf8_binary_lcase));
> insert into r(a) values(array('A' collate utf8_binary_lcase));
> select * from l join r where l.a = r.a; {code}
> Both runs should yield one row since the arrays are considered equal.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47479) Optimize cannot write data to relations with multiple paths error log
[ https://issues.apache.org/jira/browse/SPARK-47479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47479. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45605 [https://github.com/apache/spark/pull/45605] > Optimize cannot write data to relations with multiple paths error log > - > > Key: SPARK-47479 > URL: https://issues.apache.org/jira/browse/SPARK-47479 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: xiaoping.huang >Assignee: xiaoping.huang >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47443) Window aggregate support
[ https://issues.apache.org/jira/browse/SPARK-47443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47443. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45568 [https://github.com/apache/spark/pull/45568] > Window aggregate support > > > Key: SPARK-47443 > URL: https://issues.apache.org/jira/browse/SPARK-47443 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Aleksandar Tomic >Assignee: Aleksandar Tomic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47443) Window aggregate support
[ https://issues.apache.org/jira/browse/SPARK-47443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47443: Assignee: Aleksandar Tomic > Window aggregate support > > > Key: SPARK-47443 > URL: https://issues.apache.org/jira/browse/SPARK-47443 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Aleksandar Tomic >Assignee: Aleksandar Tomic >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47007) Add SortMap function
[ https://issues.apache.org/jira/browse/SPARK-47007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47007: Assignee: Stevo Mitric > Add SortMap function > > > Key: SPARK-47007 > URL: https://issues.apache.org/jira/browse/SPARK-47007 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stevo Mitric >Priority: Major > Labels: pull-request-available > > In order to properly support GROUP BY on a map type we need to first add the > ability to sort the map in order to do the comparisons later -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47007) Add SortMap function
[ https://issues.apache.org/jira/browse/SPARK-47007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47007. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45069 [https://github.com/apache/spark/pull/45069] > Add SortMap function > > > Key: SPARK-47007 > URL: https://issues.apache.org/jira/browse/SPARK-47007 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stevo Mitric >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > In order to properly support GROUP BY on a map type we need to first add the > ability to sort the map in order to do the comparisons later -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47256) Assign error classes to FILTER expression errors
[
https://issues.apache.org/jira/browse/SPARK-47256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47256:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_102[4-7]* defined
in {*}common/utils/src/main/resources/error/error-classes.json{*}. The name
should be short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_102[4-7]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign error classes to FILTER expression errors
>
>
> Key: SPARK-47256
> URL: https://issues.apache.org/jira/browse/SPARK-47256
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_102[4-7]*
> defined in {*}common/utils/src/main/resources/error/error-classes.json{*}.
> The name should be short but complete (look at the example in
> error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47423) Set operations should work with collated strings
[ https://issues.apache.org/jira/browse/SPARK-47423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47423. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45536 [https://github.com/apache/spark/pull/45536] > Set operations should work with collated strings > > > Key: SPARK-47423 > URL: https://issues.apache.org/jira/browse/SPARK-47423 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Aleksandar Tomic >Assignee: Aleksandar Tomic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47423) Set operations should work with collated strings
[ https://issues.apache.org/jira/browse/SPARK-47423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47423: Assignee: Aleksandar Tomic > Set operations should work with collated strings > > > Key: SPARK-47423 > URL: https://issues.apache.org/jira/browse/SPARK-47423 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Aleksandar Tomic >Assignee: Aleksandar Tomic >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47345) XML: Add XmlFunctionsSuite
[
https://issues.apache.org/jira/browse/SPARK-47345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-47345:
Assignee: Yousof Hosny
> XML: Add XmlFunctionsSuite
> --
>
> Key: SPARK-47345
> URL: https://issues.apache.org/jira/browse/SPARK-47345
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Yousof Hosny
>Assignee: Yousof Hosny
>Priority: Minor
> Labels: pull-request-available
>
> Convert JsonFunctiosnSuite.scala to XML equivalent. Note that XML doesn’t
> implement all json functions like {{{}json_tuple{}}},
> {{{}get_json_object{}}}, etc.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47345) XML: Add XmlFunctionsSuite
[
https://issues.apache.org/jira/browse/SPARK-47345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-47345.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45466
[https://github.com/apache/spark/pull/45466]
> XML: Add XmlFunctionsSuite
> --
>
> Key: SPARK-47345
> URL: https://issues.apache.org/jira/browse/SPARK-47345
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Yousof Hosny
>Assignee: Yousof Hosny
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Convert JsonFunctiosnSuite.scala to XML equivalent. Note that XML doesn’t
> implement all json functions like {{{}json_tuple{}}},
> {{{}get_json_object{}}}, etc.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47395) Add collate and collation to non-sql APIs
[ https://issues.apache.org/jira/browse/SPARK-47395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47395. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45517 [https://github.com/apache/spark/pull/45517] > Add collate and collation to non-sql APIs > - > > Key: SPARK-47395 > URL: https://issues.apache.org/jira/browse/SPARK-47395 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stefan Kandic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47407) Support java.sql.Types.NULL
[ https://issues.apache.org/jira/browse/SPARK-47407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47407: Assignee: Kent Yao > Support java.sql.Types.NULL > --- > > Key: SPARK-47407 > URL: https://issues.apache.org/jira/browse/SPARK-47407 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47407) Support java.sql.Types.NULL
[ https://issues.apache.org/jira/browse/SPARK-47407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47407. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45531 [https://github.com/apache/spark/pull/45531] > Support java.sql.Types.NULL > --- > > Key: SPARK-47407 > URL: https://issues.apache.org/jira/browse/SPARK-47407 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47399) Disable generated columns on expressions with collations
[ https://issues.apache.org/jira/browse/SPARK-47399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47399. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45520 [https://github.com/apache/spark/pull/45520] > Disable generated columns on expressions with collations > > > Key: SPARK-47399 > URL: https://issues.apache.org/jira/browse/SPARK-47399 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stefan Kandic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Changing the collation of a column or even just changing the ICU version > could lead to a differences in the resulting expression so it would be best > if we simply disable it for now. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47399) Disable generated columns on expressions with collations
[ https://issues.apache.org/jira/browse/SPARK-47399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47399: Assignee: Stefan Kandic > Disable generated columns on expressions with collations > > > Key: SPARK-47399 > URL: https://issues.apache.org/jira/browse/SPARK-47399 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stefan Kandic >Priority: Major > Labels: pull-request-available > > Changing the collation of a column or even just changing the ICU version > could lead to a differences in the resulting expression so it would be best > if we simply disable it for now. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47387) Remove some unused error classes
[ https://issues.apache.org/jira/browse/SPARK-47387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47387. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45509 [https://github.com/apache/spark/pull/45509] > Remove some unused error classes > > > Key: SPARK-47387 > URL: https://issues.apache.org/jira/browse/SPARK-47387 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47387) Remove some unused error classes
[ https://issues.apache.org/jira/browse/SPARK-47387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47387: Assignee: BingKun Pan > Remove some unused error classes > > > Key: SPARK-47387 > URL: https://issues.apache.org/jira/browse/SPARK-47387 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47388) Pass messageParameters by name to require()
[ https://issues.apache.org/jira/browse/SPARK-47388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47388. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45511 [https://github.com/apache/spark/pull/45511] > Pass messageParameters by name to require() > --- > > Key: SPARK-47388 > URL: https://issues.apache.org/jira/browse/SPARK-47388 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Max Gekk >Assignee: Max Gekk >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Passing *messageParameters* by value independently from requirement might > introduce perf regression. Need to pass *messageParameters* by name to avoid > eager instantiation -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47388) Pass messageParameters by name to require()
Max Gekk created SPARK-47388: Summary: Pass messageParameters by name to require() Key: SPARK-47388 URL: https://issues.apache.org/jira/browse/SPARK-47388 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Max Gekk Assignee: Max Gekk Passing *messageParameters* by value independently from requirement might introduce perf regression. Need to pass *messageParameters* by name to avoid eager instantiation -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-46654) df.show() of pyspark displayed different results between Regular Spark and Spark Connect
[
https://issues.apache.org/jira/browse/SPARK-46654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17826031#comment-17826031
]
Max Gekk commented on SPARK-46654:
--
[~panbingkun][~LuciferYang] How about *to_json*? Does it have similar issue?
> df.show() of pyspark displayed different results between Regular Spark and
> Spark Connect
>
>
> Key: SPARK-46654
> URL: https://issues.apache.org/jira/browse/SPARK-46654
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: BingKun Pan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> The following doctest will throw an error in the tests of the pyspark-connect
> module
> {code:java}
> Example 2: Converting a complex StructType to a CSV string
> >>> from pyspark.sql import Row, functions as sf
> >>> data = [(1, Row(age=2, name='Alice', scores=[100, 200, 300]))]
> >>> df = spark.createDataFrame(data, ("key", "value"))
> >>> df.select(sf.to_csv(df.value)).show(truncate=False) # doctest: +SKIP
> +---+
> |to_csv(value) |
> +---+
> |2,Alice,"[100,200,300]"|
> +---+{code}
> {code:java}
> **
> 3953File "/__w/spark/spark/python/pyspark/sql/connect/functions/builtin.py",
> line 2232, in pyspark.sql.connect.functions.builtin.to_csv
> 3954Failed example:
> 3955df.select(sf.to_csv(df.value)).show(truncate=False)
> 3956Expected:
> 3957+---+
> 3958|to_csv(value) |
> 3959+---+
> 3960|2,Alice,"[100,200,300]"|
> 3961+---+
> 3962Got:
> 3963
> +--+
> 3964|to_csv(value)
> |
> 3965
> +--+
> 3966
> |2,Alice,org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@99c5e30f|
> 3967
> +--+
> 3968
> 3969**
> 3970 1 of 18 in pyspark.sql.connect.functions.builtin.to_csv
> 3971***Test Failed*** 1 failures. {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46654) df.show() of pyspark displayed different results between Regular Spark and Spark Connect
[
https://issues.apache.org/jira/browse/SPARK-46654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-46654.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 44665
[https://github.com/apache/spark/pull/44665]
> df.show() of pyspark displayed different results between Regular Spark and
> Spark Connect
>
>
> Key: SPARK-46654
> URL: https://issues.apache.org/jira/browse/SPARK-46654
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: BingKun Pan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> The following doctest will throw an error in the tests of the pyspark-connect
> module
> {code:java}
> Example 2: Converting a complex StructType to a CSV string
> >>> from pyspark.sql import Row, functions as sf
> >>> data = [(1, Row(age=2, name='Alice', scores=[100, 200, 300]))]
> >>> df = spark.createDataFrame(data, ("key", "value"))
> >>> df.select(sf.to_csv(df.value)).show(truncate=False) # doctest: +SKIP
> +---+
> |to_csv(value) |
> +---+
> |2,Alice,"[100,200,300]"|
> +---+{code}
> {code:java}
> **
> 3953File "/__w/spark/spark/python/pyspark/sql/connect/functions/builtin.py",
> line 2232, in pyspark.sql.connect.functions.builtin.to_csv
> 3954Failed example:
> 3955df.select(sf.to_csv(df.value)).show(truncate=False)
> 3956Expected:
> 3957+---+
> 3958|to_csv(value) |
> 3959+---+
> 3960|2,Alice,"[100,200,300]"|
> 3961+---+
> 3962Got:
> 3963
> +--+
> 3964|to_csv(value)
> |
> 3965
> +--+
> 3966
> |2,Alice,org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@99c5e30f|
> 3967
> +--+
> 3968
> 3969**
> 3970 1 of 18 in pyspark.sql.connect.functions.builtin.to_csv
> 3971***Test Failed*** 1 failures. {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46654) df.show() of pyspark displayed different results between Regular Spark and Spark Connect
[
https://issues.apache.org/jira/browse/SPARK-46654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-46654:
Assignee: BingKun Pan
> df.show() of pyspark displayed different results between Regular Spark and
> Spark Connect
>
>
> Key: SPARK-46654
> URL: https://issues.apache.org/jira/browse/SPARK-46654
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: BingKun Pan
>Priority: Major
> Labels: pull-request-available
>
> The following doctest will throw an error in the tests of the pyspark-connect
> module
> {code:java}
> Example 2: Converting a complex StructType to a CSV string
> >>> from pyspark.sql import Row, functions as sf
> >>> data = [(1, Row(age=2, name='Alice', scores=[100, 200, 300]))]
> >>> df = spark.createDataFrame(data, ("key", "value"))
> >>> df.select(sf.to_csv(df.value)).show(truncate=False) # doctest: +SKIP
> +---+
> |to_csv(value) |
> +---+
> |2,Alice,"[100,200,300]"|
> +---+{code}
> {code:java}
> **
> 3953File "/__w/spark/spark/python/pyspark/sql/connect/functions/builtin.py",
> line 2232, in pyspark.sql.connect.functions.builtin.to_csv
> 3954Failed example:
> 3955df.select(sf.to_csv(df.value)).show(truncate=False)
> 3956Expected:
> 3957+---+
> 3958|to_csv(value) |
> 3959+---+
> 3960|2,Alice,"[100,200,300]"|
> 3961+---+
> 3962Got:
> 3963
> +--+
> 3964|to_csv(value)
> |
> 3965
> +--+
> 3966
> |2,Alice,org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@99c5e30f|
> 3967
> +--+
> 3968
> 3969**
> 3970 1 of 18 in pyspark.sql.connect.functions.builtin.to_csv
> 3971***Test Failed*** 1 failures. {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-46810) Clarify error class terminology
[
https://issues.apache.org/jira/browse/SPARK-46810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17826016#comment-17826016
]
Max Gekk commented on SPARK-46810:
--
> Yes I prefer option 1.
After thinking a little more, I prefer the option 1 too. [~nchammas] If there
are no objections from others, let's follow this approach, and rename all
needed places.
I prefer the option 1 over others because it conforms the SQL standard, and
won't confuse users. It would hard to defend other non-standard approaches in
the future.
The problem is we have already released the `errorClass` field as a part of
Spark exceptions in Spark 3.5.x, but I think we could rename it in 4.0.0. WDYT
[~dongjoon] [~cloud_fan] [~gurwls223]?
> Clarify error class terminology
> ---
>
> Key: SPARK-46810
> URL: https://issues.apache.org/jira/browse/SPARK-46810
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, SQL
>Affects Versions: 4.0.0
>Reporter: Nicholas Chammas
>Priority: Minor
> Labels: pull-request-available
>
> We use inconsistent terminology when talking about error classes. I'd like to
> get some clarity on that before contributing any potential improvements to
> this part of the documentation.
> Consider
> [INCOMPLETE_TYPE_DEFINITION|https://spark.apache.org/docs/3.5.0/sql-error-conditions-incomplete-type-definition-error-class.html].
> It has several key pieces of hierarchical information that have inconsistent
> names throughout our documentation and codebase:
> * 42
> ** K01
> *** INCOMPLETE_TYPE_DEFINITION
> ARRAY
> MAP
> STRUCT
> What are the names of these different levels of information?
> Some examples of inconsistent terminology:
> * [Over
> here|https://spark.apache.org/docs/latest/sql-error-conditions-sqlstates.html#class-42-syntax-error-or-access-rule-violation]
> we call 42 the "class". Yet on the main page for INCOMPLETE_TYPE_DEFINITION
> we call that an "error class". So what exactly is a class, the 42 or the
> INCOMPLETE_TYPE_DEFINITION?
> * [Over
> here|https://github.com/apache/spark/blob/26d3eca0a8d3303d0bb9450feb6575ed145bbd7e/common/utils/src/main/resources/error/README.md#L122]
> we call K01 the "subclass". But [over
> here|https://github.com/apache/spark/blob/26d3eca0a8d3303d0bb9450feb6575ed145bbd7e/common/utils/src/main/resources/error/error-classes.json#L1452-L1467]
> we call the ARRAY, MAP, and STRUCT the subclasses. And on the main page for
> INCOMPLETE_TYPE_DEFINITION we call those same things "derived error classes".
> So what exactly is a subclass?
> * [On this
> page|https://spark.apache.org/docs/3.5.0/sql-error-conditions.html#incomplete_type_definition]
> we call INCOMPLETE_TYPE_DEFINITION an "error condition", though in other
> places we refer to it as an "error class".
> I don't think we should leave this status quo as-is. I see a couple of ways
> to fix this.
> h1. Option 1: INCOMPLETE_TYPE_DEFINITION becomes an "Error Condition"
> One solution is to use the following terms:
> * Error class: 42
> * Error sub-class: K01
> * Error state: 42K01
> * Error condition: INCOMPLETE_TYPE_DEFINITION
> * Error sub-condition: ARRAY, MAP, STRUCT
> Pros:
> * This terminology seems (to me at least) the most natural and intuitive.
> * It aligns most closely to the SQL standard.
> Cons:
> * We use {{errorClass}} [all over our
> codebase|https://github.com/apache/spark/blob/15c9ec7ca3b66ec413b7964a374cb9508a80/common/utils/src/main/scala/org/apache/spark/SparkException.scala#L30]
> – literally in thousands of places – to refer to strings like
> INCOMPLETE_TYPE_DEFINITION.
> ** It's probably not practical to update all these usages to say
> {{errorCondition}} instead, so if we go with this approach there will be a
> divide between the terminology we use in user-facing documentation vs. what
> the code base uses.
> ** We can perhaps rename the existing {{error-classes.json}} to
> {{error-conditions.json}} but clarify the reason for this divide between code
> and user docs in the documentation for {{ErrorClassesJsonReader}} .
> h1. Option 2: 42 becomes an "Error Category"
> Another approach is to use the following terminology:
> * Error category: 42
> * Error sub-category: K01
> * Error state: 42K01
> * Error class: INCOMPLETE_TYPE_DEFINITION
> * Error sub-classes: ARRAY, MAP, STRUCT
> Pros:
> * We continue to use "error class" as we do today in our code base.
> * The change from calling "42" a "class" to a "category" is low impact and
> may not show up in user-facing documentation at all. (See my side note below.)
> Cons:
> * These terms do not align with the SQL standard.
> * We will have to retire the term "error condition", which we have [already
>
[jira] [Resolved] (SPARK-47333) Use checkInputDataTypes to check the parameter types of the function to_xml & remove _LEGACY_ERROR_TEMP_3234
[ https://issues.apache.org/jira/browse/SPARK-47333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47333. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45167 [https://github.com/apache/spark/pull/45167] > Use checkInputDataTypes to check the parameter types of the function to_xml & > remove _LEGACY_ERROR_TEMP_3234 > > > Key: SPARK-47333 > URL: https://issues.apache.org/jira/browse/SPARK-47333 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47333) Use checkInputDataTypes to check the parameter types of the function to_xml & remove _LEGACY_ERROR_TEMP_3234
[ https://issues.apache.org/jira/browse/SPARK-47333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47333: Assignee: BingKun Pan > Use checkInputDataTypes to check the parameter types of the function to_xml & > remove _LEGACY_ERROR_TEMP_3234 > > > Key: SPARK-47333 > URL: https://issues.apache.org/jira/browse/SPARK-47333 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47340) Change "collate" in StringType typename to lowercase
[ https://issues.apache.org/jira/browse/SPARK-47340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47340. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45475 [https://github.com/apache/spark/pull/45475] > Change "collate" in StringType typename to lowercase > > > Key: SPARK-47340 > URL: https://issues.apache.org/jira/browse/SPARK-47340 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Nikola Mandic >Assignee: Nikola Mandic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Change > [https://github.com/apache/spark/blob/37bdf5179e01f6c8cb8c83445a155646794aabef/sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala#L50] > "COLLATE" to lowercase in order to follow convention. Example from > DecimalType > https://github.com/apache/spark/blob/2a51242b1bdb4a1d469fc81dc0d21c4cde305753/sql/api/src/main/scala/org/apache/spark/sql/types/DecimalType.scala#L59-L63. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47343) Execute immediate throws NPE when `sqlString` variable value is null
[ https://issues.apache.org/jira/browse/SPARK-47343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47343. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45462 [https://github.com/apache/spark/pull/45462] > Execute immediate throws NPE when `sqlString` variable value is null > > > Key: SPARK-47343 > URL: https://issues.apache.org/jira/browse/SPARK-47343 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 4.0.0 >Reporter: Milan Stefanovic >Assignee: Stefan Bukorovic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Execute immediate throws NPE when `sqlString` variable value is null > > `EXECUTE IMMEDIATE sqlString` throws NPE when `sqlString` is null. > Throw proper error when variable value is null -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47295) startswith, endswith (non-binary collations)
[
https://issues.apache.org/jira/browse/SPARK-47295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-47295:
Assignee: Stevo Mitric
> startswith, endswith (non-binary collations)
>
>
> Key: SPARK-47295
> URL: https://issues.apache.org/jira/browse/SPARK-47295
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Uroš Bojanić
>Assignee: Stevo Mitric
>Priority: Major
> Labels: pull-request-available
>
> Implement *startsWith* and *endsWith* built-in string Spark functions using
> {_}StringSearch{_}, an efficient ICU service for string matching. Refer to
> the latest unit tests in CollationSuite to understand how these functions are
> used in SparkSQL, and feel free to use your chosen Spark SQL Editor to play
> around with the existing functions to learn more about how they work.
>
> Currently, these 2 functions support all collation types:
> # binary collations (UCS_BASIC, UNICODE) *special cases - these collation
> types work using the existing string comparison functions - i.e. contains(),
> startsWith(), endsWith()
> # special lowercase non-binary collations (UCS_BASIC) *special case - these
> collation types work by using lower() to convert both strings to lowercase,
> and then use above functions
> # other non-binary collations (UNICODE_CI; special collations for various
> languages with case and accent sensitivity) - these collation types usually
> require special handling, which can sometimes be complex
>
> To understand what changes were introduced in order to enable collation
> support for these functions, take a look at the Spark PRs and Jira tickets
> below:
> * [https://github.com/apache/spark/pull/45216] this PR enables:
> ** partial collation support for *contains* (skipping the 3rd type of
> collations shown above)
> ** complete collation support for {*}startsWith{*}, *endsWith* (using a
> special _matchAt_ implementation directly in {_}UTF8String{_})
> * [https://github.com/apache/spark/pull/45382] this PR enables:
> ** complete collation support for *contains* (using {_}StringSearch{_}) _->
> now we should also use this approach for startsWith & endsWith_
>
> Focusing on the 3rd type of collations as shown above, the goal for this Jira
> ticket is to re-implement the *startsWith* and *endsWith* functions so that
> they use _StringSearch_ instead (following the general logic in the second
> PR). As for the current test cases in CollationSuite, they should already
> mostly cover the expected behaviour of *startsWith* and *endsWith* for the
> 3rd type of collations.
>
> Read more about _StringSearch_ using the [ICU user
> guide|https://unicode-org.github.io/icu/userguide/collation/string-search.html]
> and [ICU
> docs|https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/StringSearch.html].
> Also, refer to the Unicode Technical Standard for string
> [searching|https://www.unicode.org/reports/tr10/#Searching] and
> [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47295) startswith, endswith (non-binary collations)
[
https://issues.apache.org/jira/browse/SPARK-47295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-47295.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45421
[https://github.com/apache/spark/pull/45421]
> startswith, endswith (non-binary collations)
>
>
> Key: SPARK-47295
> URL: https://issues.apache.org/jira/browse/SPARK-47295
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Uroš Bojanić
>Assignee: Stevo Mitric
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Implement *startsWith* and *endsWith* built-in string Spark functions using
> {_}StringSearch{_}, an efficient ICU service for string matching. Refer to
> the latest unit tests in CollationSuite to understand how these functions are
> used in SparkSQL, and feel free to use your chosen Spark SQL Editor to play
> around with the existing functions to learn more about how they work.
>
> Currently, these 2 functions support all collation types:
> # binary collations (UCS_BASIC, UNICODE) *special cases - these collation
> types work using the existing string comparison functions - i.e. contains(),
> startsWith(), endsWith()
> # special lowercase non-binary collations (UCS_BASIC) *special case - these
> collation types work by using lower() to convert both strings to lowercase,
> and then use above functions
> # other non-binary collations (UNICODE_CI; special collations for various
> languages with case and accent sensitivity) - these collation types usually
> require special handling, which can sometimes be complex
>
> To understand what changes were introduced in order to enable collation
> support for these functions, take a look at the Spark PRs and Jira tickets
> below:
> * [https://github.com/apache/spark/pull/45216] this PR enables:
> ** partial collation support for *contains* (skipping the 3rd type of
> collations shown above)
> ** complete collation support for {*}startsWith{*}, *endsWith* (using a
> special _matchAt_ implementation directly in {_}UTF8String{_})
> * [https://github.com/apache/spark/pull/45382] this PR enables:
> ** complete collation support for *contains* (using {_}StringSearch{_}) _->
> now we should also use this approach for startsWith & endsWith_
>
> Focusing on the 3rd type of collations as shown above, the goal for this Jira
> ticket is to re-implement the *startsWith* and *endsWith* functions so that
> they use _StringSearch_ instead (following the general logic in the second
> PR). As for the current test cases in CollationSuite, they should already
> mostly cover the expected behaviour of *startsWith* and *endsWith* for the
> 3rd type of collations.
>
> Read more about _StringSearch_ using the [ICU user
> guide|https://unicode-org.github.io/icu/userguide/collation/string-search.html]
> and [ICU
> docs|https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/StringSearch.html].
> Also, refer to the Unicode Technical Standard for string
> [searching|https://www.unicode.org/reports/tr10/#Searching] and
> [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47169) Disable bucketing on collated collumns
[ https://issues.apache.org/jira/browse/SPARK-47169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47169. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45260 [https://github.com/apache/spark/pull/45260] > Disable bucketing on collated collumns > -- > > Key: SPARK-47169 > URL: https://issues.apache.org/jira/browse/SPARK-47169 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Mihailo Milosevic >Assignee: Mihailo Milosevic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > *What changes were proposed in this pull request?* > Disable bucketing on columns that are non default collated. > *Why are the changes needed?* > With current implementation bucketIds are generated from a string value where > each unique string guarantees unique id, but when collation is turned on, > this is not the case. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47169) Disable bucketing on collated collumns
[ https://issues.apache.org/jira/browse/SPARK-47169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47169: Assignee: Mihailo Milosevic > Disable bucketing on collated collumns > -- > > Key: SPARK-47169 > URL: https://issues.apache.org/jira/browse/SPARK-47169 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Mihailo Milosevic >Assignee: Mihailo Milosevic >Priority: Major > Labels: pull-request-available > > *What changes were proposed in this pull request?* > Disable bucketing on columns that are non default collated. > *Why are the changes needed?* > With current implementation bucketIds are generated from a string value where > each unique string guarantees unique id, but when collation is turned on, > this is not the case. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47254) Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
[
https://issues.apache.org/jira/browse/SPARK-47254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-47254:
Assignee: Stefan Bukorovic
> Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
> -
>
> Key: SPARK-47254
> URL: https://issues.apache.org/jira/browse/SPARK-47254
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Assignee: Stefan Bukorovic
>Priority: Minor
> Labels: pull-request-available, starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_325[1-9]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47254) Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
[
https://issues.apache.org/jira/browse/SPARK-47254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-47254.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45407
[https://github.com/apache/spark/pull/45407]
> Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
> -
>
> Key: SPARK-47254
> URL: https://issues.apache.org/jira/browse/SPARK-47254
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Assignee: Stefan Bukorovic
>Priority: Minor
> Labels: pull-request-available, starter
> Fix For: 4.0.0
>
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_325[1-9]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47255) Assign names to the error classes _LEGACY_ERROR_TEMP_324[7-9]
[
https://issues.apache.org/jira/browse/SPARK-47255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-47255.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45423
[https://github.com/apache/spark/pull/45423]
> Assign names to the error classes _LEGACY_ERROR_TEMP_324[7-9]
> -
>
> Key: SPARK-47255
> URL: https://issues.apache.org/jira/browse/SPARK-47255
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Assignee: Milan Dankovic
>Priority: Minor
> Labels: pull-request-available, starter
> Fix For: 4.0.0
>
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_324[7-9]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47255) Assign names to the error classes _LEGACY_ERROR_TEMP_324[7-9]
[
https://issues.apache.org/jira/browse/SPARK-47255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-47255:
Assignee: Milan Dankovic
> Assign names to the error classes _LEGACY_ERROR_TEMP_324[7-9]
> -
>
> Key: SPARK-47255
> URL: https://issues.apache.org/jira/browse/SPARK-47255
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Assignee: Milan Dankovic
>Priority: Minor
> Labels: pull-request-available, starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_324[7-9]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47313) scala.MatchError should be treated as internal error
[ https://issues.apache.org/jira/browse/SPARK-47313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47313: Assignee: Stevo Mitric (was: Max Gekk) > scala.MatchError should be treated as internal error > > > Key: SPARK-47313 > URL: https://issues.apache.org/jira/browse/SPARK-47313 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wenchen Fan >Assignee: Stevo Mitric >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > We should update `QueryExecution.toInternalError` to handle scala.MatchError -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47313) scala.MatchError should be treated as internal error
[ https://issues.apache.org/jira/browse/SPARK-47313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47313. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45438 [https://github.com/apache/spark/pull/45438] > scala.MatchError should be treated as internal error > > > Key: SPARK-47313 > URL: https://issues.apache.org/jira/browse/SPARK-47313 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wenchen Fan >Assignee: Max Gekk >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > We should update `QueryExecution.toInternalError` to handle scala.MatchError -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47313) scala.MatchError should be treated as internal error
[ https://issues.apache.org/jira/browse/SPARK-47313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47313: Assignee: Max Gekk > scala.MatchError should be treated as internal error > > > Key: SPARK-47313 > URL: https://issues.apache.org/jira/browse/SPARK-47313 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wenchen Fan >Assignee: Max Gekk >Priority: Major > Labels: pull-request-available > > We should update `QueryExecution.toInternalError` to handle scala.MatchError -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47328) Change utf8 collation names to UTF8_BINARY
[ https://issues.apache.org/jira/browse/SPARK-47328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47328. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45442 [https://github.com/apache/spark/pull/45442] > Change utf8 collation names to UTF8_BINARY > -- > > Key: SPARK-47328 > URL: https://issues.apache.org/jira/browse/SPARK-47328 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stefan Kandic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47328) Change utf8 collation names to UTF8_BINARY
[ https://issues.apache.org/jira/browse/SPARK-47328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47328: Assignee: Stefan Kandic > Change utf8 collation names to UTF8_BINARY > -- > > Key: SPARK-47328 > URL: https://issues.apache.org/jira/browse/SPARK-47328 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stefan Kandic >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47326) Moving tests to related Suites
[ https://issues.apache.org/jira/browse/SPARK-47326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47326. -- Resolution: Won't Fix > Moving tests to related Suites > -- > > Key: SPARK-47326 > URL: https://issues.apache.org/jira/browse/SPARK-47326 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Mihailo Milosevic >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47302) Collation name should be identifier
[ https://issues.apache.org/jira/browse/SPARK-47302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47302: Assignee: Aleksandar Tomic > Collation name should be identifier > --- > > Key: SPARK-47302 > URL: https://issues.apache.org/jira/browse/SPARK-47302 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Aleksandar Tomic >Assignee: Aleksandar Tomic >Priority: Major > Labels: pull-request-available > > Currently collation names are parsed as string literals. > In spec they should be multi part identifiers (see spec linked with root > collation jira). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47302) Collation name should be identifier
[ https://issues.apache.org/jira/browse/SPARK-47302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47302. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45405 [https://github.com/apache/spark/pull/45405] > Collation name should be identifier > --- > > Key: SPARK-47302 > URL: https://issues.apache.org/jira/browse/SPARK-47302 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Aleksandar Tomic >Assignee: Aleksandar Tomic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Currently collation names are parsed as string literals. > In spec they should be multi part identifiers (see spec linked with root > collation jira). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47304) Split DataFrameSuite
[ https://issues.apache.org/jira/browse/SPARK-47304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47304. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45392 [https://github.com/apache/spark/pull/45392] > Split DataFrameSuite > > > Key: SPARK-47304 > URL: https://issues.apache.org/jira/browse/SPARK-47304 > Project: Spark > Issue Type: Test > Components: SQL >Affects Versions: 4.0.0 >Reporter: Max Gekk >Assignee: Max Gekk >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Split DataFrameSuite to smaller test suite, and/or distribute its tests among > other test suites. This should improve maintainability and speed up its > execution. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47304) Split DataFrameSuite
Max Gekk created SPARK-47304: Summary: Split DataFrameSuite Key: SPARK-47304 URL: https://issues.apache.org/jira/browse/SPARK-47304 Project: Spark Issue Type: Test Components: SQL Affects Versions: 4.0.0 Reporter: Max Gekk Assignee: Max Gekk Split DataFrameSuite to smaller test suite, and/or distribute its tests among other test suites. This should improve maintainability and speed up its execution. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47102) Add COLLATION_ENABLED config flag
[ https://issues.apache.org/jira/browse/SPARK-47102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47102: Assignee: Mihailo Milosevic > Add COLLATION_ENABLED config flag > - > > Key: SPARK-47102 > URL: https://issues.apache.org/jira/browse/SPARK-47102 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Mihailo Milosevic >Assignee: Mihailo Milosevic >Priority: Major > Labels: pull-request-available > > *What changes were proposed in this pull request?* > This PR adds COLLATION_ENABLED config to `SQLConf` and introduces new error > class `COLLATION_SUPPORT_NOT_ENABLED` to appropriately report error on usage > of feature under development. > *Why are the changes needed?* > We want to make collations configurable on this flag. These changes disable > usage of `collate` and `collation` functions, along with any `COLLATE` syntax > when the flag is set to false. By default, the flag is set to false. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47102) Add COLLATION_ENABLED config flag
[ https://issues.apache.org/jira/browse/SPARK-47102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47102. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45285 [https://github.com/apache/spark/pull/45285] > Add COLLATION_ENABLED config flag > - > > Key: SPARK-47102 > URL: https://issues.apache.org/jira/browse/SPARK-47102 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Mihailo Milosevic >Assignee: Mihailo Milosevic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > *What changes were proposed in this pull request?* > This PR adds COLLATION_ENABLED config to `SQLConf` and introduces new error > class `COLLATION_SUPPORT_NOT_ENABLED` to appropriately report error on usage > of feature under development. > *Why are the changes needed?* > We want to make collations configurable on this flag. These changes disable > usage of `collate` and `collation` functions, along with any `COLLATE` syntax > when the flag is set to false. By default, the flag is set to false. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46961) Adding processorHandle as a Context Variable
[ https://issues.apache.org/jira/browse/SPARK-46961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-46961: Assignee: Eric Marnadi > Adding processorHandle as a Context Variable > > > Key: SPARK-46961 > URL: https://issues.apache.org/jira/browse/SPARK-46961 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Eric Marnadi >Assignee: Eric Marnadi >Priority: Major > Labels: pull-request-available > > Instead of passing the StatefulProcessorHandle to the user in `init`, instead > embed it as a context variable, ProcessorContext, that the user can fetch -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46961) Adding processorHandle as a Context Variable
[ https://issues.apache.org/jira/browse/SPARK-46961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-46961. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45359 [https://github.com/apache/spark/pull/45359] > Adding processorHandle as a Context Variable > > > Key: SPARK-46961 > URL: https://issues.apache.org/jira/browse/SPARK-46961 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Eric Marnadi >Assignee: Eric Marnadi >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Instead of passing the StatefulProcessorHandle to the user in `init`, instead > embed it as a context variable, ProcessorContext, that the user can fetch -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-43258) Assign a name to the error class _LEGACY_ERROR_TEMP_202[3,4,5]
[
https://issues.apache.org/jira/browse/SPARK-43258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-43258.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45288
[https://github.com/apache/spark/pull/45288]
> Assign a name to the error class _LEGACY_ERROR_TEMP_202[3,4,5]
> --
>
> Key: SPARK-43258
> URL: https://issues.apache.org/jira/browse/SPARK-43258
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Max Gekk
>Assignee: Deng Ziming
>Priority: Minor
> Labels: pull-request-available, starter
> Fix For: 4.0.0
>
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2023* defined in
> {*}core/src/main/resources/error/error-classes.json{*}. The name should be
> short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-43258) Assign a name to the error class _LEGACY_ERROR_TEMP_202[3,4,5]
[
https://issues.apache.org/jira/browse/SPARK-43258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-43258:
Assignee: Deng Ziming
> Assign a name to the error class _LEGACY_ERROR_TEMP_202[3,4,5]
> --
>
> Key: SPARK-43258
> URL: https://issues.apache.org/jira/browse/SPARK-43258
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Max Gekk
>Assignee: Deng Ziming
>Priority: Minor
> Labels: pull-request-available, starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2023* defined in
> {*}core/src/main/resources/error/error-classes.json{*}. The name should be
> short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47263) Assign classes to DEFAULT value errors
Max Gekk created SPARK-47263:
Summary: Assign classes to DEFAULT value errors
Key: SPARK-47263
URL: https://issues.apache.org/jira/browse/SPARK-47263
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_22[38-40]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47263) Assign classes to DEFAULT value errors
[
https://issues.apache.org/jira/browse/SPARK-47263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47263:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_13[44-46]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_22[38-40]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign classes to DEFAULT value errors
> --
>
> Key: SPARK-47263
> URL: https://issues.apache.org/jira/browse/SPARK-47263
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_13[44-46]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47262) Assign classes to Parquet converter errors
[
https://issues.apache.org/jira/browse/SPARK-47262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47262:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_22[38-40]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_11[72-74]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign classes to Parquet converter errors
> --
>
> Key: SPARK-47262
> URL: https://issues.apache.org/jira/browse/SPARK-47262
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_22[38-40]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47261) Assign classes to Parquet type errors
[
https://issues.apache.org/jira/browse/SPARK-47261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47261:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_11[72-74]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_32[49-51]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign classes to Parquet type errors
> -
>
> Key: SPARK-47261
> URL: https://issues.apache.org/jira/browse/SPARK-47261
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_11[72-74]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47262) Assign classes to Parquet converter errors
Max Gekk created SPARK-47262:
Summary: Assign classes to Parquet converter errors
Key: SPARK-47262
URL: https://issues.apache.org/jira/browse/SPARK-47262
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_11[72-74]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47261) Assign classes to Parquet type errors
Max Gekk created SPARK-47261:
Summary: Assign classes to Parquet type errors
Key: SPARK-47261
URL: https://issues.apache.org/jira/browse/SPARK-47261
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_32[49-51]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47260) Assign classes to Row to JSON errors
[
https://issues.apache.org/jira/browse/SPARK-47260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47260:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_32[49-51]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_32[08-14]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign classes to Row to JSON errors
> -
>
> Key: SPARK-47260
> URL: https://issues.apache.org/jira/browse/SPARK-47260
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_32[49-51]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47260) Assign classes to Row to JSON errors
Max Gekk created SPARK-47260:
Summary: Assign classes to Row to JSON errors
Key: SPARK-47260
URL: https://issues.apache.org/jira/browse/SPARK-47260
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_32[08-14]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47259) Assign classes to interval errors
[
https://issues.apache.org/jira/browse/SPARK-47259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47259:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_32[08-14]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_127[0-5]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign classes to interval errors
> -
>
> Key: SPARK-47259
> URL: https://issues.apache.org/jira/browse/SPARK-47259
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_32[08-14]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47259) Assign classes to interval errors
Max Gekk created SPARK-47259:
Summary: Assign classes to interval errors
Key: SPARK-47259
URL: https://issues.apache.org/jira/browse/SPARK-47259
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_127[0-5]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47258) Assign error classes to SHOW CREATE TABLE errors
Max Gekk created SPARK-47258:
Summary: Assign error classes to SHOW CREATE TABLE errors
Key: SPARK-47258
URL: https://issues.apache.org/jira/browse/SPARK-47258
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_105[3-4]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47258) Assign error classes to SHOW CREATE TABLE errors
[
https://issues.apache.org/jira/browse/SPARK-47258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47258:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_127[0-5]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_105[3-4]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign error classes to SHOW CREATE TABLE errors
>
>
> Key: SPARK-47258
> URL: https://issues.apache.org/jira/browse/SPARK-47258
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_127[0-5]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47257) Assign error classes to ALTER COLUMN errors
Max Gekk created SPARK-47257:
Summary: Assign error classes to ALTER COLUMN errors
Key: SPARK-47257
URL: https://issues.apache.org/jira/browse/SPARK-47257
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_102[4-7]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47257) Assign error classes to ALTER COLUMN errors
[
https://issues.apache.org/jira/browse/SPARK-47257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47257:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_105[3-4]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_102[4-7]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign error classes to ALTER COLUMN errors
> ---
>
> Key: SPARK-47257
> URL: https://issues.apache.org/jira/browse/SPARK-47257
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_105[3-4]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47256) Assign error classes to FILTER expression errors
Max Gekk created SPARK-47256:
Summary: Assign error classes to FILTER expression errors
Key: SPARK-47256
URL: https://issues.apache.org/jira/browse/SPARK-47256
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_324[7-9]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47256) Assign error classes to FILTER expression errors
[
https://issues.apache.org/jira/browse/SPARK-47256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47256:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_102[4-7]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_324[7-9]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign error classes to FILTER expression errors
>
>
> Key: SPARK-47256
> URL: https://issues.apache.org/jira/browse/SPARK-47256
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_102[4-7]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47255) Assign names to the error classes _LEGACY_ERROR_TEMP_324[7-9]
[
https://issues.apache.org/jira/browse/SPARK-47255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47255:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_324[7-9]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_325[1-9]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign names to the error classes _LEGACY_ERROR_TEMP_324[7-9]
> -
>
> Key: SPARK-47255
> URL: https://issues.apache.org/jira/browse/SPARK-47255
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_324[7-9]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47255) Assign names to the error classes _LEGACY_ERROR_TEMP_324[7-9]
Max Gekk created SPARK-47255:
Summary: Assign names to the error classes
_LEGACY_ERROR_TEMP_324[7-9]
Key: SPARK-47255
URL: https://issues.apache.org/jira/browse/SPARK-47255
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_325[1-9]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47254) Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
Max Gekk created SPARK-47254:
Summary: Assign names to the error classes
_LEGACY_ERROR_TEMP_325[1-9]
Key: SPARK-47254
URL: https://issues.apache.org/jira/browse/SPARK-47254
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.5.0
Reporter: Max Gekk
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2000* defined in
{*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47254) Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
[
https://issues.apache.org/jira/browse/SPARK-47254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47254:
-
Affects Version/s: 4.0.0
(was: 3.5.0)
> Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
> -
>
> Key: SPARK-47254
> URL: https://issues.apache.org/jira/browse/SPARK-47254
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2000* defined in
> {*}core/src/main/resources/error/error-classes.json{*}. The name should be
> short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47254) Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
[
https://issues.apache.org/jira/browse/SPARK-47254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-47254:
-
Description:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_325[1-9]* defined
in {*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
was:
Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2000* defined in
{*}core/src/main/resources/error/error-classes.json{*}. The name should be
short but complete (look at the example in error-classes.json).
Add a test which triggers the error from user code if such test still doesn't
exist. Check exception fields by using {*}checkError(){*}. The last function
checks valuable error fields only, and avoids dependencies from error text
message. In this way, tech editors can modify error format in
error-classes.json, and don't worry of Spark's internal tests. Migrate other
tests that might trigger the error onto checkError().
If you cannot reproduce the error from user space (using SQL query), replace
the error by an internal error, see {*}SparkException.internalError(){*}.
Improve the error message format in error-classes.json if the current is not
clear. Propose a solution to users how to avoid and fix such kind of errors.
Please, look at the PR below as examples:
* [https://github.com/apache/spark/pull/38685]
* [https://github.com/apache/spark/pull/38656]
* [https://github.com/apache/spark/pull/38490]
> Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
> -
>
> Key: SPARK-47254
> URL: https://issues.apache.org/jira/browse/SPARK-47254
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_325[1-9]*
> defined in {*}core/src/main/resources/error/error-classes.json{*}. The name
> should be short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47158) Assign proper name and sqlState to _LEGACY_ERROR_TEMP_2134 & 2231
[ https://issues.apache.org/jira/browse/SPARK-47158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47158: Assignee: Haejoon Lee > Assign proper name and sqlState to _LEGACY_ERROR_TEMP_2134 & 2231 > - > > Key: SPARK-47158 > URL: https://issues.apache.org/jira/browse/SPARK-47158 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Haejoon Lee >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > > Assign proper name and sqlState to _LEGACY_ERROR_TEMP_2134 & 2231 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47158) Assign proper name and sqlState to _LEGACY_ERROR_TEMP_2134 & 2231
[ https://issues.apache.org/jira/browse/SPARK-47158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47158. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45244 [https://github.com/apache/spark/pull/45244] > Assign proper name and sqlState to _LEGACY_ERROR_TEMP_2134 & 2231 > - > > Key: SPARK-47158 > URL: https://issues.apache.org/jira/browse/SPARK-47158 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Haejoon Lee >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Assign proper name and sqlState to _LEGACY_ERROR_TEMP_2134 & 2231 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47237) Upgrade xmlschema-core to 2.3.1
[ https://issues.apache.org/jira/browse/SPARK-47237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47237. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45347 [https://github.com/apache/spark/pull/45347] > Upgrade xmlschema-core to 2.3.1 > --- > > Key: SPARK-47237 > URL: https://issues.apache.org/jira/browse/SPARK-47237 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47237) Upgrade xmlschema-core to 2.3.1
[ https://issues.apache.org/jira/browse/SPARK-47237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47237: Assignee: BingKun Pan > Upgrade xmlschema-core to 2.3.1 > --- > > Key: SPARK-47237 > URL: https://issues.apache.org/jira/browse/SPARK-47237 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47216) Refine layout of SQL performance tuning page
[ https://issues.apache.org/jira/browse/SPARK-47216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47216. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45322 [https://github.com/apache/spark/pull/45322] > Refine layout of SQL performance tuning page > > > Key: SPARK-47216 > URL: https://issues.apache.org/jira/browse/SPARK-47216 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Nicholas Chammas >Assignee: Nicholas Chammas >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47216) Refine layout of SQL performance tuning page
[ https://issues.apache.org/jira/browse/SPARK-47216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47216: Assignee: Nicholas Chammas > Refine layout of SQL performance tuning page > > > Key: SPARK-47216 > URL: https://issues.apache.org/jira/browse/SPARK-47216 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Nicholas Chammas >Assignee: Nicholas Chammas >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47243) Correct the package name of `StateMetadataSource.scala`
[ https://issues.apache.org/jira/browse/SPARK-47243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47243: Assignee: Yang Jie > Correct the package name of `StateMetadataSource.scala` > --- > > Key: SPARK-47243 > URL: https://issues.apache.org/jira/browse/SPARK-47243 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47243) Correct the package name of `StateMetadataSource.scala`
[ https://issues.apache.org/jira/browse/SPARK-47243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47243. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45352 [https://github.com/apache/spark/pull/45352] > Correct the package name of `StateMetadataSource.scala` > --- > > Key: SPARK-47243 > URL: https://issues.apache.org/jira/browse/SPARK-47243 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-43255) Assign a name to the error class _LEGACY_ERROR_TEMP_2020
[
https://issues.apache.org/jira/browse/SPARK-43255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-43255:
Assignee: Jin Helin
> Assign a name to the error class _LEGACY_ERROR_TEMP_2020
>
>
> Key: SPARK-43255
> URL: https://issues.apache.org/jira/browse/SPARK-43255
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Max Gekk
>Assignee: Jin Helin
>Priority: Minor
> Labels: pull-request-available, starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2020* defined in
> {*}core/src/main/resources/error/error-classes.json{*}. The name should be
> short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-43255) Assign a name to the error class _LEGACY_ERROR_TEMP_2020
[
https://issues.apache.org/jira/browse/SPARK-43255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-43255.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45302
[https://github.com/apache/spark/pull/45302]
> Assign a name to the error class _LEGACY_ERROR_TEMP_2020
>
>
> Key: SPARK-43255
> URL: https://issues.apache.org/jira/browse/SPARK-43255
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Max Gekk
>Assignee: Jin Helin
>Priority: Minor
> Labels: pull-request-available, starter
> Fix For: 4.0.0
>
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2020* defined in
> {*}core/src/main/resources/error/error-classes.json{*}. The name should be
> short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47015) Disable partitioning on collated columns
[ https://issues.apache.org/jira/browse/SPARK-47015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47015. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45104 [https://github.com/apache/spark/pull/45104] > Disable partitioning on collated columns > > > Key: SPARK-47015 > URL: https://issues.apache.org/jira/browse/SPARK-47015 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Stefan Kandic >Assignee: Stefan Kandic >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47221) Uses AbstractParser instead of CsvParser for CSV parser signature
[ https://issues.apache.org/jira/browse/SPARK-47221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47221. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45328 [https://github.com/apache/spark/pull/45328] > Uses AbstractParser instead of CsvParser for CSV parser signature > - > > Key: SPARK-47221 > URL: https://issues.apache.org/jira/browse/SPARK-47221 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > > https://github.com/uniVocity/univocity-parsers becomes inactive for the last > 3 years, and we can't land some bug fixes anymore. Maybe we should leverage > their interface, and have our CSV parser. This is a base work. > We should use the higher class if it fits for better maintenance in any event. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46077) Error in postgresql when pushing down filter by timestamp_ntz field
[
https://issues.apache.org/jira/browse/SPARK-46077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-46077.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45261
[https://github.com/apache/spark/pull/45261]
> Error in postgresql when pushing down filter by timestamp_ntz field
> ---
>
> Key: SPARK-46077
> URL: https://issues.apache.org/jira/browse/SPARK-46077
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Marina Krasilnikova
>Assignee: Pablo Langa Blanco
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> code to reproduce:
> SparkSession sparkSession = SparkSession
> .builder()
> .appName("test-app")
> .master("local[*]")
> .config("spark.sql.timestampType", "TIMESTAMP_NTZ")
> .getOrCreate();
> String url = "...";
> String catalogPropPrefix = "spark.sql.catalog.myc";
> sparkSession.conf().set(catalogPropPrefix, JDBCTableCatalog.class.getName());
> sparkSession.conf().set(catalogPropPrefix + ".url", url);
> Map options = new HashMap<>();
> options.put("driver", "org.postgresql.Driver");
> // options.put("pushDownPredicate", "false"); it works fine if this line is
> uncommented
> Dataset dataset = sparkSession.read()
> .options(options)
> .table("myc.demo.`My table`");
> dataset.createOrReplaceTempView("view1");
> String sql = "select * from view1 where `my date` = '2021-04-01 00:00:00'";
> Dataset result = sparkSession.sql(sql);
> result.show();
> result.printSchema();
> Field `my date` is of type timestamp. This code results in
> org.postgresql.util.PSQLException syntax error
>
>
> String sql = "select * from view1 where `my date` = to_timestamp('2021-04-01
> 00:00:00', '-MM-dd HH:mm:ss')"; // this query also doesn't work
> String sql = "select * from view1 where `my date` = date_trunc('DAY',
> to_timestamp('2021-04-01 00:00:00', '-MM-dd HH:mm:ss'))"; // but this is
> OK
>
> Is it a bug or I got something wrong?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46077) Error in postgresql when pushing down filter by timestamp_ntz field
[
https://issues.apache.org/jira/browse/SPARK-46077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-46077:
Assignee: Pablo Langa Blanco
> Error in postgresql when pushing down filter by timestamp_ntz field
> ---
>
> Key: SPARK-46077
> URL: https://issues.apache.org/jira/browse/SPARK-46077
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Marina Krasilnikova
>Assignee: Pablo Langa Blanco
>Priority: Minor
> Labels: pull-request-available
>
> code to reproduce:
> SparkSession sparkSession = SparkSession
> .builder()
> .appName("test-app")
> .master("local[*]")
> .config("spark.sql.timestampType", "TIMESTAMP_NTZ")
> .getOrCreate();
> String url = "...";
> String catalogPropPrefix = "spark.sql.catalog.myc";
> sparkSession.conf().set(catalogPropPrefix, JDBCTableCatalog.class.getName());
> sparkSession.conf().set(catalogPropPrefix + ".url", url);
> Map options = new HashMap<>();
> options.put("driver", "org.postgresql.Driver");
> // options.put("pushDownPredicate", "false"); it works fine if this line is
> uncommented
> Dataset dataset = sparkSession.read()
> .options(options)
> .table("myc.demo.`My table`");
> dataset.createOrReplaceTempView("view1");
> String sql = "select * from view1 where `my date` = '2021-04-01 00:00:00'";
> Dataset result = sparkSession.sql(sql);
> result.show();
> result.printSchema();
> Field `my date` is of type timestamp. This code results in
> org.postgresql.util.PSQLException syntax error
>
>
> String sql = "select * from view1 where `my date` = to_timestamp('2021-04-01
> 00:00:00', '-MM-dd HH:mm:ss')"; // this query also doesn't work
> String sql = "select * from view1 where `my date` = date_trunc('DAY',
> to_timestamp('2021-04-01 00:00:00', '-MM-dd HH:mm:ss'))"; // but this is
> OK
>
> Is it a bug or I got something wrong?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47192) Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature)
[ https://issues.apache.org/jira/browse/SPARK-47192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47192: Assignee: Serge Rielau > Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature) > -- > > Key: SPARK-47192 > URL: https://issues.apache.org/jira/browse/SPARK-47192 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > > Old: > > GRANT ROLE; > _LEGACY_ERROR_TEMP_0035 > Operation not allowed: grant role. (line 1, pos 0) > > New: > error class: HIVE_OPERATION_NOT_SUPPORTED > The Hive operation is not supported. (line 1, pos 0) > > sqlstate: 0A000 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47192) Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature)
[ https://issues.apache.org/jira/browse/SPARK-47192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47192. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45291 [https://github.com/apache/spark/pull/45291] > Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature) > -- > > Key: SPARK-47192 > URL: https://issues.apache.org/jira/browse/SPARK-47192 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Old: > > GRANT ROLE; > _LEGACY_ERROR_TEMP_0035 > Operation not allowed: grant role. (line 1, pos 0) > > New: > error class: HIVE_OPERATION_NOT_SUPPORTED > The Hive operation is not supported. (line 1, pos 0) > > sqlstate: 0A000 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
