[jira] [Updated] (SPARK-45755) Push down limit through Dataset.isEmpty()

2023-10-31 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-45755: Description: Push down LocalLimit can not optimize the case of distinct. {code:scala} def

[jira] [Created] (SPARK-45755) Push down limit through Dataset.isEmpty()

2023-10-31 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-45755: --- Summary: Push down limit through Dataset.isEmpty() Key: SPARK-45755 URL: https://issues.apache.org/jira/browse/SPARK-45755 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-45754) Support `spark.deploy.appIdPattern`

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45754: --- Labels: pull-request-available (was: ) > Support `spark.deploy.appIdPattern` >

[jira] [Created] (SPARK-45754) Support `spark.deploy.appIdPattern`

2023-10-31 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-45754: - Summary: Support `spark.deploy.appIdPattern` Key: SPARK-45754 URL: https://issues.apache.org/jira/browse/SPARK-45754 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45753) Support `spark.deploy.driverIdPattern`

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45753: --- Labels: pull-request-available (was: ) > Support `spark.deploy.driverIdPattern` >

[jira] [Created] (SPARK-45753) Support `spark.deploy.driverIdPattern`

2023-10-31 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-45753: - Summary: Support `spark.deploy.driverIdPattern` Key: SPARK-45753 URL: https://issues.apache.org/jira/browse/SPARK-45753 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45752) Unreferenced CTE should all be checked by CheckAnalysis0

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45752: --- Labels: pull-request-available (was: ) > Unreferenced CTE should all be checked by

[jira] [Created] (SPARK-45752) Unreferenced CTE should all be checked by CheckAnalysis0

2023-10-31 Thread Rui Wang (Jira)
Rui Wang created SPARK-45752: Summary: Unreferenced CTE should all be checked by CheckAnalysis0 Key: SPARK-45752 URL: https://issues.apache.org/jira/browse/SPARK-45752 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45734) Upgrade commons-io to 2.15.0

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45734: --- Labels: pull-request-available (was: ) > Upgrade commons-io to 2.15.0 >

[jira] [Resolved] (SPARK-45734) Upgrade commons-io to 2.15.0

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45734. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43592

[jira] [Assigned] (SPARK-45734) Upgrade commons-io to 2.15.0

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45734: - Assignee: Yang Jie > Upgrade commons-io to 2.15.0 > > >

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781589#comment-17781589 ] Adi Wehrli commented on SPARK-45644: Dear [~bersprockets] That's very kind of you. Thanks a lot.

[jira] [Updated] (SPARK-45751) The default value of ‘spark.executor.logs.rolling.maxRetainedFiles' on the official website is incorrect

2023-10-31 Thread chenyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenyu updated SPARK-45751: --- Attachment: the value on the website.png > The default value of

[jira] [Updated] (SPARK-45751) The default value of ‘spark.executor.logs.rolling.maxRetainedFiles' on the official website is incorrect

2023-10-31 Thread chenyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenyu updated SPARK-45751: --- Attachment: the default value.png > The default value of ‘spark.executor.logs.rolling.maxRetainedFiles' on

[jira] [Created] (SPARK-45751) The default value of ‘spark.executor.logs.rolling.maxRetainedFiles' on the official website is incorrect

2023-10-31 Thread chenyu (Jira)
chenyu created SPARK-45751: -- Summary: The default value of ‘spark.executor.logs.rolling.maxRetainedFiles' on the official website is incorrect Key: SPARK-45751 URL: https://issues.apache.org/jira/browse/SPARK-45751

[jira] [Updated] (SPARK-45748) Add a `fromSQL` functionality for Literals

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45748: --- Labels: pull-request-available (was: ) > Add a `fromSQL` functionality for Literals >

[jira] [Resolved] (SPARK-45750) The param 'spark.shuffle.io.preferDirectBufs' is useless

2023-10-31 Thread chenyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenyu resolved SPARK-45750. Resolution: Won't Fix > The param 'spark.shuffle.io.preferDirectBufs' is useless >

[jira] [Closed] (SPARK-45750) The param 'spark.shuffle.io.preferDirectBufs' is useless

2023-10-31 Thread chenyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenyu closed SPARK-45750. -- > The param 'spark.shuffle.io.preferDirectBufs' is useless >

[jira] [Created] (SPARK-45750) The param 'spark.shuffle.io.preferDirectBufs' is useless

2023-10-31 Thread chenyu (Jira)
chenyu created SPARK-45750: -- Summary: The param 'spark.shuffle.io.preferDirectBufs' is useless Key: SPARK-45750 URL: https://issues.apache.org/jira/browse/SPARK-45750 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-45749) Fix Spark History Server to sort `Duration` column properly

2023-10-31 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-45749: Assignee: Dongjoon Hyun > Fix Spark History Server to sort `Duration` column properly >

[jira] [Resolved] (SPARK-45749) Fix Spark History Server to sort `Duration` column properly

2023-10-31 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-45749. -- Fix Version/s: 3.3.4 3.5.1 4.0.0 3.4.2

[jira] [Updated] (SPARK-45749) Fix Spark History Server to sort `Duration` column properly

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45749: --- Labels: pull-request-available (was: ) > Fix Spark History Server to sort `Duration`

[jira] [Created] (SPARK-45749) Fix Spark History Server to sort `Duration` column properly

2023-10-31 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-45749: - Summary: Fix Spark History Server to sort `Duration` column properly Key: SPARK-45749 URL: https://issues.apache.org/jira/browse/SPARK-45749 Project: Spark

[jira] [Updated] (SPARK-45748) Add a `fromSQL` functionality for Literals

2023-10-31 Thread Xinyi Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yu updated SPARK-45748: - Issue Type: Improvement (was: Bug) > Add a `fromSQL` functionality for Literals >

[jira] [Created] (SPARK-45748) Add a `fromSQL` functionality for Literals

2023-10-31 Thread Xinyi Yu (Jira)
Xinyi Yu created SPARK-45748: Summary: Add a `fromSQL` functionality for Literals Key: SPARK-45748 URL: https://issues.apache.org/jira/browse/SPARK-45748 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-45654) Add Python data source write API

2023-10-31 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45654. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43516

[jira] [Assigned] (SPARK-45654) Add Python data source write API

2023-10-31 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45654: Assignee: Allison Wang > Add Python data source write API >

[jira] [Updated] (SPARK-45747) Support session window aggregation in state reader

2023-10-31 Thread Chaoqin Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoqin Li updated SPARK-45747: --- Summary: Support session window aggregation in state reader (was: Support session window operator

[jira] [Created] (SPARK-45747) Support session window operator in state reader

2023-10-31 Thread Chaoqin Li (Jira)
Chaoqin Li created SPARK-45747: -- Summary: Support session window operator in state reader Key: SPARK-45747 URL: https://issues.apache.org/jira/browse/SPARK-45747 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-45741) Upgrade Netty to 4.1.100.Final

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45741: - Assignee: Dongjoon Hyun > Upgrade Netty to 4.1.100.Final >

[jira] [Resolved] (SPARK-45741) Upgrade Netty to 4.1.100.Final

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45741. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43605

[jira] [Updated] (SPARK-45746) Return specific error messages if UDTF 'analyze' method accepts or returns wrong values

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45746: --- Labels: pull-request-available (was: ) > Return specific error messages if UDTF 'analyze'

[jira] [Commented] (SPARK-20075) Support classifier, packaging in Maven coordinates

2023-10-31 Thread Gera Shegalov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781550#comment-17781550 ] Gera Shegalov commented on SPARK-20075: --- This would be a great feature that can help spark-rapids

[jira] [Created] (SPARK-45746) Return specific error messages if UDTF 'analyze' method accepts or returns wrong values

2023-10-31 Thread Daniel (Jira)
Daniel created SPARK-45746: -- Summary: Return specific error messages if UDTF 'analyze' method accepts or returns wrong values Key: SPARK-45746 URL: https://issues.apache.org/jira/browse/SPARK-45746 Project:

[jira] [Assigned] (SPARK-45739) Catch IOException instead of EOFException alone for faulthandler

2023-10-31 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45739: Assignee: Hyukjin Kwon > Catch IOException instead of EOFException alone for

[jira] [Resolved] (SPARK-45739) Catch IOException instead of EOFException alone for faulthandler

2023-10-31 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45739. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43600

[jira] [Updated] (SPARK-45745) Extremely slow execution of sum of columns in Spark 3.4.1

2023-10-31 Thread Javier (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Javier updated SPARK-45745: --- Description: We are in the process of upgrading some pySpark jobs from Spark 3.1.2 to Spark 3.4.1 and some

[jira] [Commented] (SPARK-45745) Extremely slow execution of sum of columns in Spark 3.4.1

2023-10-31 Thread Javier (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781539#comment-17781539 ] Javier commented on SPARK-45745: I originally posted a comment in StackOverflow asking for feedback on

[jira] [Created] (SPARK-45745) Extremely slow execution of sum of columns in Spark 3.4.1

2023-10-31 Thread Javier (Jira)
Javier created SPARK-45745: -- Summary: Extremely slow execution of sum of columns in Spark 3.4.1 Key: SPARK-45745 URL: https://issues.apache.org/jira/browse/SPARK-45745 Project: Spark Issue Type:

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781531#comment-17781531 ] Bruce Robbins commented on SPARK-45644: --- I will look into it and try to submit a fix. If I can't,

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781526#comment-17781526 ] Adi Wehrli commented on SPARK-45644: Good evening, [~bersprockets] Thanks for your reproduction.

[jira] [Comment Edited] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781526#comment-17781526 ] Adi Wehrli edited comment on SPARK-45644 at 10/31/23 9:38 PM: -- Good

[jira] [Assigned] (SPARK-45744) Switch `spark.history.store.serializer` to use `PROTOBUF` by default

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45744: - Assignee: Dongjoon Hyun > Switch `spark.history.store.serializer` to use `PROTOBUF` by

[jira] [Commented] (SPARK-45699) Fix "Widening conversion from `TypeA` to `TypeB` is deprecated because it loses precision"

2023-10-31 Thread Hannah Amundson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781505#comment-17781505 ] Hannah Amundson commented on SPARK-45699: - [~LuciferYang] Do you have any suggestions for

[jira] [Updated] (SPARK-45744) Switch `spark.history.store.serializer` to use `PROTOBUF` by default

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45744: --- Labels: pull-request-available (was: ) > Switch `spark.history.store.serializer` to use

[jira] [Created] (SPARK-45744) Switch `spark.history.store.serializer` to use `PROTOBUF` by default

2023-10-31 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-45744: - Summary: Switch `spark.history.store.serializer` to use `PROTOBUF` by default Key: SPARK-45744 URL: https://issues.apache.org/jira/browse/SPARK-45744 Project:

[jira] [Updated] (SPARK-45743) Upgrade dropwizard metrics 4.2.21

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45743: --- Labels: pull-request-available (was: ) > Upgrade dropwizard metrics 4.2.21 >

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781494#comment-17781494 ] Bruce Robbins commented on SPARK-45644: --- OK, I can reproduce. I will take a look. I will also try

[jira] [Created] (SPARK-45743) Upgrade dropwizard metrics 4.2.21

2023-10-31 Thread Yang Jie (Jira)
Yang Jie created SPARK-45743: Summary: Upgrade dropwizard metrics 4.2.21 Key: SPARK-45743 URL: https://issues.apache.org/jira/browse/SPARK-45743 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-44896) Consider adding information os_prio, cpu, elapsed, tid, nid, etc., from the jstack tool

2023-10-31 Thread Hannah Amundson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781489#comment-17781489 ] Hannah Amundson commented on SPARK-44896: - Hello everyone (and [~yao]), I am a graduate student

[jira] [Commented] (SPARK-45190) XML: StructType schema issue in pyspark connect

2023-10-31 Thread Hannah Amundson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781487#comment-17781487 ] Hannah Amundson commented on SPARK-45190: - Hello everyone (and [~sandip.agarwala] ), I am a

[jira] [Commented] (SPARK-45699) Fix "Widening conversion from `TypeA` to `TypeB` is deprecated because it loses precision"

2023-10-31 Thread Hannah Amundson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781488#comment-17781488 ] Hannah Amundson commented on SPARK-45699: - Hello everyone (and [~LuciferYang]), I am a graduate

[jira] [Commented] (SPARK-38473) Use error classes in org.apache.spark.scheduler

2023-10-31 Thread Hannah Amundson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781486#comment-17781486 ] Hannah Amundson commented on SPARK-38473: - Hello, I am a graduate student at the University of

[jira] [Comment Edited] (SPARK-38473) Use error classes in org.apache.spark.scheduler

2023-10-31 Thread Hannah Amundson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781486#comment-17781486 ] Hannah Amundson edited comment on SPARK-38473 at 10/31/23 7:08 PM: ---

[jira] [Updated] (SPARK-45742) Introduce an implicit function for Scala Array to wrap into `immutable.ArraySeq`.

2023-10-31 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-45742: - Summary: Introduce an implicit function for Scala Array to wrap into `immutable.ArraySeq`. (was:

[jira] [Updated] (SPARK-45719) Upgrade AWS SDK to v2 for Kubernetes integration tests

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45719: --- Labels: pull-request-available (was: ) > Upgrade AWS SDK to v2 for Kubernetes integration

[jira] [Created] (SPARK-45741) Upgrade Netty to 4.1.100.Final

2023-10-31 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-45741: - Summary: Upgrade Netty to 4.1.100.Final Key: SPARK-45741 URL: https://issues.apache.org/jira/browse/SPARK-45741 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-45741) Upgrade Netty to 4.1.100.Final

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45741: --- Labels: pull-request-available (was: ) > Upgrade Netty to 4.1.100.Final >

[jira] [Updated] (SPARK-45172) Upgrade commons-compress.version to 1.24.0

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45172: -- Summary: Upgrade commons-compress.version to 1.24.0 (was: Upgrade commons-compress.version

[jira] [Updated] (SPARK-45282) Join loses records for cached datasets

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45282: -- Target Version/s: 3.4.2, 3.5.1 (was: 3.4.2) > Join loses records for cached datasets >

[jira] [Commented] (SPARK-45282) Join loses records for cached datasets

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781464#comment-17781464 ] Dongjoon Hyun commented on SPARK-45282: --- Thank you for sharing, [~koert]. > Join loses records

[jira] [Assigned] (SPARK-45737) Remove unnecessary `.toArray[InternalRow]` in `SparkPlan#executeTake` function.

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45737: - Assignee: Yang Jie > Remove unnecessary `.toArray[InternalRow]` in

[jira] [Resolved] (SPARK-45737) Remove unnecessary `.toArray[InternalRow]` in `SparkPlan#executeTake` function.

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45737. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43599

[jira] [Resolved] (SPARK-45700) Fix `The outer reference in this type test cannot be checked at run time`

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45700. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43582

[jira] [Assigned] (SPARK-45702) Fix `the type test for pattern TypeA cannot be checked at runtime`

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45702: - Assignee: Yang Jie > Fix `the type test for pattern TypeA cannot be checked at

[jira] [Resolved] (SPARK-45702) Fix `the type test for pattern TypeA cannot be checked at runtime`

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45702. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43582

[jira] [Resolved] (SPARK-45703) Fix `abstract type TypeA in type pattern Some[TypeA] is unchecked since it is eliminated by erasure`

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45703. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43582

[jira] [Assigned] (SPARK-45700) Fix `The outer reference in this type test cannot be checked at run time`

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45700: - Assignee: Yang Jie > Fix `The outer reference in this type test cannot be checked at

[jira] [Assigned] (SPARK-45683) Fix `method any2stringadd in object Predef is deprecated`

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45683: - Assignee: Yang Jie > Fix `method any2stringadd in object Predef is deprecated` >

[jira] [Resolved] (SPARK-45683) Fix `method any2stringadd in object Predef is deprecated`

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45683. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43580

[jira] [Resolved] (SPARK-45725) remove the non-default IN subquery runtime filter

2023-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45725. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43585

[jira] [Commented] (SPARK-45687) Fix `Passing an explicit array value to a Scala varargs method is deprecated`

2023-10-31 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781420#comment-17781420 ] Yang Jie commented on SPARK-45687: -- Thanks [~ivoson]  > Fix `Passing an explicit array value to a

[jira] [Commented] (SPARK-45687) Fix `Passing an explicit array value to a Scala varargs method is deprecated`

2023-10-31 Thread Tengfei Huang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781419#comment-17781419 ] Tengfei Huang commented on SPARK-45687: --- I will work on this. > Fix `Passing an explicit array

[jira] [Updated] (SPARK-41533) GRPC Errors on the client should be cleaned up

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-41533: --- Labels: pull-request-available (was: ) > GRPC Errors on the client should be cleaned up >

[jira] [Commented] (SPARK-45694) Fix `method signum in trait ScalaNumberProxy is deprecated`

2023-10-31 Thread Tengfei Huang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781374#comment-17781374 ] Tengfei Huang commented on SPARK-45694: --- sure, will include [SPARK-45695] Fix `method force in

[jira] [Resolved] (SPARK-45368) Remove scala2.12 compatibility logic for DoubleType, FloatType, Decimal

2023-10-31 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45368. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43456

[jira] [Assigned] (SPARK-45368) Remove scala2.12 compatibility logic for DoubleType, FloatType, Decimal

2023-10-31 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45368: Assignee: tangjiafu > Remove scala2.12 compatibility logic for DoubleType, FloatType,

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781348#comment-17781348 ] Adi Wehrli commented on SPARK-45644: But I can really not say which job statement causes this

[jira] [Updated] (SPARK-45740) Relax the node prefix of SparkPlanGraphCluster

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45740: --- Labels: pull-request-available (was: ) > Relax the node prefix of SparkPlanGraphCluster >

[jira] [Updated] (SPARK-45740) Relax the node prefix of SparkPlanGraphCluster

2023-10-31 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-45740: -- Summary: Relax the node prefix of SparkPlanGraphCluster (was: Release the node prefix of

[jira] [Created] (SPARK-45740) Release the node prefix of SparkPlanGraphCluster

2023-10-31 Thread XiDuo You (Jira)
XiDuo You created SPARK-45740: - Summary: Release the node prefix of SparkPlanGraphCluster Key: SPARK-45740 URL: https://issues.apache.org/jira/browse/SPARK-45740 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-45732) Upgrade commons-text to 1.11.0

2023-10-31 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45732. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43590

[jira] [Assigned] (SPARK-45732) Upgrade commons-text to 1.11.0

2023-10-31 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45732: Assignee: BingKun Pan > Upgrade commons-text to 1.11.0 > --

[jira] [Updated] (SPARK-45738) client will wait forever if session in spark connect server is evicted

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45738: --- Labels: pull-request-available (was: ) > client will wait forever if session in spark

[jira] [Updated] (SPARK-45739) Catch IOException instead of EOFException alone for faulthandler

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45739: --- Labels: pull-request-available (was: ) > Catch IOException instead of EOFException alone

[jira] [Created] (SPARK-45739) Catch IOException instead of EOFException alone for faulthandler

2023-10-31 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-45739: Summary: Catch IOException instead of EOFException alone for faulthandler Key: SPARK-45739 URL: https://issues.apache.org/jira/browse/SPARK-45739 Project: Spark

[jira] [Updated] (SPARK-45738) client will wait forever if session in spark connect server is evicted

2023-10-31 Thread xie shuiahu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xie shuiahu updated SPARK-45738: Description: Step1. start a spark connect server Step2. submit a spark job which will run long

[jira] [Created] (SPARK-45738) client will wait forever if session in spark connect server is evicted

2023-10-31 Thread xie shuiahu (Jira)
xie shuiahu created SPARK-45738: --- Summary: client will wait forever if session in spark connect server is evicted Key: SPARK-45738 URL: https://issues.apache.org/jira/browse/SPARK-45738 Project: Spark

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781276#comment-17781276 ] Adi Wehrli commented on SPARK-45644: Thanks for this information, [~bersprockets]. I could now cut

[jira] [Assigned] (SPARK-45737) Remove unnecessary `.toArray[InternalRow]` in `SparkPlan#executeTake` function.

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45737: -- Assignee: Apache Spark > Remove unnecessary `.toArray[InternalRow]` in

[jira] [Assigned] (SPARK-45737) Remove unnecessary `.toArray[InternalRow]` in `SparkPlan#executeTake` function.

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45737: -- Assignee: (was: Apache Spark) > Remove unnecessary `.toArray[InternalRow]` in

[jira] [Updated] (SPARK-45737) Remove unnecessary `.toArray[InternalRow]` in `SparkPlan#executeTake` function.

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45737: --- Labels: pull-request-available (was: ) > Remove unnecessary `.toArray[InternalRow]` in

[jira] [Created] (SPARK-45737) Remove unnecessary `.toArray[InternalRow]` in `SparkPlan#executeTake` function.

2023-10-31 Thread Yang Jie (Jira)
Yang Jie created SPARK-45737: Summary: Remove unnecessary `.toArray[InternalRow]` in `SparkPlan#executeTake` function. Key: SPARK-45737 URL: https://issues.apache.org/jira/browse/SPARK-45737 Project:

[jira] [Assigned] (SPARK-45735) Reenable CatalogTests without Spark Connect

2023-10-31 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45735: Assignee: Hyukjin Kwon > Reenable CatalogTests without Spark Connect >

[jira] [Resolved] (SPARK-45735) Reenable CatalogTests without Spark Connect

2023-10-31 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45735. -- Fix Version/s: 3.5.1 4.0.0 3.4.2 Resolution:

[jira] [Updated] (SPARK-45736) Use \s+ as separator when testing Kafka source or network source

2023-10-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45736: --- Labels: pull-request-available (was: ) > Use \s+ as separator when testing Kafka source or

[jira] [Assigned] (SPARK-45701) Clean up the deprecated API usage related to `SetOps`

2023-10-31 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45701: Assignee: Yang Jie > Clean up the deprecated API usage related to `SetOps` >

[jira] [Resolved] (SPARK-45701) Clean up the deprecated API usage related to `SetOps`

2023-10-31 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45701. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43575

[jira] [Created] (SPARK-45736) Use \s+ as separator when testing Kafka source or network source

2023-10-31 Thread Deng Ziming (Jira)
Deng Ziming created SPARK-45736: --- Summary: Use \s+ as separator when testing Kafka source or network source Key: SPARK-45736 URL: https://issues.apache.org/jira/browse/SPARK-45736 Project: Spark