[jira] [Created] (SPARK-46051) Cache python dep for linter and documentation

2023-11-21 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-46051: - Summary: Cache python dep for linter and documentation Key: SPARK-46051 URL: https://issues.apache.org/jira/browse/SPARK-46051 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46051) Cache python deps for linter and documentation

2023-11-21 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-46051: -- Summary: Cache python deps for linter and documentation (was: Cache python dep for linter

[jira] [Resolved] (SPARK-44570) Reuse spark build among pyspark-* modules

2023-11-21 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-44570. --- Resolution: Not A Problem > Reuse spark build among pyspark-* modules >

[jira] [Resolved] (SPARK-44568) Daily GA for Python 3.10

2023-11-21 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-44568. --- Resolution: Duplicate > Daily GA for Python 3.10 > > >

[jira] [Updated] (SPARK-46050) Allow providing additional rules for the Substitution batch

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46050: --- Labels: pull-request-available (was: ) > Allow providing additional rules for the

[jira] [Updated] (SPARK-46050) Allow providing additional rules for the Substitution batch

2023-11-21 Thread Eduard Tudenhoefner (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eduard Tudenhoefner updated SPARK-46050: Description: In order to support specifying addititional rules for the

[jira] [Created] (SPARK-46050) Allow providing additional rules for the Substitution batch

2023-11-21 Thread Eduard Tudenhoefner (Jira)
Eduard Tudenhoefner created SPARK-46050: --- Summary: Allow providing additional rules for the Substitution batch Key: SPARK-46050 URL: https://issues.apache.org/jira/browse/SPARK-46050 Project:

[jira] [Resolved] (SPARK-46048) Support groupingSets operation in PySpark

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46048. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43951

[jira] [Assigned] (SPARK-46048) Support groupingSets operation in PySpark

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46048: Assignee: Hyukjin Kwon > Support groupingSets operation in PySpark >

[jira] [Assigned] (SPARK-46038) Upgrade log4j2 to 2.22.0

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46038: - Assignee: Yang Jie > Upgrade log4j2 to 2.22.0 > > >

[jira] [Assigned] (SPARK-46045) Add individual categories for `Options and settings` to API reference

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46045: - Assignee: Haejoon Lee > Add individual categories for `Options and settings` to API

[jira] [Resolved] (SPARK-46038) Upgrade log4j2 to 2.22.0

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46038. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43940

[jira] [Resolved] (SPARK-46045) Add individual categories for `Options and settings` to API reference

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46045. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43948

[jira] [Assigned] (SPARK-46035) Upgrade ZSTD-JNI to 1.5.5-10

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46035: - Assignee: Yang Jie > Upgrade ZSTD-JNI to 1.5.5-10 > > >

[jira] [Resolved] (SPARK-46035) Upgrade ZSTD-JNI to 1.5.5-10

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46035. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43937

[jira] [Updated] (SPARK-46048) Support groupingSets operation in PySpark

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46048: --- Labels: pull-request-available (was: ) > Support groupingSets operation in PySpark >

[jira] [Updated] (SPARK-46048) Support groupingSets operation in PySpark

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46048: - Issue Type: New Feature (was: Bug) > Support groupingSets operation in PySpark >

[jira] [Created] (SPARK-46048) Support groupingSets operation in PySpark

2023-11-21 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-46048: Summary: Support groupingSets operation in PySpark Key: SPARK-46048 URL: https://issues.apache.org/jira/browse/SPARK-46048 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-46049) Support groupingSets operation in PySpark (Spark Connect)

2023-11-21 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-46049: Summary: Support groupingSets operation in PySpark (Spark Connect) Key: SPARK-46049 URL: https://issues.apache.org/jira/browse/SPARK-46049 Project: Spark

[jira] [Updated] (SPARK-46047) Remove the compilation suppression rule related to the usage of deprecated APIs in Scala 2.13

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46047: --- Labels: pull-request-available (was: ) > Remove the compilation suppression rule related

[jira] [Resolved] (SPARK-45697) Fix `Unicode escapes in triple quoted strings are deprecated`

2023-11-21 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45697. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43603

[jira] [Created] (SPARK-46047) Remove the compilation suppression rule related to the usage of deprecated APIs in Scala 2.13

2023-11-21 Thread Yang Jie (Jira)
Yang Jie created SPARK-46047: Summary: Remove the compilation suppression rule related to the usage of deprecated APIs in Scala 2.13 Key: SPARK-46047 URL: https://issues.apache.org/jira/browse/SPARK-46047

[jira] [Updated] (SPARK-33805) Eliminate deprecated usage since Scala 2.13

2023-11-21 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-33805: - Affects Version/s: 3.2.0 (was: 4.0.0) > Eliminate deprecated usage since

[jira] [Updated] (SPARK-33805) Eliminate deprecated usage since Scala 2.13

2023-11-21 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-33805: - Affects Version/s: 4.0.0 (was: 3.2.0) > Eliminate deprecated usage since

[jira] [Assigned] (SPARK-45696) Fix `method tryCompleteWith in trait Promise is deprecated`

2023-11-21 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45696: Assignee: Yang Jie > Fix `method tryCompleteWith in trait Promise is deprecated` >

[jira] [Resolved] (SPARK-45696) Fix `method tryCompleteWith in trait Promise is deprecated`

2023-11-21 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45696. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43556

[jira] [Updated] (SPARK-46043) Support create table using DSv2 sources

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46043: --- Labels: pull-request-available (was: ) > Support create table using DSv2 sources >

[jira] [Updated] (SPARK-46046) Isolated classloader per spark session

2023-11-21 Thread Faiz Halde (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faiz Halde updated SPARK-46046: --- Description: Hello, We use spark 3.5.0 and were wondering if the following is achievable using

[jira] [Updated] (SPARK-46046) Isolated classloader per spark session

2023-11-21 Thread Faiz Halde (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faiz Halde updated SPARK-46046: --- Description: Hello, We use spark 3.5.0 and were wondering if the following is achievable using

[jira] [Created] (SPARK-46046) Isolated classloader per spark session

2023-11-21 Thread Faiz Halde (Jira)
Faiz Halde created SPARK-46046: -- Summary: Isolated classloader per spark session Key: SPARK-46046 URL: https://issues.apache.org/jira/browse/SPARK-46046 Project: Spark Issue Type: Question

[jira] [Assigned] (SPARK-46017) PySpark doc build doesn't work properly on Mac

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46017: Assignee: Haejoon Lee > PySpark doc build doesn't work properly on Mac >

[jira] [Resolved] (SPARK-46022) Remove deprecated functions APIs from documents

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46022. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43932

[jira] [Assigned] (SPARK-46022) Remove deprecated functions APIs from documents

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46022: Assignee: Haejoon Lee > Remove deprecated functions APIs from documents >

[jira] [Assigned] (SPARK-46013) Improve basic datasource examples

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46013: Assignee: Allison Wang > Improve basic datasource examples >

[jira] [Resolved] (SPARK-46017) PySpark doc build doesn't work properly on Mac

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46017. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43932

[jira] [Resolved] (SPARK-46013) Improve basic datasource examples

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46013. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43917

[jira] [Updated] (SPARK-46045) Add individual categories for `Options and settings` to API reference

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46045: --- Labels: pull-request-available (was: ) > Add individual categories for `Options and

[jira] [Created] (SPARK-46045) Add individual categories for `Options and settings` to API reference

2023-11-21 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46045: --- Summary: Add individual categories for `Options and settings` to API reference Key: SPARK-46045 URL: https://issues.apache.org/jira/browse/SPARK-46045 Project: Spark

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788595#comment-17788595 ] Bobby Wang commented on SPARK-46032: BTW, This morning, I tried it on the third machine, still got

[jira] [Created] (SPARK-46043) Support create table using DSv2 sources

2023-11-21 Thread Allison Wang (Jira)
Allison Wang created SPARK-46043: Summary: Support create table using DSv2 sources Key: SPARK-46043 URL: https://issues.apache.org/jira/browse/SPARK-46043 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788584#comment-17788584 ] Bobby Wang commented on SPARK-46032: Hi [~gurwls223], I've pasted all the logs, please tell me if

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788582#comment-17788582 ] Bobby Wang edited comment on SPARK-46032 at 11/22/23 12:54 AM: --- h1.

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788583#comment-17788583 ] Bobby Wang commented on SPARK-46032: Executor Logs, {code:java} Spark Executor Command:

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788582#comment-17788582 ] Bobby Wang commented on SPARK-46032: h1. *Submit the spark application via pyspark* I installed the

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788580#comment-17788580 ] Bobby Wang edited comment on SPARK-46032 at 11/22/23 12:46 AM: --- h1.

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788581#comment-17788581 ] Bobby Wang commented on SPARK-46032: h1. Start spark-connect-server which will start Spark Driver

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788580#comment-17788580 ] Bobby Wang commented on SPARK-46032: h1. *Standalone Cluster* h2. Master Log {code:java} Spark

[jira] [Updated] (SPARK-46040) Update API for 'analyze' partitioning/ordering columns to support general expressions

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46040: --- Labels: pull-request-available (was: ) > Update API for 'analyze' partitioning/ordering

[jira] [Updated] (SPARK-46042) Reenable a `releaseSession` test case in SparkConnectServiceE2ESuite

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46042: - Description: https://github.com/apache/spark/pull/43942#issuecomment-1821896165 > Reenable a

[jira] [Resolved] (SPARK-46039) Upgrade `grpcio*` to 1.59.3 for Python 3.12

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46039. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43942

[jira] [Created] (SPARK-46042) Reenable a `releaseSession` test case in SparkConnectServiceE2ESuite

2023-11-21 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-46042: - Summary: Reenable a `releaseSession` test case in SparkConnectServiceE2ESuite Key: SPARK-46042 URL: https://issues.apache.org/jira/browse/SPARK-46042 Project:

[jira] [Created] (SPARK-46040) Update API for 'analyze' partitioning/ordering columns to support general expressions

2023-11-21 Thread Daniel (Jira)
Daniel created SPARK-46040: -- Summary: Update API for 'analyze' partitioning/ordering columns to support general expressions Key: SPARK-46040 URL: https://issues.apache.org/jira/browse/SPARK-46040 Project:

[jira] [Commented] (SPARK-44973) Fix ArrayIndexOutOfBoundsException in conv()

2023-11-21 Thread Mark Jarvin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788564#comment-17788564 ] Mark Jarvin commented on SPARK-44973: - Hmm, possibly it goes back to Spark 2.1.0 due to this line:

[jira] [Updated] (SPARK-45649) Unify the prepare framework for `OffsetWindowFunctionFrame`

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45649: -- Fix Version/s: (was: 4.0.0) > Unify the prepare framework for `OffsetWindowFunctionFrame`

[jira] [Updated] (SPARK-44973) Fix ArrayIndexOutOfBoundsException in conv()

2023-11-21 Thread Gera Shegalov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated SPARK-44973: -- Affects Version/s: 3.0.3 > Fix ArrayIndexOutOfBoundsException in conv() >

[jira] [Updated] (SPARK-44973) Fix ArrayIndexOutOfBoundsException in conv()

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44973: -- Summary: Fix ArrayIndexOutOfBoundsException in conv() (was: CONV('-9223372036854775808', 10,

[jira] [Assigned] (SPARK-44973) CONV('-9223372036854775808', 10, -2) throws ArrayIndexOutOfBoundsException

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44973: - Assignee: Mark Jarvin > CONV('-9223372036854775808', 10, -2) throws

[jira] [Resolved] (SPARK-44973) CONV('-9223372036854775808', 10, -2) throws ArrayIndexOutOfBoundsException

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44973. --- Fix Version/s: 3.3.4 3.5.1 4.0.0

[jira] [Assigned] (SPARK-46028) Make `Column.__getitem__` accept input column

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46028: - Assignee: Ruifeng Zheng > Make `Column.__getitem__` accept input column >

[jira] [Resolved] (SPARK-46028) Make `Column.__getitem__` accept input column

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46028. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43930

[jira] [Updated] (SPARK-46039) Upgrade `grpcio*` to 1.59.3 for Python 3.12

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46039: -- Summary: Upgrade `grpcio*` to 1.59.3 for Python 3.12 (was: Upgrade `grpcio*` to 1.59.3) >

[jira] [Created] (SPARK-46039) Upgrade `grpcio*` to 1.59.3

2023-11-21 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-46039: - Summary: Upgrade `grpcio*` to 1.59.3 Key: SPARK-46039 URL: https://issues.apache.org/jira/browse/SPARK-46039 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-46037) When Left Join build Left, ShuffledHashJoinExec may result in incorrect results

2023-11-21 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-46037: --- Labels: correctness pull-request-available (was: pull-request-available) > When Left Join build

[jira] [Resolved] (SPARK-46019) Fix HiveThriftServer2ListenerSuite and ThriftServerPageSuite failed when there are running on local

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46019. --- Fix Version/s: 3.3.4 3.5.1 4.0.0

[jira] [Updated] (SPARK-46019) Fix HiveThriftServer2ListenerSuite and ThriftServerPageSuite to create java.io.tmpdir if it doesn't exist

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46019: -- Summary: Fix HiveThriftServer2ListenerSuite and ThriftServerPageSuite to create

[jira] [Resolved] (SPARK-46031) Replace `!Optional.isPresent()` with `Optional.isEmpty()`

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46031. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43931

[jira] [Resolved] (SPARK-46033) Fix flaky ArithmeticExpressionSuite

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46033. --- Fix Version/s: 3.5.1 4.0.0 3.4.2 Resolution:

[jira] [Assigned] (SPARK-46033) Fix flaky ArithmeticExpressionSuite

2023-11-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46033: - Assignee: BingKun Pan > Fix flaky ArithmeticExpressionSuite >

[jira] [Commented] (SPARK-38473) Use error classes in org.apache.spark.scheduler

2023-11-21 Thread Asmita Limaye (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788495#comment-17788495 ] Asmita Limaye commented on SPARK-38473: --- Hi [~bozhang], I was working on this issue (with Hannah)

[jira] [Updated] (SPARK-46038) Upgrade log4j2 to 2.22.0

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46038: --- Labels: pull-request-available (was: ) > Upgrade log4j2 to 2.22.0 >

[jira] [Created] (SPARK-46038) Upgrade log4j2 to 2.22.0

2023-11-21 Thread Yang Jie (Jira)
Yang Jie created SPARK-46038: Summary: Upgrade log4j2 to 2.22.0 Key: SPARK-46038 URL: https://issues.apache.org/jira/browse/SPARK-46038 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-23607) Use HDFS extended attributes to store application summary to improve the Spark History Server performance

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-23607: --- Labels: bulk-closed pull-request-available (was: bulk-closed) > Use HDFS extended

[jira] [Assigned] (SPARK-45533) Use `j.l.r.Cleaner` instead of `finalize` for `RocksDBIterator/LevelDBIterator`

2023-11-21 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45533: Assignee: Min Zhao > Use `j.l.r.Cleaner` instead of `finalize` for >

[jira] [Resolved] (SPARK-45533) Use `j.l.r.Cleaner` instead of `finalize` for `RocksDBIterator/LevelDBIterator`

2023-11-21 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45533. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43502

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788430#comment-17788430 ] Hyukjin Kwon commented on SPARK-46032: -- Are executors using the same versions too? The error is

[jira] [Updated] (SPARK-46037) When Left Join build Left, ShuffledHashJoinExec may result in incorrect results

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46037: --- Labels: pull-request-available (was: ) > When Left Join build Left, ShuffledHashJoinExec

[jira] [Created] (SPARK-46037) When Left Join build Left, ShuffledHashJoinExec may result in incorrect results

2023-11-21 Thread mcdull_zhang (Jira)
mcdull_zhang created SPARK-46037: Summary: When Left Join build Left, ShuffledHashJoinExec may result in incorrect results Key: SPARK-46037 URL: https://issues.apache.org/jira/browse/SPARK-46037

[jira] [Created] (SPARK-46036) raise_error should only take the error class parameter for internal usage

2023-11-21 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-46036: --- Summary: raise_error should only take the error class parameter for internal usage Key: SPARK-46036 URL: https://issues.apache.org/jira/browse/SPARK-46036 Project:

[jira] [Updated] (SPARK-46035) Upgrade ZSTD-JNI to 1.5.5-10

2023-11-21 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-46035: - Summary: Upgrade ZSTD-JNI to 1.5.5-10 (was: Upgrade ZSTD-JNI to 1.5.5-9) > Upgrade ZSTD-JNI to

[jira] [Updated] (SPARK-46035) Upgrade ZSTD-JNI to 1.5.5-9

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46035: --- Labels: pull-request-available (was: ) > Upgrade ZSTD-JNI to 1.5.5-9 >

[jira] [Updated] (SPARK-46034) SparkContext add file should also copy file to local root path

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46034: --- Labels: pull-request-available (was: ) > SparkContext add file should also copy file to

[jira] [Created] (SPARK-46035) Upgrade ZSTD-JNI to 1.5.5-9

2023-11-21 Thread Yang Jie (Jira)
Yang Jie created SPARK-46035: Summary: Upgrade ZSTD-JNI to 1.5.5-9 Key: SPARK-46035 URL: https://issues.apache.org/jira/browse/SPARK-46035 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788384#comment-17788384 ] Bobby Wang commented on SPARK-46032: I tried it on another machine and got the same issue. The scala

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788382#comment-17788382 ] Bobby Wang edited comment on SPARK-46032 at 11/21/23 11:52 AM: ---

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788382#comment-17788382 ] Bobby Wang edited comment on SPARK-46032 at 11/21/23 11:52 AM: ---

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788382#comment-17788382 ] Bobby Wang edited comment on SPARK-46032 at 11/21/23 11:51 AM: ---

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788382#comment-17788382 ] Bobby Wang commented on SPARK-46032: {code:java} ~$ scala Welcome to Scala 2.13.12 (OpenJDK 64-Bit

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788381#comment-17788381 ] Hyukjin Kwon commented on SPARK-46032: -- and can you run without Spark Connect? Seems like just

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788380#comment-17788380 ] Hyukjin Kwon commented on SPARK-46032: -- What's your Scala version [~wbo4958]? > connect: cannot

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788380#comment-17788380 ] Hyukjin Kwon edited comment on SPARK-46032 at 11/21/23 11:34 AM: - What's

[jira] [Updated] (SPARK-46034) SparkContext add file should also copy file to local root path

2023-11-21 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-46034: -- Description: For below case, it failed with FileNotFoundException {code:java} add jar

[jira] [Updated] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46032: - Priority: Major (was: Blocker) > connect: cannot assign instance of

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788379#comment-17788379 ] Ruifeng Zheng commented on SPARK-46032: --- cc [~gurwls223]  would you mind taking a look? >

[jira] [Updated] (SPARK-46033) Fix flaky ArithmeticExpressionSuite

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46033: --- Labels: pull-request-available (was: ) > Fix flaky ArithmeticExpressionSuite >

[jira] [Created] (SPARK-46034) SparkContext add file should also copy file to local root path

2023-11-21 Thread angerszhu (Jira)
angerszhu created SPARK-46034: - Summary: SparkContext add file should also copy file to local root path Key: SPARK-46034 URL: https://issues.apache.org/jira/browse/SPARK-46034 Project: Spark

[jira] [Created] (SPARK-46033) Fix flaky ArithmeticExpressionSuite

2023-11-21 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-46033: --- Summary: Fix flaky ArithmeticExpressionSuite Key: SPARK-46033 URL: https://issues.apache.org/jira/browse/SPARK-46033 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788378#comment-17788378 ] Bobby Wang commented on SPARK-46032: I tried it on another machine but still had the same problem.

[jira] [Updated] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-46032: --- Description: I downloaded spark 3.5 from the spark official website, and then I started a Spark

[jira] [Updated] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-46032: --- Description: I downloaded spark 3.5 from the spark official website, and then I started a Spark

[jira] [Updated] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-46032: --- Description: I downloaded spark 3.5 from the spark official website, and then I started a Spark

  1   2   >