[jira] [Updated] (SPARK-45661) Add toNullable in StructType, MapType and ArrayType

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45661: --- Labels: pull-request-available (was: ) > Add toNullable in StructType, MapType and ArrayTyp

[jira] [Created] (SPARK-45661) Add toNullable in StructType, MapType and ArrayType

2023-10-24 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-45661: Summary: Add toNullable in StructType, MapType and ArrayType Key: SPARK-45661 URL: https://issues.apache.org/jira/browse/SPARK-45661 Project: Spark Issue Typ

[jira] [Commented] (SPARK-43108) org.apache.spark.storage.StorageStatus NotSerializableException when try to access StorageStatus in a MapPartitionsFunction

2023-10-24 Thread Berg Lloyd-Haig (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779355#comment-17779355 ] Berg Lloyd-Haig commented on SPARK-43108: - We are hitting this when reading from

[jira] [Updated] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-24 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adi Wehrli updated SPARK-45644: --- Description: I do not really know if this is a bug, but I am at the end with my knowledge. A Spark

[jira] [Updated] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-24 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adi Wehrli updated SPARK-45644: --- Issue Type: Bug (was: Question) > After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeExcept

[jira] [Assigned] (SPARK-45545) [CORE] Pass SSLOptions wherever we create a SparkTransportConf

2023-10-24 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45545: --- Assignee: Hasnain Lakhani > [CORE] Pass SSLOptions wherever we create a Spa

[jira] [Resolved] (SPARK-45545) [CORE] Pass SSLOptions wherever we create a SparkTransportConf

2023-10-24 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45545. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 433

[jira] [Updated] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-24 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adi Wehrli updated SPARK-45644: --- Description: A Spark job ran successfully with Spark 3.2.x and 3.3.x. But after upgrading to 3.4.1

[jira] [Created] (SPARK-45660) Re-use Literal objects when replacing timestamps in the ComputeCurrentTime rule

2023-10-24 Thread Jan-Ole Sasse (Jira)
Jan-Ole Sasse created SPARK-45660: - Summary: Re-use Literal objects when replacing timestamps in the ComputeCurrentTime rule Key: SPARK-45660 URL: https://issues.apache.org/jira/browse/SPARK-45660 Pro

[jira] [Updated] (SPARK-45659) Add `since` field to Java API marked as `@Deprecated`.

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45659: --- Labels: pull-request-available (was: ) > Add `since` field to Java API marked as `@Deprecat

[jira] [Created] (SPARK-45659) Add `since` field to Java API marked as `@Deprecated`.

2023-10-24 Thread Yang Jie (Jira)
Yang Jie created SPARK-45659: Summary: Add `since` field to Java API marked as `@Deprecated`. Key: SPARK-45659 URL: https://issues.apache.org/jira/browse/SPARK-45659 Project: Spark Issue Type: Im

[jira] [Resolved] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge resolved SPARK-45657. Fix Version/s: 3.5.0 Resolution: Fixed The issue is fixed in 3.5.0 > Caching SQL UNION of

[jira] [Updated] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-45657: --- Affects Version/s: 3.4.1 3.4.0 > Caching SQL UNION of different column data t

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-24 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45658: - Description: The canonicalization of (buildKeys: Seq[Expression]) in the class DynamicPruningSubquery is broken

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-24 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45658: - Priority: Major (was: Critical) > Canonicalization of DynamicPruningSubquery is broken > --

[jira] [Created] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-24 Thread Asif (Jira)
Asif created SPARK-45658: Summary: Canonicalization of DynamicPruningSubquery is broken Key: SPARK-45658 URL: https://issues.apache.org/jira/browse/SPARK-45658 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779281#comment-17779281 ] John Zhuge edited comment on SPARK-45657 at 10/25/23 4:55 AM:

[jira] [Commented] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779326#comment-17779326 ] John Zhuge commented on SPARK-45657: It is fixed in main branch {code:java}       __

[jira] [Commented] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779283#comment-17779283 ] John Zhuge commented on SPARK-45657: Interesting, there is warning in Dataset.union

[jira] [Updated] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-45657: --- Affects Version/s: 3.3.2 (was: 3.4.1) > Caching SQL UNION of different co

[jira] [Commented] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779282#comment-17779282 ] John Zhuge commented on SPARK-45657: Checking whether this is still an issue in main

[jira] [Comment Edited] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779281#comment-17779281 ] John Zhuge edited comment on SPARK-45657 at 10/25/23 12:38 AM: ---

[jira] [Comment Edited] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779281#comment-17779281 ] John Zhuge edited comment on SPARK-45657 at 10/25/23 12:36 AM: ---

[jira] [Commented] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779281#comment-17779281 ] John Zhuge commented on SPARK-45657: Root cause: # SQL UNION of 2 sides with differ

[jira] [Updated] (SPARK-45656) Fix observation when named observations with the same name on different datasets.

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45656: --- Labels: pull-request-available (was: ) > Fix observation when named observations with the s

[jira] [Created] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
John Zhuge created SPARK-45657: -- Summary: Caching SQL UNION of different column data types does not work inside Dataset.union Key: SPARK-45657 URL: https://issues.apache.org/jira/browse/SPARK-45657 Proje

[jira] [Created] (SPARK-45656) Fix observation when named observations with the same name on different datasets.

2023-10-24 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-45656: - Summary: Fix observation when named observations with the same name on different datasets. Key: SPARK-45656 URL: https://issues.apache.org/jira/browse/SPARK-45656 P

[jira] [Assigned] (SPARK-45648) Add sql/api and common/utils to modules.py

2023-10-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45648: Assignee: Ruifeng Zheng > Add sql/api and common/utils to modules.py > --

[jira] [Resolved] (SPARK-45648) Add sql/api and common/utils to modules.py

2023-10-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45648. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43501 [https://gi

[jira] [Updated] (SPARK-45648) Add sql/api and common/utils to modules.py

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45648: --- Labels: pull-request-available (was: ) > Add sql/api and common/utils to modules.py > -

[jira] [Resolved] (SPARK-45622) java -target should use java.version instead of 17

2023-10-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45622. -- Resolution: Invalid > java -target should use java.version instead of 17 > ---

[jira] [Assigned] (SPARK-45651) Snapshots of some packages are not published any more

2023-10-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45651: Assignee: Enrico Minack > Snapshots of some packages are not published any more > ---

[jira] [Resolved] (SPARK-45651) Snapshots of some packages are not published any more

2023-10-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45651. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43512 [https://gi

[jira] [Assigned] (SPARK-45640) Fix flaky ProtobufCatalystDataConversionSuite

2023-10-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45640: Assignee: BingKun Pan > Fix flaky ProtobufCatalystDataConversionSuite > -

[jira] [Resolved] (SPARK-45640) Fix flaky ProtobufCatalystDataConversionSuite

2023-10-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45640. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43493 [https://gi

[jira] [Commented] (SPARK-45655) current_date() not supported in Streaming Query Observed metrics

2023-10-24 Thread Bhuwan Sahni (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779264#comment-17779264 ] Bhuwan Sahni commented on SPARK-45655: -- PR link https://github.com/apache/spark/pul

[jira] [Updated] (SPARK-45655) current_date() not supported in Streaming Query Observed metrics

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45655: --- Labels: pull-request-available (was: ) > current_date() not supported in Streaming Query Ob

[jira] [Updated] (SPARK-45654) Add Python data source write API

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45654: --- Labels: pull-request-available (was: ) > Add Python data source write API > ---

[jira] [Created] (SPARK-45655) current_date() not supported in Streaming Query Observed metrics

2023-10-24 Thread Bhuwan Sahni (Jira)
Bhuwan Sahni created SPARK-45655: Summary: current_date() not supported in Streaming Query Observed metrics Key: SPARK-45655 URL: https://issues.apache.org/jira/browse/SPARK-45655 Project: Spark

[jira] [Commented] (SPARK-45655) current_date() not supported in Streaming Query Observed metrics

2023-10-24 Thread Bhuwan Sahni (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779239#comment-17779239 ] Bhuwan Sahni commented on SPARK-45655: -- I am working on a fix for this issue, and w

[jira] [Created] (SPARK-45654) Add Python data source write API

2023-10-24 Thread Allison Wang (Jira)
Allison Wang created SPARK-45654: Summary: Add Python data source write API Key: SPARK-45654 URL: https://issues.apache.org/jira/browse/SPARK-45654 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-45503) RocksDB State Store to Use LZ4 Compression

2023-10-24 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-45503. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43338 [https://gi

[jira] [Resolved] (SPARK-45653) Refractor XMLSuite to allow other test suites to easily extend and override.

2023-10-24 Thread Shujing Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shujing Yang resolved SPARK-45653. -- Resolution: Not A Problem > Refractor XMLSuite to allow other test suites to easily extend and

[jira] [Updated] (SPARK-45653) Refractor XMLSuite to allow other test suites to easily extend and override.

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45653: --- Labels: pull-request-available (was: ) > Refractor XMLSuite to allow other test suites to e

[jira] [Resolved] (SPARK-45524) Initial support for Python data source read API

2023-10-24 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-45524. --- Fix Version/s: 4.0.0 Assignee: Allison Wang Resolution: Fixed Issue resolved

[jira] [Updated] (SPARK-45653) Refractor XMLSuite to allow other test suites to easily extend and override.

2023-10-24 Thread Shujing Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shujing Yang updated SPARK-45653: - Summary: Refractor XMLSuite to allow other test suites to easily extend and override. (was: Ref

[jira] [Created] (SPARK-45653) Refractor XMLSuite

2023-10-24 Thread Shujing Yang (Jira)
Shujing Yang created SPARK-45653: Summary: Refractor XMLSuite Key: SPARK-45653 URL: https://issues.apache.org/jira/browse/SPARK-45653 Project: Spark Issue Type: Improvement Componen

[jira] [Assigned] (SPARK-45643) Replace `s.c.mutable.MapOps#transform` with `s.c.mutable.MapOps#mapValuesInPlace`

2023-10-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45643: - Assignee: Yang Jie > Replace `s.c.mutable.MapOps#transform` with > `s.c.mutable.MapOps

[jira] [Resolved] (SPARK-45643) Replace `s.c.mutable.MapOps#transform` with `s.c.mutable.MapOps#mapValuesInPlace`

2023-10-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45643. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43500 [https://

[jira] [Updated] (SPARK-45651) Snapshots of some packages are not published any more

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45651: --- Labels: pull-request-available (was: ) > Snapshots of some packages are not published any m

[jira] [Created] (SPARK-45652) SPJ: Handle empty input partitions after dynamic filtering

2023-10-24 Thread Chao Sun (Jira)
Chao Sun created SPARK-45652: Summary: SPJ: Handle empty input partitions after dynamic filtering Key: SPARK-45652 URL: https://issues.apache.org/jira/browse/SPARK-45652 Project: Spark Issue Type

[jira] [Updated] (SPARK-44405) Reduce code duplication in group-based DELETE and MERGE tests

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44405: --- Labels: pull-request-available (was: ) > Reduce code duplication in group-based DELETE and

[jira] [Created] (SPARK-45651) Snapshots of some packages are not published any more

2023-10-24 Thread Enrico Minack (Jira)
Enrico Minack created SPARK-45651: - Summary: Snapshots of some packages are not published any more Key: SPARK-45651 URL: https://issues.apache.org/jira/browse/SPARK-45651 Project: Spark Issue

[jira] [Assigned] (SPARK-45646) Remove hardcoding time variables prior to Hive 2.0

2023-10-24 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45646: Assignee: Cheng Pan > Remove hardcoding time variables prior to Hive 2.0 > --

[jira] [Resolved] (SPARK-45646) Remove hardcoding time variables prior to Hive 2.0

2023-10-24 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45646. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43506 [https://github.com

[jira] [Updated] (SPARK-45650) fix dev/mina get scala 2.12

2023-10-24 Thread tangjiafu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tangjiafu updated SPARK-45650: -- Description: Now the ci is executing ./dev/mina will generate an incompatible error with scala2.12.

[jira] [Created] (SPARK-45650) fix dev/mina get scala 2.12

2023-10-24 Thread tangjiafu (Jira)
tangjiafu created SPARK-45650: - Summary: fix dev/mina get scala 2.12 Key: SPARK-45650 URL: https://issues.apache.org/jira/browse/SPARK-45650 Project: Spark Issue Type: Sub-task Compone

[jira] [Commented] (SPARK-31836) input_file_name() gives wrong value following Python UDF usage

2023-10-24 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-31836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779048#comment-17779048 ] Rasmus Schøler Sørensen commented on SPARK-31836: - We have also encounte

[jira] [Updated] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-24 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adi Wehrli updated SPARK-45644: --- Description: A Spark job ran successfully with Spark 3.2.x and 3.3.x. But after upgrading to 3.4.1

[jira] [Updated] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-24 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adi Wehrli updated SPARK-45644: --- Summary: After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a v

[jira] [Updated] (SPARK-45649) Unify the prepare framework for `OffsetWindowFunctionFrame`

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45649: --- Labels: pull-request-available (was: ) > Unify the prepare framework for `OffsetWindowFunct

[jira] [Updated] (SPARK-45649) Unify the prepare framework for `OffsetWindowFunctionFrame`

2023-10-24 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45649: --- Summary: Unify the prepare framework for `OffsetWindowFunctionFrame` (was: Unified the prepare fram

[jira] [Created] (SPARK-45649) Unified the prepare framework for `OffsetWindowFunctionFrame`

2023-10-24 Thread Jiaan Geng (Jira)
Jiaan Geng created SPARK-45649: -- Summary: Unified the prepare framework for `OffsetWindowFunctionFrame` Key: SPARK-45649 URL: https://issues.apache.org/jira/browse/SPARK-45649 Project: Spark Is

[jira] [Created] (SPARK-45648) Add sql/api and common/utils to modules.py

2023-10-24 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-45648: - Summary: Add sql/api and common/utils to modules.py Key: SPARK-45648 URL: https://issues.apache.org/jira/browse/SPARK-45648 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-45647) Spark Connect API to propagate per request context

2023-10-24 Thread Juliusz Sompolski (Jira)
Juliusz Sompolski created SPARK-45647: - Summary: Spark Connect API to propagate per request context Key: SPARK-45647 URL: https://issues.apache.org/jira/browse/SPARK-45647 Project: Spark

[jira] [Resolved] (SPARK-45561) Convert TINYINT catalyst properly in MySQL Dialect

2023-10-24 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-45561. -- Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-45561) Convert TINYINT catalyst properly in MySQL Dialect

2023-10-24 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-45561: Assignee: Michael Zhang > Convert TINYINT catalyst properly in MySQL Dialect > --

[jira] [Updated] (SPARK-26052) Spark should output a _SUCCESS file for every partition correctly written

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-26052: --- Labels: bulk-closed pull-request-available (was: bulk-closed) > Spark should output a _SUCC

[jira] [Assigned] (SPARK-45575) support time travel options for df read API

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45575: -- Assignee: Apache Spark > support time travel options for df read API > --

[jira] [Assigned] (SPARK-45575) support time travel options for df read API

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45575: -- Assignee: (was: Apache Spark) > support time travel options for df read API > ---

[jira] [Assigned] (SPARK-45646) Remove hardcoding time variables prior to Hive 2.0

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45646: -- Assignee: (was: Apache Spark) > Remove hardcoding time variables prior to Hive 2.

[jira] [Assigned] (SPARK-45646) Remove hardcoding time variables prior to Hive 2.0

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45646: -- Assignee: Apache Spark > Remove hardcoding time variables prior to Hive 2.0 > ---

[jira] [Assigned] (SPARK-45575) support time travel options for df read API

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45575: -- Assignee: Apache Spark > support time travel options for df read API > --

[jira] [Assigned] (SPARK-45575) support time travel options for df read API

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45575: -- Assignee: (was: Apache Spark) > support time travel options for df read API > ---

[jira] [Assigned] (SPARK-45575) support time travel options for df read API

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45575: -- Assignee: Apache Spark > support time travel options for df read API > --

[jira] [Assigned] (SPARK-45575) support time travel options for df read API

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45575: -- Assignee: (was: Apache Spark) > support time travel options for df read API > ---

[jira] [Assigned] (SPARK-42746) Add the LISTAGG() aggregate function

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-42746: -- Assignee: Apache Spark > Add the LISTAGG() aggregate function > -

[jira] [Updated] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "is not valid external type"

2023-10-24 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adi Wehrli updated SPARK-45644: --- Description: A Spark job ran successfully with Spark 3.2.x and 3.3.x. But after upgrading to 3.4.1

[jira] [Updated] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "is not valid external type"

2023-10-24 Thread Adi Wehrli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adi Wehrli updated SPARK-45644: --- Description: A Spark job ran successfully with Spark 3.2.x and 3.3.x. But after upgrading to 3.4.1

[jira] [Resolved] (SPARK-44752) XML: Update Spark Docs

2023-10-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44752. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43350 [https://gi

[jira] [Assigned] (SPARK-44752) XML: Update Spark Docs

2023-10-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44752: Assignee: tangjiafu > XML: Update Spark Docs > -- > >

[jira] [Resolved] (SPARK-45641) Display the start time of an app following the Total Uptime segment on AllJobsPage

2023-10-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45641. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43495 [https://

[jira] [Assigned] (SPARK-45641) Display the start time of an app following the Total Uptime segment on AllJobsPage

2023-10-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45641: - Assignee: Kent Yao > Display the start time of an app following the Total Uptime segmen

[jira] [Updated] (SPARK-45646) Remove hardcoding time variables prior to Hive 2.0

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45646: --- Labels: pull-request-available (was: ) > Remove hardcoding time variables prior to Hive 2.0

[jira] [Assigned] (SPARK-45626) Convert _LEGACY_ERROR_TEMP_1055 to REQUIRES_SINGLE_PART_NAMESPACE

2023-10-24 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-45626: Assignee: BingKun Pan > Convert _LEGACY_ERROR_TEMP_1055 to REQUIRES_SINGLE_PART_NAMESPACE > -

[jira] [Resolved] (SPARK-45626) Convert _LEGACY_ERROR_TEMP_1055 to REQUIRES_SINGLE_PART_NAMESPACE

2023-10-24 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-45626. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43479 [https://github.com

[jira] [Created] (SPARK-45646) Remove hardcoding time variables prior to Hive 2.0

2023-10-24 Thread Cheng Pan (Jira)
Cheng Pan created SPARK-45646: - Summary: Remove hardcoding time variables prior to Hive 2.0 Key: SPARK-45646 URL: https://issues.apache.org/jira/browse/SPARK-45646 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-45626) Convert _LEGACY_ERROR_TEMP_1055 to REQUIRES_SINGLE_PART_NAMESPACE

2023-10-24 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-45626: Summary: Convert _LEGACY_ERROR_TEMP_1055 to REQUIRES_SINGLE_PART_NAMESPACE (was: Fix variable nam

[jira] [Assigned] (SPARK-45630) Replace `s.c.mutable.MapOps#retain` with `s.c.mutable.MapOps#filterInPlace`

2023-10-24 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45630: Assignee: Yang Jie > Replace `s.c.mutable.MapOps#retain` with `s.c.mutable.MapOps#filterInPlace`

[jira] [Resolved] (SPARK-45430) FramelessOffsetWindowFunctionFrame fails when ignore nulls and offset > # of rows

2023-10-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45430. - Fix Version/s: 3.3.4 3.5.1 4.0.0 3.4.2

[jira] [Assigned] (SPARK-45430) FramelessOffsetWindowFunctionFrame fails when ignore nulls and offset > # of rows

2023-10-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45430: --- Assignee: Vitalii Li > FramelessOffsetWindowFunctionFrame fails when ignore nulls and offse

[jira] [Resolved] (SPARK-45630) Replace `s.c.mutable.MapOps#retain` with `s.c.mutable.MapOps#filterInPlace`

2023-10-24 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45630. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43482 [https://github.com

[jira] [Updated] (SPARK-45645) Fix `FileSystem.isFile & FileSystem.isDirectory is deprecated`

2023-10-24 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-45645: - Parent: (was: SPARK-45314) Issue Type: Improvement (was: Sub-task) > Fix `FileSystem.isFile