[jira] [Created] (SPARK-42978) Derby&PG: RENAME cannot qualify a new-table-Name with a schema-Name.

Kent Yao created SPARK-42978: Summary: Derby&PG: RENAME cannot qualify a new-table-Name with a schema-Name. Key: SPARK-42978 URL: https://issues.apache.org/jira/browse/SPARK-42978 Project: Spark

[jira] [Created] (SPARK-42977) spark sql Disable vectorized faild

liu created SPARK-42977: --- Summary: spark sql Disable vectorized faild Key: SPARK-42977 URL: https://issues.apache.org/jira/browse/SPARK-42977 Project: Spark Issue Type: Question Components:

[jira] [Created] (SPARK-42976) spark sql Disable vectorized faild

liu created SPARK-42976: --- Summary: spark sql Disable vectorized faild Key: SPARK-42976 URL: https://issues.apache.org/jira/browse/SPARK-42976 Project: Spark Issue Type: Question Components:

[jira] [Created] (SPARK-42975) Cast result type to timestamp type for string +/- interval

Yuming Wang created SPARK-42975: --- Summary: Cast result type to timestamp type for string +/- interval Key: SPARK-42975 URL: https://issues.apache.org/jira/browse/SPARK-42975 Project: Spark Issu

[jira] [Commented] (SPARK-42932) Spark 3.3.2, with hadoop3, Error with java.io.IOException: Mkdirs failed to create file

[ https://issues.apache.org/jira/browse/SPARK-42932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706702#comment-17706702 ] Yuming Wang commented on SPARK-42932: - I can't reproduce this issue. Maybe it's your

[jira] [Commented] (SPARK-42968) Add option to skip using commit coordinator in StreamingWrite API

[ https://issues.apache.org/jira/browse/SPARK-42968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706698#comment-17706698 ] Anish Shrigondekar commented on SPARK-42968: Sent PR here: https://github.co

[jira] [Resolved] (SPARK-42971) When processing the WorkDirCleanup event, if appDirs is empty, should print workdir

[ https://issues.apache.org/jira/browse/SPARK-42971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42971. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40597 [https://gi

[jira] [Assigned] (SPARK-42971) When processing the WorkDirCleanup event, if appDirs is empty, should print workdir

[ https://issues.apache.org/jira/browse/SPARK-42971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42971: Assignee: Yang Jie > When processing the WorkDirCleanup event, if appDirs is empty, shoul

[jira] [Updated] (SPARK-42974) Restore `Utils#createTempDir` use `ShutdownHookManager.registerShutdownDeleteDir` to cleanup tempDir

[ https://issues.apache.org/jira/browse/SPARK-42974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-42974: - Summary: Restore `Utils#createTempDir` use `ShutdownHookManager.registerShutdownDeleteDir` to cleanup t

[jira] [Created] (SPARK-42974) Restoring `Utils#createTempDir` using `ShutdownHookManager.registerShutdownDeleteDir` cleanup temporary files

Yang Jie created SPARK-42974: Summary: Restoring `Utils#createTempDir` using `ShutdownHookManager.registerShutdownDeleteDir` cleanup temporary files Key: SPARK-42974 URL: https://issues.apache.org/jira/browse/SPARK-4

[jira] [Commented] (SPARK-42971) When processing the WorkDirCleanup event, if appDirs is empty, should print workdir

[ https://issues.apache.org/jira/browse/SPARK-42971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706671#comment-17706671 ] Snoot.io commented on SPARK-42971: -- User 'LuciferYang' has created a pull request for t

[jira] [Commented] (SPARK-42973) Upgrade buf to v1.16.0

[ https://issues.apache.org/jira/browse/SPARK-42973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706670#comment-17706670 ] Snoot.io commented on SPARK-42973: -- User 'panbingkun' has created a pull request for th

[jira] [Resolved] (SPARK-40610) Spark fall back to use getPartitions instead of getPartitionsByFilter when date_add functions used in where clause

[ https://issues.apache.org/jira/browse/SPARK-40610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-40610. - Resolution: Won't Fix > Spark fall back to use getPartitions instead of getPartitionsByFilter wh

[jira] [Commented] (SPARK-40610) Spark fall back to use getPartitions instead of getPartitionsByFilter when date_add functions used in where clause

[ https://issues.apache.org/jira/browse/SPARK-40610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706665#comment-17706665 ] Yuming Wang commented on SPARK-40610: - Users can {{set spark.sql.legacy.typeCoercion

[jira] [Commented] (SPARK-42787) in spark-py docker images, arrowkeys do not work in (scala) spark-shell

[ https://issues.apache.org/jira/browse/SPARK-42787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706653#comment-17706653 ] Yikun Jiang commented on SPARK-42787: - you can also paste the execute script here,

[jira] [Created] (SPARK-42973) Upgrade buf to v1.16.0

BingKun Pan created SPARK-42973: --- Summary: Upgrade buf to v1.16.0 Key: SPARK-42973 URL: https://issues.apache.org/jira/browse/SPARK-42973 Project: Spark Issue Type: Improvement Compon

[jira] [Created] (SPARK-42972) ExecutorAllocationManager cannot allocate new instances when all executors down.

Jiandan Yang created SPARK-42972: - Summary: ExecutorAllocationManager cannot allocate new instances when all executors down. Key: SPARK-42972 URL: https://issues.apache.org/jira/browse/SPARK-42972 Pr

[jira] [Resolved] (SPARK-42693) API Auditing

[ https://issues.apache.org/jira/browse/SPARK-42693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42693. -- Resolution: Done > API Auditing > > > Key: SPARK-42693 >

[jira] [Resolved] (SPARK-42862) Review and fix issues in Core API docs

[ https://issues.apache.org/jira/browse/SPARK-42862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42862. -- Resolution: Resolved > Review and fix issues in Core API docs > --

[jira] [Created] (SPARK-42971) When processing the WorkDirCleanup event, if appDirs is empty, should print workdir

Yang Jie created SPARK-42971: Summary: When processing the WorkDirCleanup event, if appDirs is empty, should print workdir Key: SPARK-42971 URL: https://issues.apache.org/jira/browse/SPARK-42971 Project:

[jira] [Resolved] (SPARK-42970) Reuse pyspark.sql.tests.test_arrow test cases

[ https://issues.apache.org/jira/browse/SPARK-42970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42970. -- Fix Version/s: 3.5.0 Assignee: Takuya Ueshin Resolution: Fixed Fixed in https:

[jira] [Created] (SPARK-42970) Reuse pyspark.sql.tests.test_arrow test cases

Takuya Ueshin created SPARK-42970: - Summary: Reuse pyspark.sql.tests.test_arrow test cases Key: SPARK-42970 URL: https://issues.apache.org/jira/browse/SPARK-42970 Project: Spark Issue Type: S

[jira] [Created] (SPARK-42969) Fix the comparison the result with Arrow optimization enabled/disabled.

Takuya Ueshin created SPARK-42969: - Summary: Fix the comparison the result with Arrow optimization enabled/disabled. Key: SPARK-42969 URL: https://issues.apache.org/jira/browse/SPARK-42969 Project: Sp

[jira] [Created] (SPARK-42968) Add option to skip using commit coordinator in StreamingWrite API

Anish Shrigondekar created SPARK-42968: -- Summary: Add option to skip using commit coordinator in StreamingWrite API Key: SPARK-42968 URL: https://issues.apache.org/jira/browse/SPARK-42968 Project

[jira] [Commented] (SPARK-42968) Add option to skip using commit coordinator in StreamingWrite API

[ https://issues.apache.org/jira/browse/SPARK-42968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706600#comment-17706600 ] Anish Shrigondekar commented on SPARK-42968: cc - [~kabhwan] - will send the

[jira] [Commented] (SPARK-37829) An outer-join using joinWith on DataFrames returns Rows with null fields instead of null values

[ https://issues.apache.org/jira/browse/SPARK-37829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706591#comment-17706591 ] Jason Xu commented on SPARK-37829: -- Our company encountered this issue during our migr

[jira] [Commented] (SPARK-42631) Support custom extensions in Spark Connect Scala client

[ https://issues.apache.org/jira/browse/SPARK-42631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706567#comment-17706567 ] Vitaliy Osipov commented on SPARK-42631: User 'tomvanbussel' has created a pull

[jira] [Commented] (SPARK-42846) Assign a name to the error class _LEGACY_ERROR_TEMP_2011

[ https://issues.apache.org/jira/browse/SPARK-42846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706566#comment-17706566 ] Max Gekk commented on SPARK-42846: -- [~shuwang] Sure. Go ahead. > Assign a name to the

[jira] [Commented] (SPARK-42846) Assign a name to the error class _LEGACY_ERROR_TEMP_2011

[ https://issues.apache.org/jira/browse/SPARK-42846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706544#comment-17706544 ] SHU WANG commented on SPARK-42846: -- I would love to try this one. Thanks.  > Assign a

[jira] [Created] (SPARK-42967) Fix SparkListenerTaskStart.stageAttemptId when a task is started after the stage is cancelled

Xingbo Jiang created SPARK-42967: Summary: Fix SparkListenerTaskStart.stageAttemptId when a task is started after the stage is cancelled Key: SPARK-42967 URL: https://issues.apache.org/jira/browse/SPARK-42967

[jira] [Commented] (SPARK-42966) Make memory stream implement SupportsReportStatistics

[ https://issues.apache.org/jira/browse/SPARK-42966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706520#comment-17706520 ] Shixiong Zhu commented on SPARK-42966: -- cc [~kabhwan]  > Make memory stream implem

[jira] [Created] (SPARK-42966) Make memory stream implement SupportsReportStatistics

Shixiong Zhu created SPARK-42966: Summary: Make memory stream implement SupportsReportStatistics Key: SPARK-42966 URL: https://issues.apache.org/jira/browse/SPARK-42966 Project: Spark Issue T

[jira] [Commented] (SPARK-35198) Add support for calling debugCodegen from Python & Java

[ https://issues.apache.org/jira/browse/SPARK-35198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706493#comment-17706493 ] Juanvi Soler commented on SPARK-35198: -- [~holden] [~maxgekk] Is this still relevant

[jira] [Resolved] (SPARK-42873) Define Spark SQL types as keywords

[ https://issues.apache.org/jira/browse/SPARK-42873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-42873. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40565 [https://github.com

[jira] [Updated] (SPARK-42965) metadata mismatch for StructField when running some tests.

[ https://issues.apache.org/jira/browse/SPARK-42965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-42965: Description: For some reason, the metadata of `StructField` is different in a few tests when usin

[jira] [Created] (SPARK-42965) metadata mismatch for StructField when running some tests.

Haejoon Lee created SPARK-42965: --- Summary: metadata mismatch for StructField when running some tests. Key: SPARK-42965 URL: https://issues.apache.org/jira/browse/SPARK-42965 Project: Spark Issu

[jira] [Commented] (SPARK-42787) in spark-py docker images, arrowkeys do not work in (scala) spark-shell

[ https://issues.apache.org/jira/browse/SPARK-42787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706426#comment-17706426 ] Max Rieger commented on SPARK-42787: [~yikunkero], sure {code:java} $ docker run -i

[jira] [Created] (SPARK-42964) PosgresDialect SqlState 42P07 also means table already exists

Kent Yao created SPARK-42964: Summary: PosgresDialect SqlState 42P07 also means table already exists Key: SPARK-42964 URL: https://issues.apache.org/jira/browse/SPARK-42964 Project: Spark Issue

[jira] [Created] (SPARK-42963) Extend SparkSessionExtensions to inject rules into AQE query stage optimizer

XiDuo You created SPARK-42963: - Summary: Extend SparkSessionExtensions to inject rules into AQE query stage optimizer Key: SPARK-42963 URL: https://issues.apache.org/jira/browse/SPARK-42963 Project: Spark

[jira] [Assigned] (SPARK-42954) Add `YearMonthIntervalType` to PySpark and Spark Connect Python Client

[ https://issues.apache.org/jira/browse/SPARK-42954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-42954: - Assignee: Ruifeng Zheng > Add `YearMonthIntervalType` to PySpark and Spark Connect Pyth

[jira] [Resolved] (SPARK-42954) Add `YearMonthIntervalType` to PySpark and Spark Connect Python Client

[ https://issues.apache.org/jira/browse/SPARK-42954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-42954. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40582 [https://

[jira] [Commented] (SPARK-42955) Skip classifyException and wrap AnalysisException for SparkThrowable

[ https://issues.apache.org/jira/browse/SPARK-42955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706290#comment-17706290 ] ASF GitHub Bot commented on SPARK-42955: User 'yaooqinn' has created a pull requ

[jira] [Resolved] (SPARK-42952) Simplify the parameter of analyzer rule PreprocessTableCreation and DataSourceAnalysis

[ https://issues.apache.org/jira/browse/SPARK-42952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-42952. Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40580 [https:

[jira] [Assigned] (SPARK-42895) ValueError when invoking any session operations on a stopped Spark session

[ https://issues.apache.org/jira/browse/SPARK-42895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42895: Assignee: Allison Wang > ValueError when invoking any session operations on a stopped Spa

[jira] [Resolved] (SPARK-42895) ValueError when invoking any session operations on a stopped Spark session

[ https://issues.apache.org/jira/browse/SPARK-42895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42895. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40536 [https://gi

[jira] [Commented] (SPARK-42840) Assign a name to the error class _LEGACY_ERROR_TEMP_2004

[ https://issues.apache.org/jira/browse/SPARK-42840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706257#comment-17706257 ] Max Gekk commented on SPARK-42840: -- [~leibniz.hu] Please, submit a PR. We will assign t

[jira] [Commented] (SPARK-42841) Assign a name to the error class _LEGACY_ERROR_TEMP_2005

[ https://issues.apache.org/jira/browse/SPARK-42841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706255#comment-17706255 ] Max Gekk commented on SPARK-42841: -- [~crazyqiqi] Sure, go ahead. > Assign a name to th

[jira] [Commented] (SPARK-42841) Assign a name to the error class _LEGACY_ERROR_TEMP_2005

[ https://issues.apache.org/jira/browse/SPARK-42841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706251#comment-17706251 ] LiuZhiqi commented on SPARK-42841: -- Hi Max, I am trying to complete this task, I will c

[jira] [Assigned] (SPARK-42956) avro functions

[ https://issues.apache.org/jira/browse/SPARK-42956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42956: Assignee: Yang Jie > avro functions > -- > > Key: SPARK-42956

[jira] [Resolved] (SPARK-42956) avro functions

[ https://issues.apache.org/jira/browse/SPARK-42956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42956. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40584 [https://gi

[jira] [Comment Edited] (SPARK-41359) Use `PhysicalDataType` instead of DataType in UnsafeRow

[ https://issues.apache.org/jira/browse/SPARK-41359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706248#comment-17706248 ] Yang Jie edited comment on SPARK-41359 at 3/29/23 7:14 AM: --- [~

[jira] [Commented] (SPARK-41359) Use `PhysicalDataType` instead of DataType in UnsafeRow

[ https://issues.apache.org/jira/browse/SPARK-41359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706248#comment-17706248 ] Yang Jie commented on SPARK-41359: -- [~cloud_fan] this ticket is created by me, but not