[jira] [Assigned] (SPARK-44361) Use PartitionEvaluator API in MapInBatchExec

[ https://issues.apache.org/jira/browse/SPARK-44361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44361: --- Assignee: Vinod KC > Use PartitionEvaluator API in MapInBatchExec >

[jira] [Resolved] (SPARK-44361) Use PartitionEvaluator API in MapInBatchExec

[ https://issues.apache.org/jira/browse/SPARK-44361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44361. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42024

[jira] [Resolved] (SPARK-44411) Use PartitionEvaluator API in ArrowEvalPythonExec, BatchEvalPythonExec

[ https://issues.apache.org/jira/browse/SPARK-44411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44411. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 41998

[jira] [Assigned] (SPARK-44411) Use PartitionEvaluator API in ArrowEvalPythonExec, BatchEvalPythonExec

[ https://issues.apache.org/jira/browse/SPARK-44411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44411: --- Assignee: Vinod KC > Use PartitionEvaluator API in ArrowEvalPythonExec,

[jira] [Assigned] (SPARK-44375) Use PartitionEvaluator API in DebugExec

[ https://issues.apache.org/jira/browse/SPARK-44375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44375: --- Assignee: Jia Fan > Use PartitionEvaluator API in DebugExec >

[jira] [Resolved] (SPARK-44375) Use PartitionEvaluator API in DebugExec

[ https://issues.apache.org/jira/browse/SPARK-44375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44375. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 41949

[jira] [Resolved] (SPARK-44474) Reenable "Test observe response" at SparkConnectServiceSuite

[ https://issues.apache.org/jira/browse/SPARK-44474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44474. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-44474) Reenable "Test observe response" at SparkConnectServiceSuite

[ https://issues.apache.org/jira/browse/SPARK-44474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44474: Assignee: Hyukjin Kwon > Reenable "Test observe response" at SparkConnectServiceSuite >

[jira] [Commented] (SPARK-44264) DeepSpeed Distrobutor

[ https://issues.apache.org/jira/browse/SPARK-44264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744410#comment-17744410 ] Hudson commented on SPARK-44264: User 'mathewjacob1002' has created a pull request for this issue:

[jira] [Updated] (SPARK-44264) DeepSpeed Distrobutor

[ https://issues.apache.org/jira/browse/SPARK-44264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-44264: - Attachment: Trying to Run Deepspeed Funcs.html > DeepSpeed Distrobutor

[jira] [Assigned] (SPARK-44401) Arrow Python UDF Use Guide

[ https://issues.apache.org/jira/browse/SPARK-44401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44401: Assignee: Xinrong Meng > Arrow Python UDF Use Guide > -- > >

[jira] [Resolved] (SPARK-44401) Arrow Python UDF Use Guide

[ https://issues.apache.org/jira/browse/SPARK-44401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44401. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Resolved] (SPARK-44464) Fix applyInPandasWithStatePythonRunner to output rows that have Null as first column value

[ https://issues.apache.org/jira/browse/SPARK-44464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-44464. -- Fix Version/s: 3.5.0 Assignee: Siying Dong Resolution: Fixed Issue resolved

[jira] [Updated] (SPARK-44448) Wrong results for dense_rank() <= k from InferWindowGroupLimit and DenseRankLimitIterator

[ https://issues.apache.org/jira/browse/SPARK-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-8: Fix Version/s: (was: 4.0.0) > Wrong results for dense_rank() <= k from InferWindowGroupLimit

[jira] [Resolved] (SPARK-44448) Wrong results for dense_rank() <= k from InferWindowGroupLimit and DenseRankLimitIterator

[ https://issues.apache.org/jira/browse/SPARK-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-8. - Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-44448) Wrong results for dense_rank() <= k from InferWindowGroupLimit and DenseRankLimitIterator

[ https://issues.apache.org/jira/browse/SPARK-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-8: --- Assignee: Jack Chen > Wrong results for dense_rank() <= k from InferWindowGroupLimit and

[jira] [Resolved] (SPARK-44324) Move CaseInsensitiveMap to sql/api

[ https://issues.apache.org/jira/browse/SPARK-44324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44324. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41882

[jira] [Created] (SPARK-44480) Add option for thread pool to perform maintenance for RocksDB/HDFS State Store Providers

Eric Marnadi created SPARK-44480: Summary: Add option for thread pool to perform maintenance for RocksDB/HDFS State Store Providers Key: SPARK-44480 URL: https://issues.apache.org/jira/browse/SPARK-44480

[jira] [Resolved] (SPARK-43755) Spark Connect - decouple query execution from RPC handler

[ https://issues.apache.org/jira/browse/SPARK-43755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43755. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-43755) Spark Connect - decouple query execution from RPC handler

[ https://issues.apache.org/jira/browse/SPARK-43755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43755: Assignee: Juliusz Sompolski > Spark Connect - decouple query execution from RPC handler

[jira] [Assigned] (SPARK-44476) JobArtifactSet is populated with all artifacts if it is not associated with an artifact

[ https://issues.apache.org/jira/browse/SPARK-44476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44476: Assignee: Venkata Sai Akhil Gudesa > JobArtifactSet is populated with all artifacts if

[jira] [Resolved] (SPARK-44476) JobArtifactSet is populated with all artifacts if it is not associated with an artifact

[ https://issues.apache.org/jira/browse/SPARK-44476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44476. -- Resolution: Fixed Issue resolved by pull request 42062

[jira] [Resolved] (SPARK-42944) Support Python foreachBatch() in streaming spark connect

[ https://issues.apache.org/jira/browse/SPARK-42944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42944. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-42944) Support Python foreachBatch() in streaming spark connect

[ https://issues.apache.org/jira/browse/SPARK-42944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42944: Assignee: Raghu Angadi > Support Python foreachBatch() in streaming spark connect >

[jira] [Commented] (SPARK-36392) pandas fixed width file support

[ https://issues.apache.org/jira/browse/SPARK-36392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744375#comment-17744375 ] Haejoon Lee commented on SPARK-36392: - Not update yet here. [~gsdionis] , are you still interested

[jira] [Commented] (SPARK-44464) Fix applyInPandasWithStatePythonRunner to output rows that have Null as first column value

[ https://issues.apache.org/jira/browse/SPARK-44464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744369#comment-17744369 ] Siying Dong commented on SPARK-44464: - PR created: [https://github.com/apache/spark/pull/42046] CC

[jira] [Updated] (SPARK-44479) Support Python UDTFs with empty schema

[ https://issues.apache.org/jira/browse/SPARK-44479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-44479: -- Description: Support UDTFs with empty schema, for example: {code:python} >>> class TestUDTF:

[jira] [Updated] (SPARK-44479) Support Python UDTFs with empty schema

[ https://issues.apache.org/jira/browse/SPARK-44479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-44479: -- Description: Support UDTFs with empty schema, for example: {code:python} >>> class TestUDTF:

[jira] [Created] (SPARK-44479) Support Python UDTFs with empty schema

Takuya Ueshin created SPARK-44479: - Summary: Support Python UDTFs with empty schema Key: SPARK-44479 URL: https://issues.apache.org/jira/browse/SPARK-44479 Project: Spark Issue Type:

[jira] [Commented] (SPARK-40296) Error Class for DISTINCT function not found

[ https://issues.apache.org/jira/browse/SPARK-40296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744346#comment-17744346 ] Ritika Maheshwari commented on SPARK-40296: --- Isn't dropDuplicates taking care of applying

[jira] [Created] (SPARK-44478) Executor decommission causes stage failure

Dale Huettenmoser created SPARK-44478: - Summary: Executor decommission causes stage failure Key: SPARK-44478 URL: https://issues.apache.org/jira/browse/SPARK-44478 Project: Spark Issue

[jira] [Commented] (SPARK-44477) CheckAnalysis uses error subclass as an error class

[ https://issues.apache.org/jira/browse/SPARK-44477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744314#comment-17744314 ] Bruce Robbins commented on SPARK-44477: --- PR here: https://github.com/apache/spark/pull/42064 >

[jira] [Created] (SPARK-44477) CheckAnalysis uses error subclass as an error class

Bruce Robbins created SPARK-44477: - Summary: CheckAnalysis uses error subclass as an error class Key: SPARK-44477 URL: https://issues.apache.org/jira/browse/SPARK-44477 Project: Spark Issue

[jira] [Commented] (SPARK-36392) pandas fixed width file support

[ https://issues.apache.org/jira/browse/SPARK-36392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744297#comment-17744297 ] John Ayoub commented on SPARK-36392: [~itholic] Hello, any update on this ticket? > pandas fixed

[jira] [Resolved] (SPARK-44465) Upgrade zstd-jni to 1.5.5-5

[ https://issues.apache.org/jira/browse/SPARK-44465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44465. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42047

[jira] [Assigned] (SPARK-44465) Upgrade zstd-jni to 1.5.5-5

[ https://issues.apache.org/jira/browse/SPARK-44465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44465: - Assignee: BingKun Pan > Upgrade zstd-jni to 1.5.5-5 > --- > >

[jira] [Created] (SPARK-44476) JobArtifactSet is populated with all artifacts if it is not associated with an artifact

Venkata Sai Akhil Gudesa created SPARK-44476: Summary: JobArtifactSet is populated with all artifacts if it is not associated with an artifact Key: SPARK-44476 URL:

[jira] [Updated] (SPARK-44448) Wrong results for dense_rank() <= k from InferWindowGroupLimit and DenseRankLimitIterator

[ https://issues.apache.org/jira/browse/SPARK-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Chen updated SPARK-8: -- Affects Version/s: 3.5.0 (was: 3.4.0) > Wrong results for dense_rank() <=

[jira] [Created] (SPARK-44475) Relocate DataType and Parser to sql/api

Rui Wang created SPARK-44475: Summary: Relocate DataType and Parser to sql/api Key: SPARK-44475 URL: https://issues.apache.org/jira/browse/SPARK-44475 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-44467) Setting master version to 4.0.0-SNAPSHOT

[ https://issues.apache.org/jira/browse/SPARK-44467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-44467: - Fix Version/s: 4.0.0 (was: 3.5.0) > Setting master version to 4.0.0-SNAPSHOT >

[jira] [Resolved] (SPARK-44467) Setting master version to 4.0.0-SNAPSHOT

[ https://issues.apache.org/jira/browse/SPARK-44467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-44467. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42048

[jira] [Assigned] (SPARK-44467) Setting master version to 4.0.0-SNAPSHOT

[ https://issues.apache.org/jira/browse/SPARK-44467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-44467: Assignee: Yang Jie > Setting master version to 4.0.0-SNAPSHOT >

[jira] [Commented] (SPARK-42972) ExecutorAllocationManager cannot allocate new instances when all executors down.

[ https://issues.apache.org/jira/browse/SPARK-42972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744133#comment-17744133 ] lvkaihua commented on SPARK-42972: -- I also encountered this issue and tested that the modification was

[jira] [Commented] (SPARK-44396) Add direct Arrow deserialization

[ https://issues.apache.org/jira/browse/SPARK-44396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744127#comment-17744127 ] ASF GitHub Bot commented on SPARK-44396: User 'hvanhovell' has created a pull request for this

[jira] [Updated] (SPARK-44472) change the external catalog thread safety way

[ https://issues.apache.org/jira/browse/SPARK-44472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Izek Greenfield updated SPARK-44472: Attachment: add_hive_concurrent_connections.diff > change the external catalog thread

[jira] [Updated] (SPARK-44474) Reenable "Test observe response" at SparkConnectServiceSuite

[ https://issues.apache.org/jira/browse/SPARK-44474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44474: - Summary: Reenable "Test observe response" at SparkConnectServiceSuite (was: Reenable Test

[jira] [Updated] (SPARK-44474) Reenable Test observe response at SparkConnectServiceSuite

[ https://issues.apache.org/jira/browse/SPARK-44474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44474: - Affects Version/s: 3.5.0 (was: 4.0.0) > Reenable Test observe

[jira] [Updated] (SPARK-44474) Reenable Test observe response at SparkConnectServiceSuite

[ https://issues.apache.org/jira/browse/SPARK-44474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44474: - Priority: Blocker (was: Major) > Reenable Test observe response at SparkConnectServiceSuite >

[jira] [Created] (SPARK-44474) Reenable Test observe response at SparkConnectServiceSuite

Hyukjin Kwon created SPARK-44474: Summary: Reenable Test observe response at SparkConnectServiceSuite Key: SPARK-44474 URL: https://issues.apache.org/jira/browse/SPARK-44474 Project: Spark

[jira] [Resolved] (SPARK-44468) Add daily test GA task for branch3.5

[ https://issues.apache.org/jira/browse/SPARK-44468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44468. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42050

[jira] [Assigned] (SPARK-44468) Add daily test GA task for branch3.5

[ https://issues.apache.org/jira/browse/SPARK-44468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44468: Assignee: BingKun Pan > Add daily test GA task for branch3.5 >

[jira] [Updated] (SPARK-44471) Change branches in build_and_test.yml for master branch

[ https://issues.apache.org/jira/browse/SPARK-44471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44471: - Summary: Change branches in build_and_test.yml for master branch (was: Add Github action test

[jira] [Resolved] (SPARK-44471) Add Github action test job for branch-3.5

[ https://issues.apache.org/jira/browse/SPARK-44471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44471. -- Resolution: Fixed Issue resolved by pull request 42057

[jira] [Commented] (SPARK-42972) ExecutorAllocationManager cannot allocate new instances when all executors down.

[ https://issues.apache.org/jira/browse/SPARK-42972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744092#comment-17744092 ] liang yu commented on SPARK-42972: -- [~tdas]  I created a PR

[jira] [Created] (SPARK-44473) Overwriting the same partition of a partitioned table multiple times with empty data yields non-idempotent results

chris Yu created SPARK-44473: Summary: Overwriting the same partition of a partitioned table multiple times with empty data yields non-idempotent results Key: SPARK-44473 URL:

[jira] [Resolved] (SPARK-44451) Make built document downloadable

[ https://issues.apache.org/jira/browse/SPARK-44451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-44451. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42028

[jira] [Assigned] (SPARK-44451) Make built document downloadable

[ https://issues.apache.org/jira/browse/SPARK-44451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-44451: - Assignee: Ruifeng Zheng > Make built document downloadable >

[jira] [Created] (SPARK-44472) change the external catalog thread safety way

Izek Greenfield created SPARK-44472: --- Summary: change the external catalog thread safety way Key: SPARK-44472 URL: https://issues.apache.org/jira/browse/SPARK-44472 Project: Spark Issue

[jira] [Created] (SPARK-44471) Add Github action test job for branch-3.5

Yuanjian Li created SPARK-44471: --- Summary: Add Github action test job for branch-3.5 Key: SPARK-44471 URL: https://issues.apache.org/jira/browse/SPARK-44471 Project: Spark Issue Type: Task

[jira] [Resolved] (SPARK-43967) Support Python UDTFs with empty return values

[ https://issues.apache.org/jira/browse/SPARK-43967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43967. -- Fix Version/s: 3.5.0 Assignee: Allison Wang Resolution: Fixed Fixed in