[jira] [Commented] (SPARK-41527) Implement DataFrame.observe

2022-12-16 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648395#comment-17648395 ] jiaan.geng commented on SPARK-41527: I'm working on. > Implement DataFrame.observe

[jira] [Resolved] (SPARK-41544) Upgrade `versions-maven-plugin` to 2.14.1

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41544. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39089 [https://gi

[jira] [Assigned] (SPARK-41544) Upgrade `versions-maven-plugin` to 2.14.1

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41544: Assignee: Yang Jie > Upgrade `versions-maven-plugin` to 2.14.1 >

[jira] [Assigned] (SPARK-41453) Implement DataFrame.subtract

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41453: Assignee: jiaan.geng > Implement DataFrame.subtract > > >

[jira] [Resolved] (SPARK-41453) Implement DataFrame.subtract

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41453. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39073 [https://gi

[jira] [Resolved] (SPARK-41541) Fix wrong child call in SQLShuffleWriteMetricsReporter.decRecordsWritten()

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41541. -- Fix Version/s: 3.3.2 3.1.4 3.2.3 3.4.0

[jira] [Commented] (SPARK-41334) move SortField from relations.proto to expressions.proto

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648571#comment-17648571 ] Apache Spark commented on SPARK-41334: -- User 'zhengruifeng' has created a pull requ

[jira] [Assigned] (SPARK-41334) move SortField from relations.proto to expressions.proto

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41334: Assignee: Apache Spark > move SortField from relations.proto to expressions.proto > -

[jira] [Assigned] (SPARK-41334) move SortField from relations.proto to expressions.proto

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41334: Assignee: (was: Apache Spark) > move SortField from relations.proto to expressions.pr

[jira] [Commented] (SPARK-41527) Implement DataFrame.observe

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648604#comment-17648604 ] Apache Spark commented on SPARK-41527: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-41527) Implement DataFrame.observe

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41527: Assignee: (was: Apache Spark) > Implement DataFrame.observe > ---

[jira] [Assigned] (SPARK-41527) Implement DataFrame.observe

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41527: Assignee: Apache Spark > Implement DataFrame.observe > --- > >

[jira] [Commented] (SPARK-41527) Implement DataFrame.observe

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648606#comment-17648606 ] Apache Spark commented on SPARK-41527: -- User 'beliefer' has created a pull request

[jira] [Commented] (SPARK-41546) pyspark_types_to_proto_types should supports StructType.

2022-12-16 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648615#comment-17648615 ] jiaan.geng commented on SPARK-41546: I'm working on. > pyspark_types_to_proto_types

[jira] [Created] (SPARK-41546) pyspark_types_to_proto_types should supports StructType.

2022-12-16 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-41546: -- Summary: pyspark_types_to_proto_types should supports StructType. Key: SPARK-41546 URL: https://issues.apache.org/jira/browse/SPARK-41546 Project: Spark Issue Ty

[jira] (SPARK-41527) Implement DataFrame.observe

2022-12-16 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41527 ] jiaan.geng deleted comment on SPARK-41527: was (Author: beliefer): I'm working on. > Implement DataFrame.observe > --- > > Key: SPARK-41527 >

[jira] [Commented] (SPARK-24497) ANSI SQL: Recursive query

2022-12-16 Thread gabrywu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648616#comment-17648616 ] gabrywu commented on SPARK-24497: - this is a useful feature, when will it be merged to m

[jira] [Created] (SPARK-41547) Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-41547: Summary: Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions Key: SPARK-41547 URL: https://issues.apache.org/jira/browse/SPARK-41547 Project: Spark

[jira] [Updated] (SPARK-41547) Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-41547: - Parent: SPARK-41285 Issue Type: Sub-task (was: Test) > Reenable ANSI mode in pyspark.sq

[jira] [Updated] (SPARK-41547) Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-41547: - Description: See https://issues.apache.org/jira/browse/SPARK-41548 We should fix the tests. > R

[jira] [Created] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-41548: Summary: Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions Key: SPARK-41548 URL: https://issues.apache.org/jira/browse/SPARK-41548 Project: Spark

[jira] [Assigned] (SPARK-41547) Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41547: Assignee: Xinrong Meng > Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_fun

[jira] [Updated] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-41548: - Description: There are failures in `test_connect_functions` with ANSI mode on (https://github.c

[jira] [Updated] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-41548: - Description: There are failures in {{test_connect_functions}} with ANSI mode on (https://github

[jira] [Assigned] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41548: Assignee: Apache Spark > Disable ANSI mode in pyspark.sql.tests.connect.test_connect_func

[jira] [Commented] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648618#comment-17648618 ] Apache Spark commented on SPARK-41548: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41548: Assignee: (was: Apache Spark) > Disable ANSI mode in pyspark.sql.tests.connect.test_c

[jira] [Resolved] (SPARK-41523) `protoc-jar-maven-plugin` should uniformly use `protoc-jar-maven-plugin.version` as the version

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41523. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39066 [https://gi

[jira] [Assigned] (SPARK-41523) `protoc-jar-maven-plugin` should uniformly use `protoc-jar-maven-plugin.version` as the version

2022-12-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41523: Assignee: Yang Jie > `protoc-jar-maven-plugin` should uniformly use > `protoc-jar-maven-

[jira] [Commented] (SPARK-41420) Protobuf serializer for ApplicationInfoWrapper

2022-12-16 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648667#comment-17648667 ] Sandeep Singh commented on SPARK-41420: --- I can work on this one > Protobuf serial

[jira] [Commented] (SPARK-41420) Protobuf serializer for ApplicationInfoWrapper

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648670#comment-17648670 ] Apache Spark commented on SPARK-41420: -- User 'techaddict' has created a pull reques

[jira] [Assigned] (SPARK-41420) Protobuf serializer for ApplicationInfoWrapper

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41420: Assignee: (was: Apache Spark) > Protobuf serializer for ApplicationInfoWrapper >

[jira] [Assigned] (SPARK-41420) Protobuf serializer for ApplicationInfoWrapper

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41420: Assignee: Apache Spark > Protobuf serializer for ApplicationInfoWrapper > ---

[jira] [Commented] (SPARK-41420) Protobuf serializer for ApplicationInfoWrapper

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648671#comment-17648671 ] Apache Spark commented on SPARK-41420: -- User 'techaddict' has created a pull reques

[jira] [Created] (SPARK-41549) Create a TIME data type

2022-12-16 Thread Patrick Grandjean (Jira)
Patrick Grandjean created SPARK-41549: - Summary: Create a TIME data type Key: SPARK-41549 URL: https://issues.apache.org/jira/browse/SPARK-41549 Project: Spark Issue Type: Wish

[jira] [Assigned] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-41548: - Assignee: Hyukjin Kwon > Disable ANSI mode in pyspark.sql.tests.connect.test_connect_fu

[jira] [Resolved] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-41548. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39092 [https://

[jira] [Updated] (SPARK-39006) Show a directional error message for PVC Dynamic Allocation Failure

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39006: -- Parent: SPARK-41515 Issue Type: Sub-task (was: Improvement) > Show a directional erro

[jira] [Created] (SPARK-41550) Dynamic Allocation on K8S GA

2022-12-16 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-41550: - Summary: Dynamic Allocation on K8S GA Key: SPARK-41550 URL: https://issues.apache.org/jira/browse/SPARK-41550 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-41550) Dynamic Allocation on K8S GA

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41550: -- Labels: releasenotes (was: ) > Dynamic Allocation on K8S GA > >

[jira] [Updated] (SPARK-38019) ExecutorMonitor.timedOutExecutors should be deterministic

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38019: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Bug) > ExecutorMonitor.timedOutExecuto

[jira] [Updated] (SPARK-38019) ExecutorMonitor.timedOutExecutors should be deterministic

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38019: -- Parent: (was: SPARK-35781) Issue Type: Bug (was: Sub-task) > ExecutorMonitor.time

[jira] [Resolved] (SPARK-41515) PVC-oriented executor pod allocation

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-41515. --- Fix Version/s: 3.4.0 Resolution: Fixed > PVC-oriented executor pod allocation > -

[jira] [Updated] (SPARK-39846) Enable spark.dynamicAllocation.shuffleTracking.enabled by default

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39846: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Enable spark.dynamicAllocation.shuffleTracking

[jira] [Updated] (SPARK-39450) Reuse PVCs by default

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39450: -- Parent: (was: SPARK-41515) Issue Type: Bug (was: Sub-task) > Reuse PVCs by defaul

[jira] [Updated] (SPARK-39450) Reuse PVCs by default

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39450: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Bug) > Reuse PVCs by default > ---

[jira] [Updated] (SPARK-40198) Enable spark.storage.decommission.(rdd|shuffle)Blocks.enabled by default

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40198: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Enable spark.storage.decommission.(rdd|shuffle

[jira] [Updated] (SPARK-39324) Log ExecutorDecommission as INFO level in TaskSchedulerImpl

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39324: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Log ExecutorDecommission as INFO level in Task

[jira] [Updated] (SPARK-40304) Add decomTestTag to K8s Integration Test

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40304: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Add decomTestTag to K8s Integration Test > ---

[jira] [Updated] (SPARK-41536) Remove `Dynamic Resource Allocation` from K8s Future Work

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41536: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Remove `Dynamic Resource Allocation` from K8s

[jira] [Updated] (SPARK-41540) Add `DISK_USED` executor roll policy

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41540: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Add `DISK_USED` executor roll policy > ---

[jira] [Updated] (SPARK-41543) Add `TOTAL_SHUFFLE_WRITE` executor roll policy

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41543: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Add `TOTAL_SHUFFLE_WRITE` executor roll policy

[jira] [Created] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning

2022-12-16 Thread Steve Loughran (Jira)
Steve Loughran created SPARK-41551: -- Summary: Improve/complete PathOutputCommitProtocol support for dynamic partitioning Key: SPARK-41551 URL: https://issues.apache.org/jira/browse/SPARK-41551 Projec

[jira] [Updated] (SPARK-39987) Support PEAK_JVM_(ON|OFF)HEAP_MEMORY executor rolling policy

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39987: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Support PEAK_JVM_(ON|OF

[jira] [Updated] (SPARK-38023) ExecutorMonitor.onExecutorRemoved should handle ExecutorDecommission as finished

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38023: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Bug) > ExecutorMonitor.onExecutorRemov

[jira] [Updated] (SPARK-33870) Enable spark.storage.replication.proactive by default

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33870: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Enable spark.storage.re

[jira] [Commented] (SPARK-41162) Anti-join must not be pushed below aggregation with ambiguous predicates

2022-12-16 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648751#comment-17648751 ] Shardul Mahadik commented on SPARK-41162: - [~cloud_fan] Can you help take a look

[jira] [Updated] (SPARK-38417) Remove `Experimental` from `RDD.cleanShuffleDependencies` API

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38417: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Task) > Remove `Experimental` from `RD

[jira] [Comment Edited] (SPARK-41162) Anti-join must not be pushed below aggregation with ambiguous predicates

2022-12-16 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648751#comment-17648751 ] Shardul Mahadik edited comment on SPARK-41162 at 12/16/22 6:27 PM: ---

[jira] [Updated] (SPARK-38418) Add PySpark cleanShuffleDependencies API

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38418: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Add PySpark cleanShuffl

[jira] [Updated] (SPARK-39322) Remove `Experimental` from `spark.dynamicAllocation.shuffleTracking.enabled`

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39322: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Documentation) > Remove `Experimental`

[jira] [Created] (SPARK-41552) Upgrade kubernetes-client to 6.3.1

2022-12-16 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-41552: - Summary: Upgrade kubernetes-client to 6.3.1 Key: SPARK-41552 URL: https://issues.apache.org/jira/browse/SPARK-41552 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-41552) Upgrade kubernetes-client to 6.3.1

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41552: Assignee: Apache Spark > Upgrade kubernetes-client to 6.3.1 > ---

[jira] [Assigned] (SPARK-41552) Upgrade kubernetes-client to 6.3.1

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41552: Assignee: (was: Apache Spark) > Upgrade kubernetes-client to 6.3.1 >

[jira] [Commented] (SPARK-41552) Upgrade kubernetes-client to 6.3.1

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648752#comment-17648752 ] Apache Spark commented on SPARK-41552: -- User 'dongjoon-hyun' has created a pull req

[jira] [Updated] (SPARK-40379) Propagate decommission executor loss reason during onDisconnect in K8s

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40379: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Propagate decommission

[jira] [Commented] (SPARK-40379) Propagate decommission executor loss reason during onDisconnect in K8s

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648756#comment-17648756 ] Dongjoon Hyun commented on SPARK-40379: --- Hi, [~holden]. We want to go GA with `Dyn

[jira] [Updated] (SPARK-40979) Keep removed executor info in decommission state

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40979: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Keep removed executor i

[jira] [Updated] (SPARK-40596) Populate ExecutorDecommission with more informative messages

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40596: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Populate ExecutorDecomm

[jira] [Commented] (SPARK-40979) Keep removed executor info in decommission state

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648757#comment-17648757 ] Dongjoon Hyun commented on SPARK-40979: --- I collected this as a subtask of SPARK-41

[jira] [Commented] (SPARK-40596) Populate ExecutorDecommission with more informative messages

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648758#comment-17648758 ] Dongjoon Hyun commented on SPARK-40596: --- I collected this as a subtask of SPARK-41

[jira] [Updated] (SPARK-40636) Fix wrong remained shuffles log in BlockManagerDecommissioner

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40636: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Bug) > Fix wrong remained shuffles log

[jira] [Updated] (SPARK-40481) Ignore stage fetch failure caused by decommissioned executor

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40481: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Ignore stage fetch fail

[jira] [Commented] (SPARK-40481) Ignore stage fetch failure caused by decommissioned executor

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648759#comment-17648759 ] Dongjoon Hyun commented on SPARK-40481: --- I collected this as a subtask of SPARK-41

[jira] [Commented] (SPARK-40636) Fix wrong remained shuffles log in BlockManagerDecommissioner

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648760#comment-17648760 ] Dongjoon Hyun commented on SPARK-40636: --- I collected this as a subtask of SPARK-41

[jira] [Updated] (SPARK-40269) Randomize the orders of peer in BlockManagerDecommissioner

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40269: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Randomize the orders of

[jira] [Commented] (SPARK-40269) Randomize the orders of peer in BlockManagerDecommissioner

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648761#comment-17648761 ] Dongjoon Hyun commented on SPARK-40269: --- I collected this as a subtask of SPARK-41

[jira] [Commented] (SPARK-40060) Add numberDecommissioningExecutors metric

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648762#comment-17648762 ] Dongjoon Hyun commented on SPARK-40060: --- I collected this as a subtask of SPARK-41

[jira] [Updated] (SPARK-40060) Add numberDecommissioningExecutors metric

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40060: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Add numberDecommissioni

[jira] [Updated] (SPARK-38062) FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38062: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > FallbackStorage shouldn

[jira] [Assigned] (SPARK-41552) Upgrade kubernetes-client to 6.3.1

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-41552: - Assignee: Dongjoon Hyun > Upgrade kubernetes-client to 6.3.1 >

[jira] [Resolved] (SPARK-41365) Stages UI page fails to load for proxy in some yarn versions

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-41365. --- Fix Version/s: 3.3.2 3.4.0 Target Version/s: (was: 3.3.1)

[jira] [Assigned] (SPARK-41365) Stages UI page fails to load for proxy in some yarn versions

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-41365: - Assignee: miracle > Stages UI page fails to load for proxy in some yarn versions > ---

[jira] [Assigned] (SPARK-41365) Stages UI page fails to load for proxy in some yarn versions

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-41365: - Assignee: (was: miracle) > Stages UI page fails to load for proxy in some yarn vers

[jira] [Assigned] (SPARK-41365) Stages UI page fails to load for proxy in some yarn versions

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-41365: - Assignee: Mars (was: miracle) > Stages UI page fails to load for proxy in some yarn ve

[jira] [Assigned] (SPARK-41365) Stages UI page fails to load for proxy in some yarn versions

2022-12-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-41365: - Assignee: miracle > Stages UI page fails to load for proxy in some yarn versions > ---

[jira] [Commented] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications

2022-12-16 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648782#comment-17648782 ] Sandeep Singh commented on SPARK-41053: --- [~Gengliang.Wang] I'm willing to take som

[jira] [Assigned] (SPARK-41421) Protobuf serializer for ApplicationEnvironmentInfoWrapper

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41421: Assignee: (was: Apache Spark) > Protobuf serializer for ApplicationEnvironmentInfoWra

[jira] [Commented] (SPARK-41421) Protobuf serializer for ApplicationEnvironmentInfoWrapper

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648784#comment-17648784 ] Apache Spark commented on SPARK-41421: -- User 'techaddict' has created a pull reques

[jira] [Assigned] (SPARK-41421) Protobuf serializer for ApplicationEnvironmentInfoWrapper

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41421: Assignee: Apache Spark > Protobuf serializer for ApplicationEnvironmentInfoWrapper >

[jira] [Commented] (SPARK-41421) Protobuf serializer for ApplicationEnvironmentInfoWrapper

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648785#comment-17648785 ] Apache Spark commented on SPARK-41421: -- User 'techaddict' has created a pull reques

[jira] [Commented] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications

2022-12-16 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648786#comment-17648786 ] Gengliang Wang commented on SPARK-41053: [~techaddict] Thanks! Feel free to take

[jira] [Comment Edited] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications

2022-12-16 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648786#comment-17648786 ] Gengliang Wang edited comment on SPARK-41053 at 12/16/22 8:30 PM:

[jira] [Created] (SPARK-41553) Change num_files to repartition

2022-12-16 Thread Jira
Bjørn Jørgensen created SPARK-41553: --- Summary: Change num_files to repartition Key: SPARK-41553 URL: https://issues.apache.org/jira/browse/SPARK-41553 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-41049) Nondeterministic expressions have unstable values if they are children of CodegenFallback expressions

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648792#comment-17648792 ] Apache Spark commented on SPARK-41049: -- User 'NarekDW' has created a pull request f

[jira] [Assigned] (SPARK-41049) Nondeterministic expressions have unstable values if they are children of CodegenFallback expressions

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41049: Assignee: (was: Apache Spark) > Nondeterministic expressions have unstable values if

[jira] [Assigned] (SPARK-41049) Nondeterministic expressions have unstable values if they are children of CodegenFallback expressions

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41049: Assignee: Apache Spark > Nondeterministic expressions have unstable values if they are ch

[jira] [Assigned] (SPARK-41553) Change num_files to repartition

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41553: Assignee: (was: Apache Spark) > Change num_files to repartition > ---

[jira] [Commented] (SPARK-41553) Change num_files to repartition

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648793#comment-17648793 ] Apache Spark commented on SPARK-41553: -- User 'bjornjorgensen' has created a pull re

[jira] [Assigned] (SPARK-41553) Change num_files to repartition

2022-12-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41553: Assignee: Apache Spark > Change num_files to repartition > --

  1   2   >