[jira] [Updated] (SPARK-40320) When the Executor plugin fails to initialize, the Executor shows active but does not accept tasks forever, just like being hung

[ https://issues.apache.org/jira/browse/SPARK-40320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-40320: Fix Version/s: 3.4.0 > When the Executor plugin fails to initialize, the Execu

[jira] [Resolved] (SPARK-40320) When the Executor plugin fails to initialize, the Executor shows active but does not accept tasks forever, just like being hung

[ https://issues.apache.org/jira/browse/SPARK-40320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40320. - Resolution: Fixed > When the Executor plugin fails to initialize, the Execu

[jira] [Updated] (SPARK-40902) Quick submission of drivers in tests to mesos scheduler results in dropping drivers

[ https://issues.apache.org/jira/browse/SPARK-40902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-40902: Priority: Minor (was: Major) > Quick submission of drivers in tests to me

[jira] [Commented] (SPARK-40902) Quick submission of drivers in tests to mesos scheduler results in dropping drivers

[ https://issues.apache.org/jira/browse/SPARK-40902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17623270#comment-17623270 ] Mridul Muralidharan commented on SPARK-40902: - +CC [~dongjoon], [~yum

[jira] [Created] (SPARK-40902) Quick submission of drivers in tests to mesos scheduler results in dropping drivers

Mridul Muralidharan created SPARK-40902: --- Summary: Quick submission of drivers in tests to mesos scheduler results in dropping drivers Key: SPARK-40902 URL: https://issues.apache.org/jira/browse/SPARK-40902

Re: [VOTE] Release Spark 3.3.1 (RC4)

GitHub Action: > https://github.com/apache/spark/actions?query=branch%3Abranch-3.3 > Apple Silicon Jenkins Farm: > https://apache-spark.s3.fr-par.scw.cloud/BRANCH-3.3.html > > Dongjoon. > > > On Fri, Oct 21, 2022 at 8:48 AM Mridul Muralidharan > wrote: > >> Hi, &

Re: [VOTE] Release Spark 3.3.1 (RC4)

Hi, I saw a couple of test failures I have not observed before: a) FsHistoryProviderSuite - "SPARK-33146: don't let one bad rolling log folder prevent loading other applications" b) MesosClusterSchedulerSuite - "accept/decline offers with driver constraints" I ended up 'ignore''ing them to ma

Re: Welcome Yikun Jiang as a Spark committer

Congratulations ! Regards, Mridul On Sat, Oct 8, 2022 at 12:19 AM Yuming Wang wrote: > Congratulations Yikun! > > On Sat, Oct 8, 2022 at 12:40 PM Hyukjin Kwon wrote: > >> Hi all, >> >> The Spark PMC recently added Yikun Jiang as a committer on the project. >> Yikun is the major contributor of

Re: [VOTE] Release Spark 3.3.1 (RC2)

+1 from me, with a few comments. I saw the following failures, are these known issues/flakey tests ? * PersistenceEngineSuite.ZooKeeperPersistenceEngine Looks like a port conflict issue from a quick look into logs (conflict with starting admin port at 8080) - is this expected behavior for the tes

[jira] [Assigned] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40096: --- Assignee: Wan Kun > Finalize shuffle merge slow due to connection creat

[jira] [Resolved] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40096. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-38888) Add `RocksDBProvider` similar to `LevelDBProvider`

[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-3. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-38888) Add `RocksDBProvider` similar to `LevelDBProvider`

[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-3: --- Assignee: Yang Jie > Add `RocksDBProvider` similar to `LevelDBProvi

[jira] [Resolved] (SPARK-40186) mergedShuffleCleaner should have been shutdown before db closed

[ https://issues.apache.org/jira/browse/SPARK-40186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40186. - Target Version/s: 3.4.0 Assignee: Yang Jie Resolution

[jira] [Resolved] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40094. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40094: --- Assignee: wangshengjie > Send TaskEnd event when task failed w

[jira] [Assigned] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

[ https://issues.apache.org/jira/browse/SPARK-38909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38909: --- Assignee: Yang Jie > Encapsulate LevelDB used

[jira] [Resolved] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

[ https://issues.apache.org/jira/browse/SPARK-38909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38909. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Kai-Hsun Chen (was: Kai-Hsun Chen) > Improve LaunchTask proc

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Kai-Hsun Chen (was: Mridul Muralidharan) > Improve LaunchT

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Mridul Muralidharan > Improve LaunchTask process to avoid St

[jira] [Resolved] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-39955. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

Re: How to set platform-level defaults for array-like configs?

Hi, Wenchen, would be great if you could chime in with your thoughts - given the feedback you originally had on the PR. It would be great to hear feedback from others on this, particularly folks managing spark deployments - how this is mitigated/avoided in your case, any other pain points with c

Re: Welcoming three new PMC members

Congratulations ! Great to have you join the PMC !! Regards, Mridul On Tue, Aug 9, 2022 at 11:57 AM vaquar khan wrote: > Congratulations > > On Tue, Aug 9, 2022, 11:40 AM Xiao Li wrote: > >> Hi all, >> >> The Spark PMC recently voted to add three new PMC members. Join me in >> welcoming them t

Re: Welcome Xinrong Meng as a Spark committer

Congratulations Xinrong ! Regards, Mridul On Tue, Aug 9, 2022 at 3:13 AM Hyukjin Kwon wrote: > Hi all, > > The Spark PMC recently added Xinrong Meng as a committer on the project. > Xinrong is the major contributor of PySpark especially Pandas API on Spark. > She has guided a lot of new contrib

[jira] [Resolved] (SPARK-40004) Redundant `LevelDB.get` in `RemoteBlockPushResolver`

[ https://issues.apache.org/jira/browse/SPARK-40004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40004. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-40004) Redundant `LevelDB.get` in `RemoteBlockPushResolver`

[ https://issues.apache.org/jira/browse/SPARK-40004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40004: --- Assignee: Yang Jie > Redundant `LevelDB.get` in `RemoteBlockPushResol

Re: [VOTE] Apache Bahir Flink 1.1.0 (RC2)

Boto wrote: > Hi Mridul, > > Thanks for checking.. > Do you have installed on your computer a docker environment? > This is needed for testcontainers that we use on testing.. > > Le me know if is that > > On 2022/07/29 15:11:21 Mridul Muralidharan wrote: > > The t

Re: [VOTE] Apache Bahir Flink 1.1.0 (RC2)

The tests failed for me with [1] I also saw this at the top of build [2], but dont think it was related. Any pointers ? Thanks, Mridul [1] [INFO] Running org.apache.flink.streaming.connectors.flume.FlumeSinkTest SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaultin

[jira] [Commented] (SPARK-39696) Uncaught exception in thread executor-heartbeater java.util.ConcurrentModificationException: mutation occurred during iteration

[ https://issues.apache.org/jira/browse/SPARK-39696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17571169#comment-17571169 ] Mridul Muralidharan commented on SPARK-39696: - Can you give more detail

[jira] [Resolved] (SPARK-33236) Enable Push-based shuffle service to store state in NM level DB for work preserving restart

[ https://issues.apache.org/jira/browse/SPARK-33236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33236. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-33236) Enable Push-based shuffle service to store state in NM level DB for work preserving restart

[ https://issues.apache.org/jira/browse/SPARK-33236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33236: --- Assignee: Ye Zhou > Enable Push-based shuffle service to store state in

Re: [VOTE] Release Spark 3.2.2 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with "-Pyarn -Pmesos -Pkubernetes" As always, the test "SPARK-33084: Add jar support Ivy URI in SQL" in sql.SQLQuerySuite fails in my env; but other than that, the rest looks good. Regards, Mridul On Tue, Jul 12, 2022

[jira] [Resolved] (SPARK-39647) Block push fails with java.lang.IllegalArgumentException: Active local dirs list has not been updated by any executor registration even when the NodeManager hasn't been

[ https://issues.apache.org/jira/browse/SPARK-39647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-39647. - Fix Version/s: 3.3.1 3.4.0 Resolution: Fixed Issue

[jira] [Assigned] (SPARK-39647) Block push fails with java.lang.IllegalArgumentException: Active local dirs list has not been updated by any executor registration even when the NodeManager hasn't been

[ https://issues.apache.org/jira/browse/SPARK-39647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39647: --- Assignee: Chandni Singh > Block push fails w

Re: Apache Spark 3.2.2 Release?

+1 Thanks for driving this Dongjoon ! Regards, Mridul On Thu, Jul 7, 2022 at 12:36 AM Gengliang Wang wrote: > +1. > Thank you, Dongjoon. > > On Wed, Jul 6, 2022 at 10:21 PM Wenchen Fan wrote: > >> +1 >> >> On Thu, Jul 7, 2022 at 10:41 AM Xinrong Meng >> wrote: >> >>> +1 >>> >>> Thanks! >>> >

[jira] [Resolved] (SPARK-30835) Add support for YARN decommissioning & pre-emption

[ https://issues.apache.org/jira/browse/SPARK-30835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-30835. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-30835) Add support for YARN decommissioning & pre-emption

[ https://issues.apache.org/jira/browse/SPARK-30835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-30835: --- Assignee: Abhishek Dixit > Add support for YARN decommissioning &a

Re: [VOTE] Release Spark 3.3.0 (RC6)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes The test "SPARK-33084: Add jar support Ivy URI in SQL" in sql.SQLQuerySuite fails; but other than that, rest looks good. Regards, Mridul On Mon, Jun 13, 2022 at 4:25 PM Tom Graves wr

[jira] [Commented] (SPARK-38987) Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true

[ https://issues.apache.org/jira/browse/SPARK-38987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17550681#comment-17550681 ] Mridul Muralidharan commented on SPARK-38987: - Manually marking as reso

[jira] [Resolved] (SPARK-38987) Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true

[ https://issues.apache.org/jira/browse/SPARK-38987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38987. - Fix Version/s: 3.4.0 Assignee: Aravind Patnam Resolution: Fixed

Re: Spark Push-Based Shuffle causing multiple stage failures

+CC zhouye...@gmail.com On Mon, May 23, 2022 at 7:11 AM Han Altae-Tran wrote: > Hi, > > First of all, I am very thankful for all of the amazing work that goes > into this project! It has opened up so many doors for me! I am a long > time Spark user, and was very excited to start working with th

Re: [VOTE] Release Spark 3.3.0 (RC1)

I will also try to get a PR out to fix the first test failure that Sean reported. I will have a PR ready by EOD. Regards, Mridul On Fri, May 6, 2022 at 10:31 AM Gengliang Wang wrote: > Hi Maxim, > > Thanks for the work! > There is a bug fix from Bruce merged on branch-3.3 right after the RC1 i

[jira] [Assigned] (SPARK-38973) When push-based shuffle is enabled, a stage may not complete when retried

[ https://issues.apache.org/jira/browse/SPARK-38973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38973: --- Assignee: Chandni Singh > When push-based shuffle is enabled, a stage

[jira] [Resolved] (SPARK-38973) When push-based shuffle is enabled, a stage may not complete when retried

[ https://issues.apache.org/jira/browse/SPARK-38973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38973. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-38856) Fix a rejectedExecutionException error when push-based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-38856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38856. - Target Version/s: 3.4.0 Assignee: weixiuli Resolution

[jira] [Resolved] (SPARK-38555) Avoid contention and get or create clientPools quickly in the TransportClientFactory

[ https://issues.apache.org/jira/browse/SPARK-38555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38555. - Assignee: weixiuli Resolution: Fixed > Avoid contention and get

[jira] [Updated] (SPARK-38555) Avoid contention and get or create clientPools quickly in the TransportClientFactory

[ https://issues.apache.org/jira/browse/SPARK-38555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-38555: Fix Version/s: 3.4.0 > Avoid contention and get or create clientPools quickly

[jira] [Commented] (SPARK-38388) Repartition + Stage retries could lead to incorrect data

[ https://issues.apache.org/jira/browse/SPARK-38388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507257#comment-17507257 ] Mridul Muralidharan commented on SPARK-38388: - Agree with [~jiangxb

[jira] [Resolved] (SPARK-38428) Check the FetchShuffleBlocks message only once to improve iteration in external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-38428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38428. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-38309) SHS has incorrect percentiles for shuffle read bytes and shuffle total blocks metrics

[ https://issues.apache.org/jira/browse/SPARK-38309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38309. - Fix Version/s: 3.3.0 3.2.2 3.1.3

[jira] [Assigned] (SPARK-38309) SHS has incorrect percentiles for shuffle read bytes and shuffle total blocks metrics

[ https://issues.apache.org/jira/browse/SPARK-38309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38309: --- Assignee: Rob Reeves > SHS has incorrect percentiles for shuffle r

Re: Apache Spark 3.3 Release

Agree with Sean, code freeze by mid March sounds good. Regards, Mridul On Thu, Mar 3, 2022 at 12:47 PM Sean Owen wrote: > I think it's fine to pursue the existing plan - code freeze in two weeks > and try to close off key remaining issues. Final release pending on how > those go, and testing, b

[jira] [Resolved] (SPARK-38344) Avoid to submit task when there are no requests to push up in push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-38344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38344. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-38344) Avoid to submit task when there are no requests to push up in push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-38344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38344: --- Assignee: weixiuli > Avoid to submit task when there are no requests

[jira] [Resolved] (SPARK-37688) ExecutorMonitor should ignore SparkListenerBlockUpdated event if executor was not active

[ https://issues.apache.org/jira/browse/SPARK-37688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37688. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37688) ExecutorMonitor should ignore SparkListenerBlockUpdated event if executor was not active

[ https://issues.apache.org/jira/browse/SPARK-37688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37688: --- Assignee: hujiahua > ExecutorMonitor should ign

Re: [VOTE] Spark 3.1.3 RC4

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Feb 16, 2022 at 8:32 AM Thomas graves wrote: > +1 > > Tom > > On Mon, Feb 14, 2022 at 2:55 PM Holden Karau wrote: > > > > Please vote on releasing the followi

[jira] [Resolved] (SPARK-34826) Adaptive fetch of shuffle mergers for Push based shuffle

[ https://issues.apache.org/jira/browse/SPARK-34826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34826. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-34826) Adaptive fetch of shuffle mergers for Push based shuffle

[ https://issues.apache.org/jira/browse/SPARK-34826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34826: --- Assignee: Venkata krishnan Sowrirajan > Adaptive fetch of shuffle merg

Re: [VOTE] Spark 3.1.3 RC3

ds, Mridul [1] "The tag to be voted on is v3.2.1-rc1" - the commit hash and git url are correct. On Wed, Feb 2, 2022 at 9:30 AM Mridul Muralidharan wrote: > > Thanks Tom ! > I missed [1] (or probably forgot) the 3.1 part of the discussion given it > centered around 3.2 ..

Re: [VOTE] Spark 3.1.3 RC3

nce lines back at beginning of > December (Dec 6) when we were talking about release 3.2.1. > > Tom > > On Wed, Feb 2, 2022 at 2:07 AM Mridul Muralidharan > wrote: > > > > Hi Holden, > > > > Not that I am against releasing 3.1.3 (given the fixes tha

Re: [VOTE] Spark 3.1.3 RC3

Hi Holden, Not that I am against releasing 3.1.3 (given the fixes that have already gone in), but did we discuss releasing it ? I might have missed the thread ... Regards, Mridul On Tue, Feb 1, 2022 at 7:12 PM Holden Karau wrote: > Please vote on releasing the following candidate as Apache S

[jira] [Assigned] (SPARK-37793) Invalid LocalMergedBlockData cause task hang

[ https://issues.apache.org/jira/browse/SPARK-37793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37793: --- Assignee: Chandni Singh > Invalid LocalMergedBlockData cause task h

[jira] [Resolved] (SPARK-37793) Invalid LocalMergedBlockData cause task hang

[ https://issues.apache.org/jira/browse/SPARK-37793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37793. - Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed

[jira] [Resolved] (SPARK-37675) Push-based merge finalization bugs in the RemoteBlockPushResolver

[ https://issues.apache.org/jira/browse/SPARK-37675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37675. - Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue

[jira] [Assigned] (SPARK-37675) Push-based merge finalization bugs in the RemoteBlockPushResolver

[ https://issues.apache.org/jira/browse/SPARK-37675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37675: --- Assignee: Chandni Singh > Push-based merge finalization bugs in

Re: [VOTE] Release Spark 3.2.1 (RC2)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Fri, Jan 21, 2022 at 9:01 PM Sean Owen wrote: > +1 with same result as last time. > > On Thu, Jan 20, 2022 at 9:59 PM huaxin gao wrote: > >> Please vote on releasin

[jira] [Assigned] (SPARK-37712) Spark request yarn cluster metrics slow and cause unnecessary delay

[ https://issues.apache.org/jira/browse/SPARK-37712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37712: --- Assignee: angerszhu > Spark request yarn cluster metrics slow and ca

[jira] [Resolved] (SPARK-37712) Spark request yarn cluster metrics slow and cause unnecessary delay

[ https://issues.apache.org/jira/browse/SPARK-37712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37712. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

+1 (binding) This should be a great improvement ! Regards, Mridul On Wed, Jan 12, 2022 at 4:04 AM Kent Yao wrote: > +1 (non-binding) > > Thomas Graves 于2022年1月12日周三 11:52写道： > >> +1 (binding). >> >> One minor note since I haven't had time to look at the implementation >> details is please make

[jira] [Assigned] (SPARK-37847) PushBlockStreamCallback should check isTooLate first to avoid NPE

[ https://issues.apache.org/jira/browse/SPARK-37847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37847: --- Assignee: Cheng Pan > PushBlockStreamCallback should check isTooLate fi

[jira] [Resolved] (SPARK-37847) PushBlockStreamCallback should check isTooLate first to avoid NPE

[ https://issues.apache.org/jira/browse/SPARK-37847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37847. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37710) Add detailed log message for java.io.IOException occurring on Kryo flow

[ https://issues.apache.org/jira/browse/SPARK-37710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37710: --- Assignee: Eren Avsarogullari > Add detailed log message

[jira] [Resolved] (SPARK-37710) Add detailed log message for java.io.IOException occurring on Kryo flow

[ https://issues.apache.org/jira/browse/SPARK-37710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37710. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-33701) Adaptive shuffle merge finalization for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-33701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33701. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-33701) Adaptive shuffle merge finalization for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-33701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33701: --- Assignee: Venkata krishnan Sowrirajan > Adaptive shuffle me

Re: Time for Spark 3.2.1?

+1 for maintenance release, and also +1 for doing this in Jan ! Thanks, Mridul On Tue, Dec 7, 2021 at 11:41 PM Gengliang Wang wrote: > +1 for new maintenance releases for all 3.x branches as well. > > On Wed, Dec 8, 2021 at 8:19 AM Hyukjin Kwon wrote: > >> SGTM! >> >> On Wed, 8 Dec 2021 at 09:

Re: [FYI] Build and run tests on Java 17 for Apache Spark 3.3

Nice job ! There are some nice API's which should be interesting to explore with JDK 17 :-) Regards. Mridul On Fri, Nov 12, 2021 at 7:08 PM Yuming Wang wrote: > Cool, thank you Dongjoon. > > On Sat, Nov 13, 2021 at 4:09 AM shane knapp ☠ wrote: > >> woot! nice work everyone! :) >> >> On Fri,

[jira] [Resolved] (SPARK-37023) Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry

[ https://issues.apache.org/jira/browse/SPARK-37023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37023. - Fix Version/s: 3.3.0 3.2.1 Resolution: Fixed Issue

[jira] [Assigned] (SPARK-37023) Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry

[ https://issues.apache.org/jira/browse/SPARK-37023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37023: --- Assignee: Minchu Yang > Avoid fetching merge status w

Re: Update Spark 3.3 release window?

+1 to EOL 2.x Mid march sounds like a good placeholder for 3.3. Regards, Mridul On Wed, Oct 27, 2021 at 10:38 PM Sean Owen wrote: > Seems fine to me - as good a placeholder as anything. > Would that be about time to call 2.x end-of-life? > > On Wed, Oct 27, 2021 at 9:36 PM Hyukjin Kwon wrote:

Re: [ANNOUNCE] Apache Spark 3.2.0

Congratulations everyone ! And thanks Gengliang for sheparding the release out :-) Regards, Mridul On Tue, Oct 19, 2021 at 9:25 AM Yuming Wang wrote: > Congrats and thanks! > > On Tue, Oct 19, 2021 at 10:17 PM Gengliang Wang wrote: > >> Hi all, >> >> Apache Spark 3.2.0 is the third release of

Re: [ANNOUNCE] Apache Spark 3.2.0

Congratulations everyone ! And thanks Gengliang for sheparding the release out :-) Regards, Mridul On Tue, Oct 19, 2021 at 9:25 AM Yuming Wang wrote: > Congrats and thanks! > > On Tue, Oct 19, 2021 at 10:17 PM Gengliang Wang wrote: > >> Hi all, >> >> Apache Spark 3.2.0 is the third release of

[jira] [Resolved] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36834. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36834: --- Assignee: Thejdeep Gudivada > Namespace log lines in External Shuf

[jira] [Assigned] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36834: --- Assignee: (was: Thejdeep Gudivada) > Namespace log lines in Exter

[jira] [Assigned] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36834: --- Assignee: Thejdeep Gudivada > Namespace log lines in External Shuf

[jira] [Assigned] (SPARK-36798) When SparkContext is stopped, metrics system should be flushed after listeners have finished processing

[ https://issues.apache.org/jira/browse/SPARK-36798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36798: --- Assignee: Harsh Panchal > When SparkContext is stopped, metrics sys

[jira] [Resolved] (SPARK-36798) When SparkContext is stopped, metrics system should be flushed after listeners have finished processing

[ https://issues.apache.org/jira/browse/SPARK-36798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36798. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phadoop-2.7 -Pyarn -Pmesos -Pkubernetes. Regards, Mridul On Wed, Oct 6, 2021 at 12:55 PM Michael Heuer wrote: > +1 (non-binding) > >michael > > > On Oct 6, 2021, at 11:49 AM, Gengliang Wang wrote: > > Start

[jira] [Commented] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425016#comment-17425016 ] Mridul Muralidharan commented on SPARK-36892: - Sounds good [~Gengliang.

[jira] [Commented] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424097#comment-17424097 ] Mridul Muralidharan commented on SPARK-36926: - +CC [~gengl

[jira] [Comment Edited] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424052#comment-17424052 ] Mridul Muralidharan edited comment on SPARK-36892 at 10/4/21, 5:2

[jira] [Commented] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424052#comment-17424052 ] Mridul Muralidharan commented on SPARK-36892: - [~Gengliang.Wang] Thanks

Re: [VOTE] Release Spark 3.2.0 (RC6)

Yi Wu helped identify an issue which causes correctness (duplication) and hangs - waiting for validation to complete before submitting a patch. Regards, Mridul On Wed, Sep 29, 2021 at 11:34 AM Holden Karau wrote: > PySpark smoke tests pass, I'

[jira] [Created] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

Mridul Muralidharan created SPARK-36892: --- Summary: Disable batch fetch for a shuffle when push based shuffle is enabled Key: SPARK-36892 URL: https://issues.apache.org/jira/browse/SPARK-36892

Re: [VOTE] Release Spark 3.2.0 (RC3)

;> On Tue, Sep 21, 2021 at 2:05 PM Chao Sun wrote: >>> >>>> Mridul, is the LZ4 failure about Parquet? I think Parquet currently >>>> uses Hadoop compression codec while Hadoop 2.7 still depends on native lib >>>> for the LZ4. Maybe we should run the t

[jira] [Resolved] (SPARK-36615) SparkContext should register shutdown hook earlier

[ https://issues.apache.org/jira/browse/SPARK-36615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36615. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-36615) SparkContext should register shutdown hook earlier

[ https://issues.apache.org/jira/browse/SPARK-36615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36615: --- Assignee: angerszhu > SparkContext should register shutdown hook earl

<    1   2   3   4   5   6   7   8   9   10   >