Re: [VOTE] Release Spark 3.4.3 (RC2)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pkubernetes Regards, Mridul On Sun, Apr 14, 2024 at 11:31 PM Dongjoon Hyun wrote: > I'll start with my +1. > > - Checked checksum and signature > - Checked Scala/Java/R/Python/SQL Document's

Re: Versioning of Spark Operator

I am trying to understand if we can simply align with Spark's version for this ? Makes the release and jira management much more simpler for developers and intuitive for users. Regards, Mridul On Tue, Apr 9, 2024 at 10:09 AM Dongjoon Hyun wrote: > Hi, Liang-Chi. > > Thank you for leading

Re: Apache Spark 3.4.3 (?)

Hi Dongjoon, Thanks for volunteering ! I would suggest to wait for SPARK-47318 to be merged as well for 3.4 Regards, Mridul On Sat, Apr 6, 2024 at 6:49 PM Dongjoon Hyun wrote: > Hi, All. > > Apache Spark 3.4.2 tag was created on Nov 24th and `branch-3.4` has 85 > commits including important

Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

--- >>> *From:* Denny Lee >>> *Sent:* Monday, April 1, 2024 10:06:14 AM >>> *To:* Hussein Awala >>> *Cc:* Chao Sun ; Hyukjin Kwon ; >>> Mridul Muralidharan ; dev >>> *Subject:* Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

Can you point me to the SPIP’s discussion thread please ? I was not able to find it, but I was on vacation, and so might have missed this … Regards, Mridul On Sun, Mar 31, 2024 at 9:08 PM Haejoon Lee wrote: > +1 > > On Mon, Apr 1, 2024 at 10:15 AM Hyukjin Kwon wrote: > >> Hi all, >> >> I'd

Re: [Spark-Core] Improving Reliability of spark when Executors OOM

are some > open discussion threads on the doc you shared. > > @Mridul Muralidharan In what state are your efforts > along this? Is it something that your team is actively pursuing/ building > or are mostly planning right now? Asking so that we can align efforts on > this. > &

Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark

I am supportive of the proposal - this is a step in the right direction ! Additional metadata (explicit and inferred) for log records, and exposing them for indexing is extremely useful. The specifics of the API still need some work IMO and does not need to be this disruptive, but I consider

Re: [DISCUSS] SPIP: Structured Spark Logging

Hi Gengling, Thanks for sharing this ! I added a few queries to the proposal doc, and we can continue discussing there, but overall I am in favor of this. Regards, Mridul On Fri, Mar 1, 2024 at 1:35 AM Gengliang Wang wrote: > Hi All, > > I propose to enhance our logging system by

Re: [Spark-Core] Improving Reliability of spark when Executors OOM

Hi, We are internally exploring adding support for dynamically changing the resource profile of a stage based on runtime characteristics. This includes failures due to OOM and the like, slowness due to excessive GC, resource wastage due to excessive overprovisioning, etc. Essentially handles

Re: [VOTE] Release Spark 3.3.4 (RC1)

I am seeing a bunch of python related (43) failures in the sql module (for example [1]) ... I am currently on Python 3.11.6, java 8. Not sure if ubuntu modified anything from under me, thoughts ? I am currently testing this against an older branch to make sure it is not an issue with my desktop.

Re: Apache Spark 3.3.4 EOL Release?

+1 Regards, Mridul On Mon, Dec 4, 2023 at 11:40 AM L. C. Hsieh wrote: > +1 > > Thanks Dongjoon! > > On Mon, Dec 4, 2023 at 9:26 AM Yang Jie wrote: > > > > +1 for a 3.3.4 EOL Release. Thanks Dongjoon. > > > > Jie Yang > > > > On 2023/12/04 15:08:25 Tom Graves wrote: > > > +1 for a 3.3.4 EOL

Re: [VOTE] Release Spark 3.4.2 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Nov 29, 2023 at 5:08 AM Yang Jie wrote: > +1(non-binding) > > Jie Yang > > On 2023/11/29 02:08:04 Kent Yao wrote: > > +1(non-binding) > > > > Kent Yao >

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 Regards, Mridul On Fri, Nov 24, 2023 at 8:21 AM Kent Yao wrote: > Hi Spark Dev, > > Following the discussion [1], I'd like to start the vote for the SPIP [2]. > > The SPIP aims to improve the test coverage and develop experience for > Spark UI-related javascript codes. > > This thread will

Re: [DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

This should be a very good addition ! Regards, Mridul On Tue, Nov 21, 2023 at 7:46 PM Dongjoon Hyun wrote: > Thank you for proposing a new UI test framework for Apache Spark 4.0. > > It looks very useful. > > Thanks, > Dongjoon. > > > On Tue, Nov 21, 2023 at 1:51 AM Kent Yao wrote: > >> Hi

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 Regards, Mridul On Tue, Nov 14, 2023 at 12:45 PM Holden Karau wrote: > +1 > > On Tue, Nov 14, 2023 at 10:21 AM DB Tsai wrote: > >> +1 >> >> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >> >> On Nov 14, 2023, at 10:14 AM, Vakaris Baškirov < >> vakaris.bashki...@gmail.com>

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations ! Looking forward to more exciting contributions :-) Regards, Mridul On Tue, Oct 3, 2023 at 2:51 AM Hussein Awala wrote: > Congrats to all of you! > > On Tue 3 Oct 2023 at 08:15, Rui Wang wrote: > >> Congratulations! Well deserved! >> >> -Rui >> >> >> On Mon, Oct 2, 2023 at

Re: Migrating the Junit framework used in Apache Spark 4.0 from 4.x to 5.x

+1 for moving to a newer version. Thanks for driving this Jie Yang ! Regards, Mridul On Mon, Sep 25, 2023 at 10:15 AM 杨杰 wrote: > Hi all, > > In SPARK-44170 (apache/spark#43074 [1]), I’m trying to migrate the Junit > test framework used in Spark 4.0 from Junit4 to Junit5. > > > Although this

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Sat, Sep 9, 2023 at 10:02 AM Yuanjian Li wrote: > Please vote on releasing the following candidate(RC5) as Apache Spark > version 3.5.0. > > The vote is open

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Aug 30, 2023 at 6:10 AM yangjie01 wrote: > Hi, Sean > > > > I have performed testing with Java 17 and Scala 2.13 using maven (`mvn > clean install` and

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Fri, Aug 11, 2023 at 2:00 AM Cheng Pan wrote: > +1 (non-binding) > > Passed integration test with Apache Kyuubi. > > Thanks for driving this release. > >

Re: [ANNOUNCE] Apache Spark 3.4.1 released

Thanks Dongjoon ! Regards, Mridul On Fri, Jun 23, 2023 at 6:58 PM Dongjoon Hyun wrote: > We are happy to announce the availability of Apache Spark 3.4.1! > > Spark 3.4.1 is a maintenance release containing stability fixes. This > release is based on the branch-3.4 maintenance branch of Spark.

Re: [VOTE][RESULT] Release Spark 3.4.1 (RC1)

A late +1 from me too … forgot to send this yesterday :-) Regards, Mridul On Fri, Jun 23, 2023 at 3:20 AM Dongjoon Hyun wrote: > The vote passes with 15 +1s (10 binding +1s). > Thanks to all who helped with the release! > > (* = binding) > +1: > - Jia Fan > - Dongjoon Hyun * > - Liang-Chi

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

I agree with Holden, we should have some understanding of what we are targeting for 4.0, given it is a major ver bump - and work from there on the release date. Regards, Mridul On Mon, Jun 12, 2023 at 8:53 PM Jia Fan wrote: > By the way, like Holden said, what's big feature for 4.0.0? I think

Re: Apache Spark 3.4.1 Release?

+1, thanks Dongjoon ! Regards, Mridul On Thu, Jun 8, 2023 at 7:16 PM Jia Fan wrote: > +1 > > > > > Jia Fan > > > > 2023年6月9日 08:00，Yuming Wang 写道： > > +1. > > On Fri, Jun 9, 2023 at 7:14 AM Chao Sun wrote: > >> +1 too >> >> On Thu, Jun 8, 2023 at 2:34 PM kazuyuki

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Mon, Apr 10, 2023 at 10:34 AM huaxin gao wrote: > +1 > > On Mon, Apr 10, 2023 at 8:17 AM Chao Sun wrote: > >> +1 (non-binding) >> >> On Mon, Apr 10, 2023

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Sat, Apr 8, 2023 at 12:13 PM L. C. Hsieh wrote: > +1 > > Thanks Xinrong. > > On Sat, Apr 8, 2023 at 8:23 AM yangjie01 wrote: > > > > +1 > > > > > > > >

Re: Apache Spark 3.2.4 EOL Release?

+1 Sounds good to me. Thanks, Mridul On Tue, Apr 4, 2023 at 1:39 PM huaxin gao wrote: > +1 > > On Tue, Apr 4, 2023 at 11:17 AM Chao Sun wrote: > >> +1 >> >> On Tue, Apr 4, 2023 at 11:12 AM Holden Karau >> wrote: >> >>> +1 >>> >>> On Tue, Apr 4, 2023 at 11:04 AM L. C. Hsieh wrote: >>>

Re: Slack for PySpark users

Thanks for flagging the concern Dongjoon, I was not aware of the discussion - but I can understand the concern. Would be great if you or Matei could update the thread on the result of deliberations, once it reaches a logical consensus: before we set up official policy around it. Regards, Mridul

Re: Ammonite as REPL for Spark Connect

ng started > with connect, and/or doing debugging. > > On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan > wrote: > >> >> What is unclear to me is why we are introducing this integration, how >> users will leverage it. >> >> * Are we replacing spark-shell

Re: Ammonite as REPL for Spark Connect

che Spark. > > On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan > wrote: > >> >> Will this be maintained externally or included into Apache Spark ? >> >> Regards , >> Mridul >> >> >> >> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell >

Re: Ammonite as REPL for Spark Connect

Will this be maintained externally or included into Apache Spark ? Regards , Mridul On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell wrote: > Hi All, > > For Spark Connect Scala Client we are working on making the REPL > experience a bit nicer .

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

Other than the tag issue, the sigs/artifacts/build/etc worked for me. So the next RC candidate looks promising ! Regards, Mridul On Thu, Mar 9, 2023 at 5:07 PM Xinrong Meng wrote: > Thank you Hyukjin! :) > > I would prefer to cut v3.4.0-rc4 now if there are no objections. > > On Fri, Mar 10,

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ... On Wed, Feb 22, 2023 at 2:07 AM Mridul Muralidharan wrote: > > Thanks Xinrong ! > The signature verifications are fine now ... will continue with testing > the release. > > > Regards, > Mridul > &

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

Thanks Xinrong ! The signature verifications are fine now ... will continue with testing the release. Regards, Mridul On Wed, Feb 22, 2023 at 1:27 AM Xinrong Meng wrote: > Hi Mridul, > > Would you please try that again? It should work now. > > On Wed, Feb 22, 2023 at

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

Hi Xinrong, Was it signed with the same key as present in KEYS [1] ? I am seeing errors with gpg when validating. For example: $ gpg --verify pyspark-3.4.0.tar.gz.asc gpg: assuming signed data in 'pyspark-3.4.0.tar.gz' gpg: Signature made Tue 21 Feb 2023 05:56:05 AM CST gpg:

Re: [VOTE] Release Spark 3.3.2 (RC1)

ct in > > https://repository.apache.org/content/repositories/orgapachespark-1433/org/apache/spark/spark-mllib-local_2.13/3.3.2/ > . > Did I miss something? > > Liang-Chi > > On Sat, Feb 11, 2023 at 10:08 AM Mridul Muralidharan > wrote: > > > > > > Hi, >

Re: [VOTE] Release Spark 3.3.2 (RC1)

Hi, The following file is missing in the staging repository - there is a corresponding asc sig file, without the artifact. * org/apache/spark/spark-mllib-local_2.13/3.3.2/spark-mllib-local_2.13-3.3.2-test-sources.jar Can we have this fixed please ? Rest of the signatures, digests, etc check out

Re: Time for Spark 3.4.0 release?

+1, Thanks ! Regards, Mridul On Wed, Jan 4, 2023 at 2:20 AM Gengliang Wang wrote: > +1, thanks for driving the release! > > > Gengliang > > On Tue, Jan 3, 2023 at 10:55 PM Dongjoon Hyun > wrote: > >> +1 >> >> Thank you! >> >> Dongjoon >> >> On Tue, Jan 3, 2023 at 9:44 PM Rui Wang wrote: >>

Re: [VOTE][SPIP] Asynchronous Offset Management in Structured Streaming

+1 Regards, Mridul On Wed, Nov 30, 2022 at 8:55 PM Xingbo Jiang wrote: > +1 > > On Wed, Nov 30, 2022 at 5:59 PM Jungtaek Lim > wrote: > >> Starting with +1 from me. >> >> On Thu, Dec 1, 2022 at 10:54 AM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Hi all, >>> >>> I'd like to

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

goal of this project. If that >> happens eventually, that would be a side-effect. Someone may have concerns >> that we have two different projects aiming for similar thing, but I'd >> rather see both projects having competition. If anyone willing to improve >> continuous m

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

Hi Jungtaek, Given the goal of the SPIP is reducing latency for stateless apps, and should reasonably fit continuous mode design goals, it feels odd to not support it fin the proposal. I know you have raised concerns about continuous mode in past as well in dev@ list, and we are further

Re: [VOTE][RESULT] Release Spark 3.2.3, RC1

eh (*) > - Huaxin Gao (*) > - Kazuyuki Tanimura > - Mridul Muralidharan (*) > - Yuming Wang > - Chris Nauroth > - Yang Jie > - Wenche Fan (*) > - Ruifeng Zheng > - Chao Sun > > +0: None > > -1: None > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1 Would be great to see history server performance improvements and lower resource utilization at driver ! Regards, Mridul On Wed, Nov 16, 2022 at 2:38 AM Kent Yao wrote: > +1, non-binding > > Gengliang Wang 于2022年11月16日周三 16:36写道： > > > > Hi all, > > > > I’d like to start a vote for SPIP:

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Tue, Nov 15, 2022 at 1:00 PM kazuyuki tanimura wrote: > +1 (non-binding) > > Thank you Chao > > Kazu > > >  | Kazuyuki Tanimura | ktanim...@apple.com |

Re: [VOTE] Release Spark 3.3.1 (RC4)

ion: > https://github.com/apache/spark/actions?query=branch%3Abranch-3.3 > Apple Silicon Jenkins Farm: > https://apache-spark.s3.fr-par.scw.cloud/BRANCH-3.3.html > > Dongjoon. > > > On Fri, Oct 21, 2022 at 8:48 AM Mridul Muralidharan > wrote: > >> Hi, >> &g

Re: [VOTE] Release Spark 3.3.1 (RC4)

Hi, I saw a couple of test failures I have not observed before: a) FsHistoryProviderSuite - "SPARK-33146: don't let one bad rolling log folder prevent loading other applications" b) MesosClusterSchedulerSuite - "accept/decline offers with driver constraints" I ended up 'ignore''ing them to

Re: Welcome Yikun Jiang as a Spark committer

Congratulations ! Regards, Mridul On Sat, Oct 8, 2022 at 12:19 AM Yuming Wang wrote: > Congratulations Yikun! > > On Sat, Oct 8, 2022 at 12:40 PM Hyukjin Kwon wrote: > >> Hi all, >> >> The Spark PMC recently added Yikun Jiang as a committer on the project. >> Yikun is the major contributor of

Re: [VOTE] Release Spark 3.3.1 (RC2)

+1 from me, with a few comments. I saw the following failures, are these known issues/flakey tests ? * PersistenceEngineSuite.ZooKeeperPersistenceEngine Looks like a port conflict issue from a quick look into logs (conflict with starting admin port at 8080) - is this expected behavior for the

Re: How to set platform-level defaults for array-like configs?

Hi, Wenchen, would be great if you could chime in with your thoughts - given the feedback you originally had on the PR. It would be great to hear feedback from others on this, particularly folks managing spark deployments - how this is mitigated/avoided in your case, any other pain points with

Re: Welcoming three new PMC members

Congratulations ! Great to have you join the PMC !! Regards, Mridul On Tue, Aug 9, 2022 at 11:57 AM vaquar khan wrote: > Congratulations > > On Tue, Aug 9, 2022, 11:40 AM Xiao Li wrote: > >> Hi all, >> >> The Spark PMC recently voted to add three new PMC members. Join me in >> welcoming them

Re: Welcome Xinrong Meng as a Spark committer

Congratulations Xinrong ! Regards, Mridul On Tue, Aug 9, 2022 at 3:13 AM Hyukjin Kwon wrote: > Hi all, > > The Spark PMC recently added Xinrong Meng as a committer on the project. > Xinrong is the major contributor of PySpark especially Pandas API on Spark. > She has guided a lot of new

Re: [VOTE] Release Spark 3.2.2 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with "-Pyarn -Pmesos -Pkubernetes" As always, the test "SPARK-33084: Add jar support Ivy URI in SQL" in sql.SQLQuerySuite fails in my env; but other than that, the rest looks good. Regards, Mridul On Tue, Jul 12,

Re: Apache Spark 3.2.2 Release?

+1 Thanks for driving this Dongjoon ! Regards, Mridul On Thu, Jul 7, 2022 at 12:36 AM Gengliang Wang wrote: > +1. > Thank you, Dongjoon. > > On Wed, Jul 6, 2022 at 10:21 PM Wenchen Fan wrote: > >> +1 >> >> On Thu, Jul 7, 2022 at 10:41 AM Xinrong Meng >> wrote: >> >>> +1 >>> >>> Thanks! >>>

Re: [VOTE] Release Spark 3.3.0 (RC6)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes The test "SPARK-33084: Add jar support Ivy URI in SQL" in sql.SQLQuerySuite fails; but other than that, rest looks good. Regards, Mridul On Mon, Jun 13, 2022 at 4:25 PM Tom Graves

Re: [VOTE] Release Spark 3.3.0 (RC1)

I will also try to get a PR out to fix the first test failure that Sean reported. I will have a PR ready by EOD. Regards, Mridul On Fri, May 6, 2022 at 10:31 AM Gengliang Wang wrote: > Hi Maxim, > > Thanks for the work! > There is a bug fix from Bruce merged on branch-3.3 right after the RC1

Re: Apache Spark 3.3 Release

Agree with Sean, code freeze by mid March sounds good. Regards, Mridul On Thu, Mar 3, 2022 at 12:47 PM Sean Owen wrote: > I think it's fine to pursue the existing plan - code freeze in two weeks > and try to close off key remaining issues. Final release pending on how > those go, and testing,

Re: [VOTE] Spark 3.1.3 RC4

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Feb 16, 2022 at 8:32 AM Thomas graves wrote: > +1 > > Tom > > On Mon, Feb 14, 2022 at 2:55 PM Holden Karau wrote: > > > > Please vote on releasing the

Re: [VOTE] Spark 3.1.3 RC3

ds, Mridul [1] "The tag to be voted on is v3.2.1-rc1" - the commit hash and git url are correct. On Wed, Feb 2, 2022 at 9:30 AM Mridul Muralidharan wrote: > > Thanks Tom ! > I missed [1] (or probably forgot) the 3.1 part of the discussion given it > centered around 3.2 ..

Re: [VOTE] Spark 3.1.3 RC3

nce lines back at beginning of > December (Dec 6) when we were talking about release 3.2.1. > > Tom > > On Wed, Feb 2, 2022 at 2:07 AM Mridul Muralidharan > wrote: > > > > Hi Holden, > > > > Not that I am against releasing 3.1.3 (given the fixes tha

Re: [VOTE] Spark 3.1.3 RC3

Hi Holden, Not that I am against releasing 3.1.3 (given the fixes that have already gone in), but did we discuss releasing it ? I might have missed the thread ... Regards, Mridul On Tue, Feb 1, 2022 at 7:12 PM Holden Karau wrote: > Please vote on releasing the following candidate as Apache

Re: [VOTE] Release Spark 3.2.1 (RC2)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Fri, Jan 21, 2022 at 9:01 PM Sean Owen wrote: > +1 with same result as last time. > > On Thu, Jan 20, 2022 at 9:59 PM huaxin gao wrote: > >> Please vote on

Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

+1 (binding) This should be a great improvement ! Regards, Mridul On Wed, Jan 12, 2022 at 4:04 AM Kent Yao wrote: > +1 (non-binding) > > Thomas Graves 于2022年1月12日周三 11:52写道： > >> +1 (binding). >> >> One minor note since I haven't had time to look at the implementation >> details is please

Re: Time for Spark 3.2.1?

+1 for maintenance release, and also +1 for doing this in Jan ! Thanks, Mridul On Tue, Dec 7, 2021 at 11:41 PM Gengliang Wang wrote: > +1 for new maintenance releases for all 3.x branches as well. > > On Wed, Dec 8, 2021 at 8:19 AM Hyukjin Kwon wrote: > >> SGTM! >> >> On Wed, 8 Dec 2021 at

Re: [FYI] Build and run tests on Java 17 for Apache Spark 3.3

Nice job ! There are some nice API's which should be interesting to explore with JDK 17 :-) Regards. Mridul On Fri, Nov 12, 2021 at 7:08 PM Yuming Wang wrote: > Cool, thank you Dongjoon. > > On Sat, Nov 13, 2021 at 4:09 AM shane knapp ☠ wrote: > >> woot! nice work everyone! :) >> >> On Fri,

Re: Update Spark 3.3 release window?

+1 to EOL 2.x Mid march sounds like a good placeholder for 3.3. Regards, Mridul On Wed, Oct 27, 2021 at 10:38 PM Sean Owen wrote: > Seems fine to me - as good a placeholder as anything. > Would that be about time to call 2.x end-of-life? > > On Wed, Oct 27, 2021 at 9:36 PM Hyukjin Kwon wrote:

Re: [ANNOUNCE] Apache Spark 3.2.0

Congratulations everyone ! And thanks Gengliang for sheparding the release out :-) Regards, Mridul On Tue, Oct 19, 2021 at 9:25 AM Yuming Wang wrote: > Congrats and thanks! > > On Tue, Oct 19, 2021 at 10:17 PM Gengliang Wang wrote: > >> Hi all, >> >> Apache Spark 3.2.0 is the third release

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phadoop-2.7 -Pyarn -Pmesos -Pkubernetes. Regards, Mridul On Wed, Oct 6, 2021 at 12:55 PM Michael Heuer wrote: > +1 (non-binding) > >michael > > > On Oct 6, 2021, at 11:49 AM, Gengliang Wang wrote: > >

Re: [VOTE] Release Spark 3.2.0 (RC6)

Yi Wu helped identify an issue which causes correctness (duplication) and hangs - waiting for validation to complete before submitting a patch. Regards, Mridul On Wed, Sep 29, 2021 at 11:34 AM Holden Karau wrote: > PySpark smoke tests pass,

Re: [VOTE] Release Spark 3.2.0 (RC3)

ue, Sep 21, 2021 at 2:05 PM Chao Sun wrote: >>> >>>> Mridul, is the LZ4 failure about Parquet? I think Parquet currently >>>> uses Hadoop compression codec while Hadoop 2.7 still depends on native lib >>>> for the LZ4. Maybe we should run the test only f

Re: [VOTE] Release Spark 3.2.0 (RC3)

Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes, this worked fine. I found that including "-Phadoop-2.7" failed on lz4 tests ("native lz4 library not available"). Regards, Mridul On Tue, Sep 21, 2021 at 10:18 AM Gengliang Wang wrote:

Re: [VOTE] Release Spark 3.2.0 (RC2)

I have filed a blocker, SPARK-36705 which will need to be addressed. Regards, Mridul On Sun, Sep 5, 2021 at 8:47 AM Gengliang Wang wrote: > Hi all, > > the voting fails. > Liang-Chi reported a new block SPARK-36669 >

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi, Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Pmesos -Pkubernetes I am seeing test failures which are addressed by #33790 - this is in branch-3.2, but after the RC tag. After updating to the

Re: -1s on committed but not released code?

Hi Holden, In the past, I have seen discussions on the merged pr to thrash out the details. Usually it would be clear whether to revert and reformulate the change or concerns get addressed and possibly result in follow up work. This is usually helped by the fact that we typically are

Re: ASF board report draft for August

Hi Matei, 3.2 will also include support for pushed based shuffle (spip SPARK-30602). Regards, Mridul On Mon, Aug 9, 2021 at 9:26 PM Hyukjin Kwon wrote: > > Are you referring to what version of Koala project? 1.8.1? > > Yes, the latest version 1.8.1. > > 2021년 8월 10일 (화) 오전 11:07, Igor Costa

Re: [VOTE] Release Spark 3.0.3 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Pmesos -Pkubernetes Regards, Mridul PS: Might be related to some quirk of my local env - the first test run (after clean + package) usually fails for me (typically for hive tests) - with a

Re: Apache Spark 3.0.3 Release?

+1 Regards, Mridul On Tue, Jun 8, 2021 at 10:11 PM Hyukjin Kwon wrote: > Yeah, +1 > > 2021년 6월 9일 (수) 오후 12:06, Yi Wu 님이 작성: > >> Hi, All. >> >> Since Apache Spark 3.0.2 tag creation (Feb 16), >> new 119 patches (92 issues >> >>

Re: Resolves too old JIRAs as incomplete

+1, thanks Takeshi ! Regards, Mridul On Wed, May 19, 2021 at 8:48 PM Takeshi Yamamuro wrote: > Hi, dev, > > As you know, we have too many open JIRAs now: > # of open JIRAs=2698: JQL='project = SPARK AND status in (Open, "In > Progress", Reopened)' > > We've recently released v2.4.8(EOL), so

Re: [VOTE] Release Spark 2.4.8 (RC4)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested. Regards, Mridul On Sun, May 9, 2021 at 4:22 PM Liang-Chi Hsieh wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.4.8. > > The vote is open until May 14th at 9AM PST and passes if

Re: [VOTE] Release Spark 2.4.8 (RC1)

Do we have a fix for this in 3.x/master which can be backported without too much surrounding change ? Given we are expecting 2.4.7 to probably be the last release for 2.4, if we can fix it, that would be great. Regards, Mridul On Wed, Apr 7, 2021 at 9:31 PM Liang-Chi Hsieh wrote: > Thanks for

Re: [VOTE] SPIP: Support pandas API layer on PySpark

+1 Regards, Mridul On Sat, Mar 27, 2021 at 6:09 PM Xiao Li wrote: > +1 > > Xiao > > Takeshi Yamamuro 于2021年3月26日周五 下午4:14写道： > >> +1 (non-binding) >> >> On Sat, Mar 27, 2021 at 4:53 AM Liang-Chi Hsieh wrote: >> >>> +1 (non-binding) >>> >>> >>> rxin wrote >>> > +1. Would open up a huge

Re: Welcoming six new Apache Spark committers

Congratulations, looking forward to more exciting contributions ! Regards, Mridul On Fri, Mar 26, 2021 at 8:21 PM Dongjoon Hyun wrote: > > Congratulations! :) > > Bests, > Dongjoon. > > On Fri, Mar 26, 2021 at 5:55 PM angers zhu wrote: > >> Congratulations >> >> Prashant Sharma

Re: [ANNOUNCE] Announcing Apache Spark 3.1.1

Thanks Hyukjin and congratulations everyone on the release ! Regards, Mridul On Tue, Mar 2, 2021 at 8:54 PM Yuming Wang wrote: > Great work, Hyukjin! > > On Wed, Mar 3, 2021 at 9:50 AM Hyukjin Kwon wrote: > >> We are excited to announce Spark 3.1.1 today. >> >> Apache Spark 3.1.1 is the

Re: Apache Spark 3.2 Expectation

Nit: Java 17 -> should be available by Sept 2021 :-) Adoption would also depend on some of our nontrivial dependencies supporting it - it might be a stretch to get it in for Apache Spark 3.2 ? Features: Push based shuffle and disaggregated shuffle should also be in 3.2 Regards, Mridul On

Re: [VOTE] Release Spark 3.1.1 (RC3)

different > results between Spark 3.0 and Spark 3.1. We need a few more days to > understand whether these changes are expected. > > Xiao > > > Mridul Muralidharan 于2021年2月24日周三 上午10:41写道： > >> >> Sounds good, thanks for clarifying Hyukjin ! >> +1 on release. >

Re: [VOTE] Release Spark 3.1.1 (RC3)

rk/commit/0d5d248bdc4cdc71627162a3d20c42ad19f24ef4 > and .. KafkaDelegationTokenSuite is flaky ( > https://issues.apache.org/jira/browse/SPARK-31250). > > 2021년 2월 24일 (수) 오후 5:19, Mridul Muralidharan 님이 작성: > >> >> Signatures, digests, etc check out fine. >> Checked out tag and build/tested

Re: [VOTE] Release Spark 3.1.1 (RC3)

Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes I keep getting test failures with * org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *

Re: [DISCUSS] assignee practice on committers+ (possible issue on preemption)

I agree, Assignee has been used primarily to give recognition to the contributor who ended up submitting the patch which got merged. Typically jira's remain unassigned - even if it were to be assigned, it conveys no meaning or ownership or ongoing work : IMO it is equivalent to an unassigned

Re: [VOTE] Release Spark 3.1.1 (RC2)

Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes I keep getting test failures with org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite: removing this suite gets the build through though - does

Re: [VOTE] Release Spark 3.1.1 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes The sha512 signature for spark-3.1.1.tgz tripped up my scripts :-) Regards, Mridul On Wed, Jan 20, 2021 at 8:17 PM 郑瑞峰 wrote: > +1

Re: Recovering SparkR on CRAN?

I agree, is there something we can do to ensure CRAN publish goes through consistently and predictably ? If possible, it would be good to continue supporting it. Regards, Mridul On Tue, Dec 22, 2020 at 7:48 PM Felix Cheung wrote: > Ok - it took many years to get it first published, so it was

Re: [DISCUSS] Review/merge phase, and post-review

I try to follow the second option. In general, when multiple reviewers are looking at the code, sometimes addressing review comments might open up other avenues of discussion/optimization/design discussions : atleast in core, I have seen this happen often. A day or so delay is worth the increased

Re: [VOTE] Standardize Spark Exception Messages SPIP

+1 Regards, Mridul On Wed, Nov 4, 2020 at 12:41 PM Xinyi Yu wrote: > Hi all, > > We had the discussion of SPIP: Standardize Spark Exception Messages at > > http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-SPIP-Standardize-Spark-Exception-Messages-td30341.html > < >

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

I like the idea of consistent messages; it makes understanding errors easier. Having said that, Exception messages themselves are not part of the exposed contract to users; and are subject to change. We should leave that flexibility open to spark developers ... I am currently viewing this proposal

Re: Apache Spark 3.1 Preparation Status (Oct. 2020)

+1 on pushing the branch cut for increased dev time to match previous releases. Regards, Mridul On Sat, Oct 3, 2020 at 10:22 PM Xiao Li wrote: > Thank you for your updates. > > Spark 3.0 got released on Jun 18, 2020. If Nov 1st is the target date of > the 3.1 branch cut, the feature

[RESULT] [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

Hi, The vote passed with 16 +1's (6 binding) and no -1's +1s (* = binding): Xingbo Jiang Venkatakrishnan Sowrirajan Tom Graves (*) Chandni Singh DB Tsai (*) Xiao Li (*) Angers Zhu Joseph Torres Kalyan Dongjoon Hyun (*) Wenchen Fan (*) Yi Wu 叶先进 郑瑞峰 Takeshi Yamamuro Mridul Muralidharan

Re: [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

Adding my +1 as well, before closing the vote. Regards, Mridul On Sun, Sep 13, 2020 at 9:59 PM Mridul Muralidharan wrote: > Hi, > > I'd like to call for a vote on SPARK-30602 - SPIP: Support push-based > shuffle to improve shuffle efficiency. > Please take a look at: > >

[VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

Hi, I'd like to call for a vote on SPARK-30602 - SPIP: Support push-based shuffle to improve shuffle efficiency. Please take a look at: - SPIP jira: https://issues.apache.org/jira/browse/SPARK-30602 - SPIP doc:

Re: [VOTE] Release Spark 2.4.7 (RC3)

t; On Wed, Sep 9, 2020 at 6:12 AM Mridul Muralidharan > wrote: > >> >> +1 >> >> Signatures, digests, etc check out fine. >> Checked out tag and built/tested with -Pyarn -Phadoop-2.7 -Phive >> -Phive-thriftserver -Pmesos -Pkubernetes >> >&

Re: [VOTE] Release Spark 2.4.7 (RC3)

+1 Signatures, digests, etc check out fine. Checked out tag and built/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes Thanks, Mridul On Tue, Sep 8, 2020 at 8:55 AM Prashant Sharma wrote: > Please vote on releasing the following candidate as Apache Spark >

Re: Push-based shuffle SPIP

Hi, Thanks for sending out the proposal Min ! For the SPIP requirements, I am willing to act as the shepherd for this proposal. The jira + paper + proposal provides the high level design and implementation details. The vldb paper discusses the performance gains in detail for the inhouse

  1   2   3   >