Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1 RuifengZheng ruife...@foxmail.com --Original-- From: "Wenchen Fan"

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1 RuifengZheng ruife...@foxmail.com --Original-- From: "Wenchen Fan"

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1, I'm looking forward to it! On Thu, Nov 17, 2022 at 9:44 AM Ye Zhou wrote: > +1 (non-binding) > Thanks for proposing this improvement to SHS, it resolves the main > performance issue within SHS. > > On Wed, Nov 16, 2022 at 1:15 PM Jungtaek Lim > wrote: > >> +1 >> >> Nice to see the chance

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1 On Thu, Nov 17, 2022 at 10:20 AM Yang,Jie(INF) wrote: > +1,non-binding > > > > The test combination of Java 11 + Scala 2.12 and Java 11 + Scala 2.13 has > passed. > > > > Yang Jie > > > > *发件人**: *Chris Nauroth > *日期**: *2022年11月17日 星期四 04:27 > *收件人**: *Yuming Wang > *抄送**:

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1,non-binding The test combination of Java 11 + Scala 2.12 and Java 11 + Scala 2.13 has passed. Yang Jie 发件人: Chris Nauroth 日期: 2022年11月17日 星期四 04:27 收件人: Yuming Wang 抄送: "Yang,Jie(INF)" , Dongjoon Hyun , huaxin gao , "L. C. Hsieh" , Chao Sun , dev 主题: Re: [VOTE] Release Spark 3.2.3

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1 (non-binding) Thanks for proposing this improvement to SHS, it resolves the main performance issue within SHS. On Wed, Nov 16, 2022 at 1:15 PM Jungtaek Lim wrote: > +1 > > Nice to see the chance for driver to reduce resource usage and increase > stability, especially the fact that the driver

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1 Nice to see the chance for driver to reduce resource usage and increase stability, especially the fact that the driver is SPOF. It's even promising to have a future plan to pre-bake the kvstore for SHS from the driver. Thanks for driving the effort, Gengliang! On Thu, Nov 17, 2022 at 5:32 AM

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1 (non-binding) Gengliang, thank you for the SPIP. Chris Nauroth On Wed, Nov 16, 2022 at 4:27 AM Maciej wrote: > +1 > > On 11/16/22 13:19, Yuming Wang wrote: > > +1, non-binding > > > > On Wed, Nov 16, 2022 at 8:12 PM Yang,Jie(INF) > > wrote: > > > > +1,

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1 (non-binding) * Verified all checksums. * Verified all signatures. * Built from source, with multiple profiles, to full success, for Java 11 and Scala 2.12: * build/mvn -Phadoop-3.2 -Phadoop-cloud -Phive-2.3 -Phive-thriftserver -Pkubernetes -Pscala-2.12 -Psparkr -Pyarn -DskipTests clean

[DISCUSS] Python file to be passed as argument using command-line option for PySpark

Hi All, I have raised an improvement jira for passing a python file as an command line argument for Pyspark. The alternative is to use spark-submit but if there can be a uniform way to pass files as arguments like it is done in case of spark-shell and spark-sql (like -i or -f option). I have

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1 On Wed, Nov 16, 2022 at 2:28 PM Yang,Jie(INF) wrote: > I switched Scala 2.13 to Scala 2.12 today. The test is still in progress > and it has not been hung. > > > > Yang Jie > > > > *发件人**: *Dongjoon Hyun > *日期**: *2022年11月16日 星期三 01:17 > *收件人**: *"Yang,Jie(INF)" > *抄送**: *huaxin gao , "L.

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1 On 11/16/22 13:19, Yuming Wang wrote: +1, non-binding On Wed, Nov 16, 2022 at 8:12 PM Yang,Jie(INF) > wrote: +1, non-binding __ __ Yang Jie __ __ *发件人**: *Mridul Muralidharan mailto:mri...@gmail.com>> *日期**:

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1, non-binding On Wed, Nov 16, 2022 at 8:12 PM Yang,Jie(INF) wrote: > +1, non-binding > > > > Yang Jie > > > > *发件人**: *Mridul Muralidharan > *日期**: *2022年11月16日 星期三 17:35 > *收件人**: *Kent Yao > *抄送**: *Gengliang Wang , dev > *主题**: *Re: [VOTE][SPIP] Better Spark UI scalability and Driver

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1, non-binding Yang Jie 发件人: Mridul Muralidharan 日期: 2022年11月16日 星期三 17:35 收件人: Kent Yao 抄送: Gengliang Wang , dev 主题: Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications +1 Would be great to see history server performance improvements and lower

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1 Would be great to see history server performance improvements and lower resource utilization at driver ! Regards, Mridul On Wed, Nov 16, 2022 at 2:38 AM Kent Yao wrote: > +1, non-binding > > Gengliang Wang 于2022年11月16日周三 16:36写道： > > > > Hi all, > > > > I’d like to start a vote for SPIP:

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1, non-binding Gengliang Wang 于2022年11月16日周三 16:36写道： > > Hi all, > > I’d like to start a vote for SPIP: "Better Spark UI scalability and Driver > stability for large applications" > > The goal of the SPIP is to improve the Driver's stability by supporting > storing Spark's UI data on

[VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

Hi all, I’d like to start a vote for SPIP: "Better Spark UI scalability and Driver stability for large applications" The goal of the SPIP is to improve the Driver's stability by supporting storing Spark's UI data on RocksDB. Furthermore, to fasten the read and write operations on RocksDB, it

Re: [DISCUSS] SPIP: Better Spark UI scalability and Driver stability for large applications

With the positive feedback from Mridul and Wenchen, I will officially start the vote. On Tue, Nov 15, 2022 at 8:57 PM Wenchen Fan wrote: > This looks great! UI stability/scalability has been a pain point for a > long time. > > On Sat, Nov 12, 2022 at 5:24 AM Gengliang Wang wrote: > >> Hi