Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

+1 Chao Sun 于2024年4月1日周一 07:56写道： > +1 > > On Sun, Mar 31, 2024 at 10:31 PM Hyukjin Kwon > wrote: > >> Oh I didn't send the discussion thread out as it's pretty simple, >> non-invasive and the discussion was sort of done as part of the Spark >> Connect initial discussion .. >> >> On Mon, Apr

Re: [VOTE] SPIP: State Data Source - Reader

+1 Jungtaek Lim 于2023年10月25日周三 01:06写道： > Friendly reminder: the VOTE thread got 2 binding votes and needs 1 more > binding vote to pass. > > On Wed, Oct 25, 2023 at 1:21 AM Bartosz Konieczny > wrote: > >> +1 >> >> On Tuesday, October 24, 2023, Jia Fan wrote: >> >>> +1 >>> >>> L. C. Hsieh

Re: [DISCUSS] SPIP: State Data Source - Reader

+1, I have no issues with the practicality and value of this feature itself. I've left some comments concerning ongoing maintenance and compatibility-related matters, which we can continue to discuss. Jungtaek Lim 于2023年10月17日周二 05:23写道： > Thanks Bartosz and Anish for your support! > > I'll

Re: [ANNOUNCE] Apache Spark 3.5.0 released

s release, Congratulations! > > On Mon, Sep 18, 2023 at 2:16 PM Maxim Gekk > wrote: > >> Thank you for the work, Yuanjian! >> >> On Mon, Sep 18, 2023 at 6:28 AM beliefer wrote: >> >>> Congratulations! Apache Spark. >>> >>> >>> >&

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 Denny Lee 于2023年9月26日周二 12:07写道： > +1 > > On Tue, Sep 26, 2023 at 10:52 Maciej wrote: > >> +1 >> >> Best regards, >> Maciej Szymkiewicz >> >> Web: https://zero323.net >> PGP: A30CEF0C31A501EC >> >> On 9/26/23 17:12, Michel Miotto Barbosa wrote: >> >> +1 >> >> A disposição | At your disposal

[ANNOUNCE] Apache Spark 3.5.0 released

Hi All, We are happy to announce the availability of *Apache Spark 3.5.0*! Apache Spark 3.5.0 is the sixth release of the 3.x line. To download Spark 3.5.0, head over to the download page: https://spark.apache.org/downloads.html (Please note: the PyPi upload is pending due to a size limit

[VOTE][RESULT] Release Apache Spark 3.5.0 (RC5)

The vote passes with 13 +1s (8 binding +1s). Thank you all who helped with the release! (* = binding) +1: - Mridul Muralidharan (*) - Yuanjian Li - Xiao Li (*) - Gengliang Wang (*) - Hyukjin Kwon (*) - Ruifeng Zheng (*) - Jungtaek Lim - Wenchen Fan (*) - Jia Fan - Jie Yang - Yuming Wang

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 (non-binding) Yuanjian Li 于2023年9月11日周一 09:36写道： > @Peter Toth I've looked into the details of this > issue, and it appears that it's neither a regression in version 3.5.0 nor a > correctness issue. It's a bug related to a new feature. I think we can fix > this in 3.

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

Muralidharan 于2023年9月10日周日 04:12写道： > > +1 > > Signatures, digests, etc check out fine. > Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes > > Regards, > Mridul > > On Sat, Sep 9, 2023 at 10:02 AM Yuanjian Li > wrote: > >> Please

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

thub.com/apache/spark/pull/42850 >> - >> https://github.com/apache/spark/commit/b2b2ba97d3003d25d159943ab8a4bf50e421fdab >> (branch-3.5) >> >> Dongjoon. >> >> >>>>>> >>>>>> On Wed, Sep 6, 2023 at 8:11 AM Yuanjian Li >>

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

ch_Talebzadeh >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The

[VOTE] Release Apache Spark 3.5.0 (RC5)

y targeted please ping me or a committer to help target the issue. Thanks, Yuanjian Li

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

bf50e421fdab > (branch-3.5) > > Dongjoon. > > >>>>> >>>>> On Wed, Sep 6, 2023 at 8:11 AM Yuanjian Li >>>>> wrote: >>>>> >>>>> Please vote on releasing the following candidate(RC4) as Apache Spark >>>>>

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1 (non-binding) Xiao Li 于2023年9月6日周三 15:27写道： > +1 > > Xiao > > Herman van Hovell 于2023年9月6日周三 22:08写道： > >> Tested connect, and everything looks good. >> >> +1 >> >> On Wed, Sep 6, 2023 at 8:11 AM Yuanjian Li >> wrote: >> >>

Release Note of Apache Spark 3.5.0

draft release note of Apache Spark 3.5.0 and feel free to add your comments if any. Thanks, Yuanjian Li

[VOTE] Release Apache Spark 3.5.0 (RC4)

y targeted please ping me or a committer to help target the issue. Thanks, Yuanjian Li

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

Sure, no problem. Holden Karau 于2023年9月2日周六 22:10写道： > Can we delay the next RC cut until after Labor Day? > > On Sat, Sep 2, 2023 at 9:59 PM Yuanjian Li wrote: > >> Thank you for all the reports! >> The vote has failed. I plan to cut RC4 in two days. >> >

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

; this fix in 3.5. >> >> Thanks, >> Wenchen >> >> On Thu, Aug 31, 2023 at 9:09 PM Ian Manning >> wrote: >> >>> +1 (non-binding) >>> >>> Using Spark Core, Spark SQL, Structured Streaming. >>> >>> On Tue, Aug 29, 2023

[VOTE] Release Apache Spark 3.5.0 (RC3)

y targeted please ping me or a committer to help target the issue. Thanks, Yuanjian Li

Re: [VOTE] Release Apache Spark 3.5.0 (RC2)

due to SPARK-43646 <https://issues.apache.org/jira/browse/SPARK-43646> > and SPARK-44784 <https://issues.apache.org/jira/browse/SPARK-44784> not > yet being fixed. > > > > Jie Yang > > > > *发件人**: *Sean Owen > *日期**: *2023年8月20日 星期日 04:43 > *收件人**: *

[VOTE] Release Apache Spark 3.5.0 (RC2)

y targeted please ping me or a committer to help target the issue. Thanks, Yuanjian Li

Re: [VOTE] Release Apache Spark 3.5.0 (RC1)

/github.com/apache/spark/actions/runs/5832898984/job/15819181762 >> >> >> >> I think we should address this issue before the release of Apache Spark >> 3.5.0. >> >> >> >> Jie Yang >> >> >> >> *发件人**: *Yuanjian Li >>

Re: [VOTE] Release Apache Spark 3.5.0 (RC1)

36. >> >> >> >> *发件人**: *Sean Owen >> *日期**: *2023年8月7日 星期一 11:05 >> *收件人**: *Yuanjian Li >> *抄送**: *Spark dev list >> *主题**: *Re: [VOTE] Release Apache Spark 3.5.0 (RC1) >> >> >> -- >> >> *【外

[VOTE] Release Apache Spark 3.5.0 (RC1)

y targeted please ping me or a committer to help target the issue. Thanks, Yuanjian Li

[Reminder] Spark 3.5 RC Cut

Hi everyone, Following the release timeline, I will cut the RC on* Tuesday, Aug 1st at 1 pm PST* as scheduled. DateEvent July 17th 2023 Late July 2023 Code freeze. Release branch cut. QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged. August 2023

Re: Spark 3.5 Branch Cut

begin your QA against branch-3.5 now. Thank you! Raghu Angadi 于2023年7月17日周一 13:29写道： > Thanks Yuanjian for accepting these for warmfix. > > Raghu. > > On Mon, Jul 17, 2023 at 1:04 PM Yuanjian Li > wrote: > >> Hi, all >> >> FYI, I cut branch-3.5 as https://

Spark 3.5 Branch Cut

Hi, all FYI, I cut branch-3.5 as https://github.com/apache/spark/tree/branch-3.5 Here is the complete list of exception merge requests received before the cut: - SPARK-44421: Reattach to existing execute in Spark Connect (server mechanism) - SPARK-44423: Reattach to existing

Re: Time for Spark v3.5.0 release

;>> > >>>>> > >>>>> > 发件人: Maxim Gekk >>>>> > 日期: 2023年7月4日 星期二 17:24 >>>>> > 收件人: Kent Yao >>>>> > 抄送: "dev@spark.apache.org" >>>>> > 主题: Re: Time for Spark v3.5.0 release &

[Reminder] Spark 3.5 Branch Cut

Hi everyone, As discussed earlier in "Time for Spark v3.5.0 release", I will cut branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled. Please plan your PR merge accordingly with the given timeline. Currently, we have received the following exception merge requests: - SPARK-44421:

Time for Spark v3.5.0 release

Hi All, According to the Spark versioning policy at https://spark.apache.org/versioning-policy.html, should we cut *branch-3.5* on *July 17th, 2023*? (We initially proposed January 16th, but since it's a Sunday, I suggest we postpone it by one day). I would like to volunteer as the release

Re: Welcoming three new PMC members

Congrats everyone! L. C. Hsieh 于2022年8月9日 周二19:01写道： > Congrats! > > On Tue, Aug 9, 2022 at 5:38 PM Chao Sun wrote: > > > > Congrats everyone! > > > > On Tue, Aug 9, 2022 at 5:36 PM Dongjoon Hyun > wrote: > > > > > > Congrat to all! > > > > > > Dongjoon. > > > > > > On Tue, Aug 9, 2022 at 5:13

Re: Welcome Xinrong Meng as a Spark committer

Congratulations, Xinrong! XiDuo You 于2022年8月9日 周二19:18写道： > Congratulations! > > Haejoon Lee 于2022年8月10日周三 09:30写道： > > > > Congrats, Xinrong!! > > > > On Tue, Aug 9, 2022 at 5:12 PM Hyukjin Kwon wrote: > >> > >> Hi all, > >> > >> The Spark PMC recently added Xinrong Meng as a committer on the

Re: [DISCUSS] Add RocksDB StateStore

Hi all, Following the latest comments in SPARK-34198 , Databricks decided to donate the commercial implementation of the RocksDBStateStore. Compared with the original decision, there’s only one topic we want to raise again for discussion: can we

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Awesome! Thanks for making this happen, Hyukjin! Yi Wu 于2021年4月14日周三 下午2:51写道： > Thanks for the great work, Hyukjin! > > On Wed, Apr 14, 2021 at 1:00 PM Gengliang Wang wrote: > >> Thanks for the amazing work, Hyukjin! >> I created a PR for trial and it looks well so far: >>

Re: Welcoming six new Apache Spark committers

Congrats all! Well deserved!! Yi Wu 于2021年3月29日周一 上午10:01写道： > Thank you, everyone! Thanks for all the help! > > Yi > > On Sun, Mar 28, 2021 at 4:53 PM Gengliang Wang wrote: > >> Congrats all! >> >> On Sun, Mar 28, 2021 at 7:09 AM Xiao Li wrote: >> >>> Congratulations, everyone! >>> >>> Xiao

Re: What's the root cause of not supporting multiple aggregations in structured streaming?

this subject I wrote a blog article that gives details about the > watermark architecture proposal that was discussed in the design doc and in > the PR: > > > https://echauchot.blogspot.com/2020/11/watermark-architecture-proposal-for.html > > Best > > Etienne > On 29/09/2020

Re: [DISCUSS] Disable streaming query with possible correctness issue by default

Already +1 in the PR. It would be great to mention the new config in the SS migration guide. Ryan Blue 于2020年11月11日周三 上午7:48写道： > +1, I agree with Tom. > > On Tue, Nov 10, 2020 at 3:00 PM Dongjoon Hyun > wrote: > >> +1 for Apache Spark 3.1.0. >> >> Bests, >> Dongjoon. >> >> On Tue, Nov 10,

Re: What's the root cause of not supporting multiple aggregations in structured streaming?

Thanks for the great discussion! Also interested in this feature and did some investigation before. As Arun mentioned, similar to the "update" mode, the "complete" mode also needs more design. We might need an operation level output mode for the complete mode support. That is to say, if we use

Re: Welcoming some new Apache Spark committers

Congratulations!! huaxin gao 于2020年7月16日周四 上午6:24写道： > Thanks everyone! I am looking forward to working with you all in the > future. > > On Tue, Jul 14, 2020 at 5:02 PM Hyukjin Kwon wrote: > >> Congrats! >> >> 2020년 7월 15일 (수) 오전 7:56, Takeshi Yamamuro 님이 작성: >> >>> Congrats, all! >>> >>> On

Re: [DISCUSS] Drop Python 2, 3.4 and 3.5

+1, especially Python 2 Holden Karau 于2020年7月2日周四 上午10:20写道： > I’m ok with us dropping Python 2, 3.4, and 3.5 in Spark 3.1 forward. It > will be exciting to get to use more recent Python features. The most recent > Ubuntu LTS ships with 3.7, and while the previous LTS ships with 3.5, if > folks

[DISCUSS] Apache Spark 3.0.1 Release

Hi dev-list, I’m writing this to raise the discussion about Spark 3.0.1 feasibility since 4 blocker issues were found after Spark 3.0.0: 1. [SPARK-31990] The state store compatibility broken will cause a correctness issue when

Re: [VOTE] Release Spark 2.4.6 (RC1)

Spark 2.4.0. >>> >>> Bests, >>> Dongjoon. >>> >>> On Fri, May 8, 2020 at 10:26 AM Holden Karau >>> wrote: >>> >>>> Can you provide a bit more context (is it a regression?) >>>> >>>> On Fri, May 8, 20

Re: [VOTE] Release Spark 2.4.6 (RC1)

Hi Holden, I'm working on the bugfix of SPARK-31663 , let me post it here since it's a correctness bug and also affects 2.4.6. Best, Yuanjian Sean Owen 于2020年5月8日周五 下午11:42写道： > +1 from me. The usual: sigs OK, license looks as intended, tests

Re: [DISCUSS] PostgreSQL dialect

Thanks all of you for joining the discussion. The PR is given in https://github.com/apache/spark/pull/26763, all the PostgreSQL dialect related PRs are linked in the description. Hoping the authors could help in reviewing. Best, Yuanjian Driesprong, Fokko 于2019年12月1日周日 下午7:24写道： > +1

Re: Welcoming some new committers and PMC members

Congratulations! sujith chacko 于2019年9月10日周二 上午10:15写道： > Congratulations all. > > On Tue, 10 Sep 2019 at 7:27 AM, Haibo wrote: > >> congratulations~ >> >> >> >> 在2019年09月10日 09:30，Joseph Torres >> 写道： >> >> congratulations! >> >> On Mon, Sep 9, 2019 at 6:27 PM 王 斐 wrote: >> >>>

Re: [SPARK-23207] Repro

Hi Tyson, Thanks for the reporting! I reproduced this locally based on your code with some changes, which only keep the wrong answer job. The code as below: import scala.sys.process._ import org.apache.spark.TaskContext val res = spark.range(0, 1 * 1, 1).map{ x => (x % 1000, x)} // kill

Re: Welcome Jose Torres as a Spark committer

Congrats Jose! Best, Yuanjian Takeshi Yamamuro 于2019年1月30日周三 上午8:21写道： > Congrats, Jose! > > Best, > Takeshi > > On Wed, Jan 30, 2019 at 6:10 AM Jungtaek Lim wrote: > >> Congrats Jose! Well deserved. >> >> - Jungtaek Lim (HeartSaVioR) >> >> 2019년 1월 30일 (수) 오전 5:19, Dongjoon Hyun 님이 작성: >>

Re: Continuous task retry support

> > *I found that task retries are currently not supported > > in > continuous processing mode. Is there

Re: [DISCUSS] SPIP: Native support of session window

Cool, thanks! Sorry for the late reply, we'll check out the UT and your design doc ASAP when we back from National Day holiday. Thanks, Yuanjian Li Jungtaek Lim 于2018年9月29日周六 上午5:21写道： > Btw, just wrote up detailed design doc on existing patch: > > https://docs.google.com/d

Re: welcome a new batch of committers

Congratulations to all and thanks for all your help!! Bhupendra Mishra 于2018年10月6日周六 上午11:38写道： > Congratulations to all of you > Good Luck > Regards > > On Wed, Oct 3, 2018 at 2:29 PM Reynold Xin wrote: > >> Hi all, >> >> The Apache Spark PMC has recently voted to add several new committers

Re: [DISCUSS] SPIP: Native support of session window

<https://docs.google.com/document/d/1zeAc7QKSO7J4-Yk06kc76kvldl-QHLCDJuu04d7k2bg/edit?usp=sharing> Thanks, Yuanjian Li > 在 2018年9月28日，06:22，Jungtaek Lim <mailto:kabh...@gmail.com>> 写道： > > Hi all, > > I would like to initiate discussion thread to discuss "Native supp

Re: Something wrong of Jenkins proxy

, 2018 at 8:37 PM, shane knapp wrote: >> >>> i just noticed this... taking a look now. >>> >>> On Sun, Sep 23, 2018 at 4:38 AM, Yuanjian Li >>> wrote: >>> >>>> Hi devs, >>>> Is there something wrong of Jenkins proxy? >>

Something wrong of Jenkins proxy

Hi devs, Is there something wrong of Jenkins proxy? [image: image.png] I got this proxy 500 whole days. Thanks, Yuanjian Li

Re: [Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data

will be done at October as expect. We'll post more benchmark and detailed work at that time. I'm still reading your discussion document and happy to give more feedback in the doc. Thanks, Yuanjian Li Matt Cheah 于2018年9月1日周六 上午8:42写道： > Hi everyone, > > > > I filed SPA

Re: [DISCUSS] Adaptive execution in Spark SQL

componentsalgorithm in GraphFrame. With enabling AE, the duration of app reduce from 58min to 32min, almost 100% boosting on performance improvement. The detailed screenshot and config in the JIRA SPARK-23128 <https://issues.apache.org/jira/browse/SPARK-23128> attached pdf. Thanks, Yuanjian Li Wang, Car

Re: Design for continuous processing shuffle

Hi Joseph and devs, Happy to see the discussion of CP shuffle, as comment in https://issues.apache.org/jira/browse/SPARK-20928?focusedCommentId=16245556=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16245556

Re: Welcome Zhenhua Wang as a Spark committer

Congratulations Zhenhua!! 2018-04-02 13:28 GMT+08:00 Wenchen Fan : > Hi all, > > The Spark PMC recently added Zhenhua Wang as a committer on the project. > Zhenhua is the major contributor of the CBO project, and has been > contributing across several areas of Spark for a