Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations! On Mon, Oct 9, 2023 at 5:06 AM Kent Yao wrote: > Congrats! > > Kent > > > 在 2023年10月7日星期六，John Zhuge 写道： > >> Congratulations! >> >> On Fri, Oct 6, 2023 at 6:41 PM Yi Wu >> wrote: >> >>> Congrats! >>> >>> On Sat, Oct 7, 2023 at 9:24 AM XiDuo You wrote: >>>

Re: Welcome to Our New Apache Spark Committer and PMCs

Congrats! Kent 在 2023年10月7日星期六，John Zhuge 写道： > Congratulations! > > On Fri, Oct 6, 2023 at 6:41 PM Yi Wu wrote: > >> Congrats! >> >> On Sat, Oct 7, 2023 at 9:24 AM XiDuo You wrote: >> >>> Congratulations! >>> >>> Prashant Sharma 于2023年10月6日周五 00:26写道： >>> > >>> > Congratulations  >>> >

Watermark on late data only

Hi, I've been analyzing the watermark propagation added in the 3.5.0 recently and had to return to the basics of watermarks. One question is still unanswered in my head. Why are the watermarks reserved to stateful queries? Can't they apply to the filtering late date out only? The reason is only

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations! On Fri, Oct 6, 2023 at 6:41 PM Yi Wu wrote: > Congrats! > > On Sat, Oct 7, 2023 at 9:24 AM XiDuo You wrote: > >> Congratulations! >> >> Prashant Sharma 于2023年10月6日周五 00:26写道： >> > >> > Congratulations  >> > >> > On Wed, 4 Oct, 2023, 8:52 pm huaxin gao, >> wrote: >> >> >> >>

Re: Welcome to Our New Apache Spark Committer and PMCs

Congrats! On Sat, Oct 7, 2023 at 9:24 AM XiDuo You wrote: > Congratulations! > > Prashant Sharma 于2023年10月6日周五 00:26写道： > > > > Congratulations  > > > > On Wed, 4 Oct, 2023, 8:52 pm huaxin gao, wrote: > >> > >> Congratulations! > >> > >> On Wed, Oct 4, 2023 at 7:39 AM Chao Sun wrote: > >>>

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations! Prashant Sharma 于2023年10月6日周五 00:26写道： > > Congratulations  > > On Wed, 4 Oct, 2023, 8:52 pm huaxin gao, wrote: >> >> Congratulations! >> >> On Wed, Oct 4, 2023 at 7:39 AM Chao Sun wrote: >>> >>> Congratulations! >>> >>> On Wed, Oct 4, 2023 at 5:11 AM Jungtaek Lim >>>

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations  On Wed, 4 Oct, 2023, 8:52 pm huaxin gao, wrote: > Congratulations! > > On Wed, Oct 4, 2023 at 7:39 AM Chao Sun wrote: > >> Congratulations! >> >> On Wed, Oct 4, 2023 at 5:11 AM Jungtaek Lim >> wrote: >> >>> Congrats! >>> >>> 2023년 10월 4일 (수) 오후 5:04, yangjie01 님이 작성: >>>

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations! On Wed, Oct 4, 2023 at 7:39 AM Chao Sun wrote: > Congratulations! > > On Wed, Oct 4, 2023 at 5:11 AM Jungtaek Lim > wrote: > >> Congrats! >> >> 2023년 10월 4일 (수) 오후 5:04, yangjie01 님이 작성: >> >>> Congratulations! >>> >>> >>> >>> Jie Yang >>> >>> >>> >>> *发件人**: *Dongjoon Hyun

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations! On Wed, Oct 4, 2023 at 5:11 AM Jungtaek Lim wrote: > Congrats! > > 2023년 10월 4일 (수) 오후 5:04, yangjie01 님이 작성: > >> Congratulations! >> >> >> >> Jie Yang >> >> >> >> *发件人**: *Dongjoon Hyun >> *日期**: *2023年10月4日 星期三 13:04 >> *收件人**: *Hyukjin Kwon >> *抄送**: *Hussein Awala , Rui

Re: Welcome to Our New Apache Spark Committer and PMCs

Congrats! 2023년 10월 4일 (수) 오후 5:04, yangjie01 님이 작성: > Congratulations! > > > > Jie Yang > > > > *发件人**: *Dongjoon Hyun > *日期**: *2023年10月4日 星期三 13:04 > *收件人**: *Hyukjin Kwon > *抄送**: *Hussein Awala , Rui Wang , > Gengliang Wang , Xiao Li , " > dev@spark.apache.org" > *主题**: *Re: Welcome to

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations! Jie Yang 发件人: Dongjoon Hyun 日期: 2023年10月4日 星期三 13:04 收件人: Hyukjin Kwon 抄送: Hussein Awala , Rui Wang , Gengliang Wang , Xiao Li , "dev@spark.apache.org" 主题: Re: Welcome to Our New Apache Spark Committer and PMCs Congratulations! Dongjoon. On Tue, Oct 3, 2023 at 5:25 PM

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations! Dongjoon. On Tue, Oct 3, 2023 at 5:25 PM Hyukjin Kwon wrote: > Woohoo! > > On Tue, 3 Oct 2023 at 22:47, Hussein Awala wrote: > >> Congrats to all of you! >> >> On Tue 3 Oct 2023 at 08:15, Rui Wang wrote: >> >>> Congratulations! Well deserved! >>> >>> -Rui >>> >>> >>> On Mon,

Re: Welcome to Our New Apache Spark Committer and PMCs

Congrats! On Wed, Oct 4, 2023 at 8:25 AM Hyukjin Kwon wrote: > Woohoo! > > On Tue, 3 Oct 2023 at 22:47, Hussein Awala wrote: > >> Congrats to all of you! >> >> On Tue 3 Oct 2023 at 08:15, Rui Wang wrote: >> >>> Congratulations! Well deserved! >>> >>> -Rui >>> >>> >>> On Mon, Oct 2, 2023 at

Re: Welcome to Our New Apache Spark Committer and PMCs

Woohoo! On Tue, 3 Oct 2023 at 22:47, Hussein Awala wrote: > Congrats to all of you! > > On Tue 3 Oct 2023 at 08:15, Rui Wang wrote: > >> Congratulations! Well deserved! >> >> -Rui >> >> >> On Mon, Oct 2, 2023 at 10:32 PM Gengliang Wang wrote: >> >>> Congratulations to all! Well deserved! >>>

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations ! Looking forward to more exciting contributions :-) Regards, Mridul On Tue, Oct 3, 2023 at 2:51 AM Hussein Awala wrote: > Congrats to all of you! > > On Tue 3 Oct 2023 at 08:15, Rui Wang wrote: > >> Congratulations! Well deserved! >> >> -Rui >> >> >> On Mon, Oct 2, 2023 at

Re: Welcome to Our New Apache Spark Committer and PMCs

Congrats to all of you! On Tue 3 Oct 2023 at 08:15, Rui Wang wrote: > Congratulations! Well deserved! > > -Rui > > > On Mon, Oct 2, 2023 at 10:32 PM Gengliang Wang wrote: > >> Congratulations to all! Well deserved! >> >> On Mon, Oct 2, 2023 at 10:16 PM Xiao Li wrote: >> >>> Hi all, >>> >>>

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations! Well deserved! -Rui On Mon, Oct 2, 2023 at 10:32 PM Gengliang Wang wrote: > Congratulations to all! Well deserved! > > On Mon, Oct 2, 2023 at 10:16 PM Xiao Li wrote: > >> Hi all, >> >> The Spark PMC is delighted to announce that we have voted to add one new >> committer and

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations to all! Well deserved! On Mon, Oct 2, 2023 at 10:16 PM Xiao Li wrote: > Hi all, > > The Spark PMC is delighted to announce that we have voted to add one new > committer and two new PMC members. These individuals have consistently > contributed to the project and have clearly

Welcome to Our New Apache Spark Committer and PMCs

Hi all, The Spark PMC is delighted to announce that we have voted to add one new committer and two new PMC members. These individuals have consistently contributed to the project and have clearly demonstrated their expertise. New Committer: - Jiaan Geng (focusing on Spark Connect and Spark SQL)

[RESULT] Updating documentation hosted for EOL and maintenance releases

The vote passes with 9 +1s (6 binding +1s). (* = binding) +1: - Hyukjin Kwon * - Ruifeng Zheng * - Jiaan Geng - Yikun Jiang * - Herman van Hovell * - Michel Miotto Barbosa - Maciej Szymkiewicz * - Denny Lee - Yuanjian Li *

unsubscribe

Re: [ANNOUNCE] Apache Spark 3.5.0 released

Awesome! On Wed, 27 Sept 2023 at 11:02, Hussein Awala wrote: > I installed the package, tested it with kubernetes master from Jupyter, > and tested it with Spark Connect server, all looks good. > > On Tue, Sep 26, 2023 at 10:45 PM Yuanjian Li > wrote: > >> FYI, we received the handling from

Re: Migrating the Junit framework used in Apache Spark 4.0 from 4.x to 5.x

+1 for moving to a newer version. Thanks for driving this Jie Yang ! Regards, Mridul On Mon, Sep 25, 2023 at 10:15 AM 杨杰 wrote: > Hi all, > > In SPARK-44170 (apache/spark#43074 [1]), I’m trying to migrate the Junit > test framework used in Spark 4.0 from Junit4 to Junit5. > > > Although this

Re: [ANNOUNCE] Apache Spark 3.5.0 released

I installed the package, tested it with kubernetes master from Jupyter, and tested it with Spark Connect server, all looks good. On Tue, Sep 26, 2023 at 10:45 PM Yuanjian Li wrote: > FYI, we received the handling from Pypi > org yesterday, and the >

Re: [ANNOUNCE] Apache Spark 3.5.0 released

FYI, we received the handling from Pypi org yesterday, and the upload of version 3.5.0 has just been completed. Please assist in verifying it. Thank you! Ruifeng Zheng 于2023年9月17日周日 23:28写道： > Thanks Yuanjian for driving this release,

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 Denny Lee 于2023年9月26日周二 12:07写道： > +1 > > On Tue, Sep 26, 2023 at 10:52 Maciej wrote: > >> +1 >> >> Best regards, >> Maciej Szymkiewicz >> >> Web: https://zero323.net >> PGP: A30CEF0C31A501EC >> >> On 9/26/23 17:12, Michel Miotto Barbosa wrote: >> >> +1 >> >> A disposição | At your disposal

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 On Tue, Sep 26, 2023 at 10:52 Maciej wrote: > +1 > > Best regards, > Maciej Szymkiewicz > > Web: https://zero323.net > PGP: A30CEF0C31A501EC > > On 9/26/23 17:12, Michel Miotto Barbosa wrote: > > +1 > > A disposição | At your disposal > > Michel Miotto Barbosa >

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 Best regards, Maciej Szymkiewicz Web:https://zero323.net PGP: A30CEF0C31A501EC On 9/26/23 17:12, Michel Miotto Barbosa wrote: +1 A disposição | At your disposal Michel Miotto Barbosa https://www.linkedin.com/in/michelmiottobarbosa/ mmiottobarb...@gmail.com +55 11 984 342 347 On Tue,

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 A disposição | At your disposal Michel Miotto Barbosa https://www.linkedin.com/in/michelmiottobarbosa/ mmiottobarb...@gmail.com +55 11 984 342 347 On Tue, Sep 26, 2023 at 11:44 AM Herman van Hovell wrote: > +1 > > On Tue, Sep 26, 2023 at 10:39 AM yangjie01 > wrote: > >> +1 >> >> >> >>

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 On Tue, Sep 26, 2023 at 10:39 AM yangjie01 wrote: > +1 > > > > *发件人**: *Yikun Jiang > *日期**: *2023年9月26日 星期二 18:06 > *收件人**: *dev > *抄送**: *Hyukjin Kwon , Ruifeng Zheng < > ruife...@apache.org> > *主题**: *Re: [VOTE] Updating documentation hosted for EOL and maintenance > releases > > > >

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 发件人: Yikun Jiang 日期: 2023年9月26日 星期二 18:06 收件人: dev 抄送: Hyukjin Kwon , Ruifeng Zheng 主题: Re: [VOTE] Updating documentation hosted for EOL and maintenance releases +1, I believe it is a wise choice to update the EOL policy of the document based on the real demands of community users.

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1, I believe it is a wise choice to update the EOL policy of the document based on the real demands of community users. Regards, Yikun On Tue, Sep 26, 2023 at 1:06 PM Ruifeng Zheng wrote: > +1 > > On Tue, Sep 26, 2023 at 12:51 PM Hyukjin Kwon > wrote: > >> Hi all, >> >> I would like to

Re:Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 At 2023-09-26 13:03:56, "Ruifeng Zheng" wrote: +1 On Tue, Sep 26, 2023 at 12:51 PM Hyukjin Kwon wrote: Hi all, I would like to start the vote for updating documentation hosted for EOL and maintenance releases to improve the usability here, and in order for end users to read the

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 On Tue, Sep 26, 2023 at 12:51 PM Hyukjin Kwon wrote: > Hi all, > > I would like to start the vote for updating documentation hosted for EOL > and maintenance releases to improve the usability here, and in order for > end users to read the proper and correct documentation. > > For discussion

[VOTE] Updating documentation hosted for EOL and maintenance releases

Hi all, I would like to start the vote for updating documentation hosted for EOL and maintenance releases to improve the usability here, and in order for end users to read the proper and correct documentation. For discussion thread, please refer to

Migrating the Junit framework used in Apache Spark 4.0 from 4.x to 5.x

Hi all, In SPARK-44170 (apache/spark#43074 [1]), I’m trying to migrate the Junit test framework used in Spark 4.0 from Junit4 to Junit5. Although this involves a fair amount of code modifications, given that Junit 4 is still developed based on Java 6 source code and it hasn't released a new

unsubscribe

unsubscribe

Re: Are DataFrame rows ordered without an explicit ordering clause?

LOL, Hindsight is a very good thing and often one learns these through experience.Once told off because strict ordering was not maintained, then the lesson will never be forgotten! HTH Mich Talebzadeh, Distinguished Technologist, Solutions Architect & Engineer London United Kingdom view

Re: Are DataFrame rows ordered without an explicit ordering clause?

Now, if you are ruthless it'd make sense to randomise the order of results if someone left out the order by, to stop complacency. like that time sun changed the ordering that methods were returned in a Class.listMethods() call and everyone's junit test cases failed if they'd assumed that ordering

Re:Are DataFrame rows ordered without an explicit ordering clause?

AFAIK, The order is free whether it's SQL without spcified ORDER BY clause or DataFrame without sort. The behavior is consistent between them. At 2023-09-18 23:47:40, "Nicholas Chammas" wrote: I’ve always considered DataFrames to be logically equivalent to SQL tables or queries. In

[DISCUSS] Porting back SPARK-45178 to 3.5/3.4 version lines

Hi devs, I'd like to get some inputs for dealing with the possible correctness issue we figured. The JIRA ticket is SPARK-45178 and I described the issue and solution I proposed. Context: Source might behave incorrectly leading to correctness

Re: Plans for built-in v2 data sources in Spark 4

Instead of that, I believe you are looking for `spark.sql.sources.useV1SourceList` if the question is about "Concretely, is the plan for Spark 4 to continue defaulting to the built-in v1 data sources?". Here is the code.

Re: Plans for built-in v2 data sources in Spark 4

Thank you for linking that, Dongjoon! I found SPARK-44518 in that list which wants to turn Spark’s Hive integration into a data source. To think out loud: The big gaps between built-in v1 and v2 data sources are support for bucketing and

Re: Plans for built-in v2 data sources in Spark 4

Thank you for linking that, Dongjoon! I found SPARK-44518 in that list which wants to turn Spark’s Hive integration into a data source. IIUC, that’s very related but I’m curious if I’m thinking about this correctly: Big gaps between built-in

Re: Are DataFrame rows ordered without an explicit ordering clause?

These are good points. In traditional RDBMSs, SQL query results without an explicit *ORDER BY* clause may vary in order due to optimization, especially when no clustered index is defined. In contrast, systems like Hive and Spark SQL, which are based on distributed file storage, do not rely on

Re: Are DataFrame rows ordered without an explicit ordering clause?

Hi Nicholas, Your point "In SQL, the result order of any query is implementation-dependent without an explicit ORDER BY clause. Technically, you could run `SELECT * FROM table;` 10 times in a row and get 10 different orderings." yes I concur my understanding is the same. In SQL, the result

Re: Are DataFrame rows ordered without an explicit ordering clause?

It should be the same as SQL. Otherwise it takes away a lot of potential future optimization opportunities. On Mon, Sep 18 2023 at 8:47 AM, Nicholas Chammas < nicholas.cham...@gmail.com > wrote: > > I’ve always considered DataFrames to be logically equivalent to SQL tables > or queries. > >

Re: Are DataFrame rows ordered without an explicit ordering clause?

I think it's the same, and always has been - yes you don't have a guaranteed ordering unless an operation produces a specific ordering. Could be the result of order by, yes; I believe you would be guaranteed that reading input files results in data in the order they appear in the file, etc. 1:1

Are DataFrame rows ordered without an explicit ordering clause?

I’ve always considered DataFrames to be logically equivalent to SQL tables or queries. In SQL, the result order of any query is implementation-dependent without an explicit ORDER BY clause. Technically, you could run `SELECT * FROM table;` 10 times in a row and get 10 different orderings. I

Re: [ANNOUNCE] Apache Spark 3.5.0 released

Thanks Yuanjian for driving this release, Congratulations! On Mon, Sep 18, 2023 at 2:16 PM Maxim Gekk wrote: > Thank you for the work, Yuanjian! > > On Mon, Sep 18, 2023 at 6:28 AM beliefer wrote: > >> Congratulations! Apache Spark. >> >> >> >> At 2023-09-16 01:01:40, "Yuanjian Li" wrote: >>

Re: [ANNOUNCE] Apache Spark 3.5.0 released

Thank you for the work, Yuanjian! On Mon, Sep 18, 2023 at 6:28 AM beliefer wrote: > Congratulations! Apache Spark. > > > > At 2023-09-16 01:01:40, "Yuanjian Li" wrote: > > Hi All, > > We are happy to announce the availability of *Apache Spark 3.5.0*! > > Apache Spark 3.5.0 is the sixth release

Re:[ANNOUNCE] Apache Spark 3.5.0 released

Congratulations! Apache Spark. At 2023-09-16 01:01:40, "Yuanjian Li" wrote: Hi All, We are happy to announce the availability of Apache Spark 3.5.0! Apache Spark 3.5.0 is the sixth release of the 3.x line. To download Spark 3.5.0, head over to the download page:

Re: First Time contribution.

Welcome Ram! :-) I would recommend you to check https://issues.apache.org/jira/browse/SPARK-37935 out as a starter task. Refer to https://github.com/apache/spark/pull/41504, https://github.com/apache/spark/pull/41455 as an example PR. Or you can also add a new sub-task if you find any error

Re: First Time contribution.

Hi Ram, We have some good guidance at https://spark.apache.org/contributing.html HTH! Denny On Sun, Sep 17, 2023 at 17:18 ram manickam wrote: > > > > Hello All, > Recently, joined this community and would like to contribute. Is there a > guideline or recommendation on tasks that can be

[ANNOUNCE] Apache Spark 3.5.0 released

Hi All, We are happy to announce the availability of *Apache Spark 3.5.0*! Apache Spark 3.5.0 is the sixth release of the 3.x line. To download Spark 3.5.0, head over to the download page: https://spark.apache.org/downloads.html (Please note: the PyPi upload is pending due to a size limit

Re: Plans for built-in v2 data sources in Spark 4

Hi, Will. According to the following JIRA, as of now, there is no plan or on-going discussion to switch it. https://issues.apache.org/jira/browse/SPARK-44111 (Prepare Apache Spark 4.0.0) Thanks, Dongjoon. On Wed, Sep 13, 2023 at 9:02 AM Will Raschkowski wrote: > Hey everyone, > > > > I was

Re: Write Spark Connection client application in Go

Thanks Holden and Martin for the nice words and feedback :) On Wed, Sep 13, 2023 at 8:22 AM Martin Grund wrote: > This is absolutely awesome! Thank you so much for dedicating your time to > this project! > > > On Wed, Sep 13, 2023 at 6:04 AM Holden Karau wrote: > >> That’s so cool! Great work

Plans for built-in v2 data sources in Spark 4

Hey everyone, I was wondering what the plans are for Spark's built-in v2 file data sources in Spark 4. Concretely, is the plan for Spark 4 to continue defaulting to the built-in v1 data sources? And if yes, what are the blockers for defaulting to v2? I see, just as example, that writing

Re: Write Spark Connection client application in Go

This is absolutely awesome! Thank you so much for dedicating your time to this project! On Wed, Sep 13, 2023 at 6:04 AM Holden Karau wrote: > That’s so cool! Great work y’all :) > > On Tue, Sep 12, 2023 at 8:14 PM bo yang wrote: > >> Hi Spark Friends, >> >> Anyone interested in using Golang

unsubscribe

Re: Write Spark Connection client application in Go

That’s so cool! Great work y’all :) On Tue, Sep 12, 2023 at 8:14 PM bo yang wrote: > Hi Spark Friends, > > Anyone interested in using Golang to write Spark application? We created a > Spark > Connect Go Client library . > Would love to hear

unsubscribe

unsubscribe

Write Spark Connection client application in Go

Hi Spark Friends, Anyone interested in using Golang to write Spark application? We created a Spark Connect Go Client library . Would love to hear feedback/thoughts from the community. Please see the quick start guide

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 (non-binding) Jungtaek Lim 于2023年9月12日周二 15:14写道： > > +1 (non-binding) > > Thanks for driving this release and the patience on multiple RCs! > > On Tue, Sep 12, 2023 at 10:00 AM Yuanjian Li wrote: >> >> +1 (non-binding) >> >> Yuanjian Li 于2023年9月11日周一 09:36写道： >>> >>> @Peter Toth I've

[VOTE][RESULT] Release Apache Spark 3.5.0 (RC5)

The vote passes with 13 +1s (8 binding +1s). Thank you all who helped with the release! (* = binding) +1: - Mridul Muralidharan (*) - Yuanjian Li - Xiao Li (*) - Gengliang Wang (*) - Hyukjin Kwon (*) - Ruifeng Zheng (*) - Jungtaek Lim - Wenchen Fan (*) - Jia Fan - Jie Yang - Yuming Wang (*) -

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 Dongjoon. On 2023/09/12 03:38:37 Kent Yao wrote: > +1 (non-binding), great work! > > Kent Yao > > Yuming Wang 于2023年9月12日周二 11:32写道： > > > > +1. > > > > On Tue, Sep 12, 2023 at 10:57 AM yangjie01 > > wrote: > >> > >> +1 > >> > >> > >> > >> 发件人: Jia Fan > >> 日期: 2023年9月12日 星期二 10:08 > >>

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 (non-binding), great work! Kent Yao Yuming Wang 于2023年9月12日周二 11:32写道： > > +1. > > On Tue, Sep 12, 2023 at 10:57 AM yangjie01 > wrote: >> >> +1 >> >> >> >> 发件人: Jia Fan >> 日期: 2023年9月12日 星期二 10:08 >> 收件人: Ruifeng Zheng >> 抄送: Hyukjin Kwon , Xiao Li , >> Mridul Muralidharan , Peter Toth

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1. On Tue, Sep 12, 2023 at 10:57 AM yangjie01 wrote: > +1 > > > > *发件人**: *Jia Fan > *日期**: *2023年9月12日 星期二 10:08 > *收件人**: *Ruifeng Zheng > *抄送**: *Hyukjin Kwon , Xiao Li , > Mridul Muralidharan , Peter Toth , > Spark dev list , Yuanjian Li > > *主题**: *Re: [VOTE] Release Apache Spark 3.5.0

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 发件人: Jia Fan 日期: 2023年9月12日 星期二 10:08 收件人: Ruifeng Zheng 抄送: Hyukjin Kwon , Xiao Li , Mridul Muralidharan , Peter Toth , Spark dev list , Yuanjian Li 主题: Re: [VOTE] Release Apache Spark 3.5.0 (RC5) +1 Ruifeng Zheng mailto:ruife...@apache.org>> 于2023年9月12日周二 08:46写道： +1 On Tue, Sep 12,

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 Ruifeng Zheng 于2023年9月12日周二 08:46写道： > +1 > > On Tue, Sep 12, 2023 at 7:24 AM Hyukjin Kwon wrote: > >> +1 >> >> On Tue, Sep 12, 2023 at 7:05 AM Xiao Li wrote: >> >>> +1 >>> >>> Xiao >>> >>> Yuanjian Li 于2023年9月11日周一 10:53写道： >>> @Peter Toth I've looked into the details of this

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 On Tue, Sep 12, 2023 at 9:00 AM Yuanjian Li wrote: > +1 (non-binding) > > Yuanjian Li 于2023年9月11日周一 09:36写道： > >> @Peter Toth I've looked into the details of this >> issue, and it appears that it's neither a regression in version 3.5.0 nor a >> correctness issue. It's a bug related to a

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 (non-binding) Thanks for driving this release and the patience on multiple RCs! On Tue, Sep 12, 2023 at 10:00 AM Yuanjian Li wrote: > +1 (non-binding) > > Yuanjian Li 于2023年9月11日周一 09:36写道： > >> @Peter Toth I've looked into the details of this >> issue, and it appears that it's neither a

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 On Tue, Sep 12, 2023 at 7:24 AM Hyukjin Kwon wrote: > +1 > > On Tue, Sep 12, 2023 at 7:05 AM Xiao Li wrote: > >> +1 >> >> Xiao >> >> Yuanjian Li 于2023年9月11日周一 10:53写道： >> >>> @Peter Toth I've looked into the details of this >>> issue, and it appears that it's neither a regression in

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 On Tue, Sep 12, 2023 at 7:05 AM Xiao Li wrote: > +1 > > Xiao > > Yuanjian Li 于2023年9月11日周一 10:53写道： > >> @Peter Toth I've looked into the details of this >> issue, and it appears that it's neither a regression in version 3.5.0 nor a >> correctness issue. It's a bug related to a new

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 On Mon, Sep 11, 2023 at 11:28 AM Xiao Li wrote: > +1 > > Xiao > > Yuanjian Li 于2023年9月11日周一 10:53写道： > >> @Peter Toth I've looked into the details of this >> issue, and it appears that it's neither a regression in version 3.5.0 nor a >> correctness issue. It's a bug related to a new

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 Xiao Yuanjian Li 于2023年9月11日周一 10:53写道： > @Peter Toth I've looked into the details of this > issue, and it appears that it's neither a regression in version 3.5.0 nor a > correctness issue. It's a bug related to a new feature. I think we can fix > this in 3.5.1 and list it as a known issue

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

Thanks Yuanjian. Please disregard my -1 then. Yuanjian Li ezt írta (időpont: 2023. szept. 11., H, 18:36): > @Peter Toth I've looked into the details of this > issue, and it appears that it's neither a regression in version 3.5.0 nor a > correctness issue. It's a bug related to a new feature. I

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 (non-binding) Yuanjian Li 于2023年9月11日周一 09:36写道： > @Peter Toth I've looked into the details of this > issue, and it appears that it's neither a regression in version 3.5.0 nor a > correctness issue. It's a bug related to a new feature. I think we can fix > this in 3.5.1 and list it as a

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

@Peter Toth I've looked into the details of this issue, and it appears that it's neither a regression in version 3.5.0 nor a correctness issue. It's a bug related to a new feature. I think we can fix this in 3.5.1 and list it as a known issue of the Scala client of Spark Connect in 3.5.0. Mridul

unsubscribe

unsubscribe

unsubscribe

unsubscribe

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Sat, Sep 9, 2023 at 10:02 AM Yuanjian Li wrote: > Please vote on releasing the following candidate(RC5) as Apache Spark > version 3.5.0. > > The vote is open

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

Hi Yuanjian, Sorry, -1 from me. Let's not introduce this bugs in 3.5: https://issues.apache.org/jira/browse/SPARK-45109 / https://github.com/apache/spark/pull/42863 Best, Peter Yuanjian Li ezt írta (időpont: 2023. szept. 10., V, 10:39): > Yes, SPARK-44805 has been included. For the commits

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

@ian.a.mann...@gmail.com Thank you for your question. Because the voting period hasn't ended yet and this fix has just been merged, we don't want to release version 3.5.0 with a known correctness bug. We've quickly cut RC5, and we welcome you to continue assisting with the testing. Ian Manning

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

Yes, SPARK-44805 has been included. For the commits from RC4 to RC5, please refer to https://github.com/apache/spark/commits/v3.5.0-rc5. Mich Talebzadeh 于2023年9月9日周六 08:09写道： > Apologies that should read ... release 3.5.0 (RC4) plus .. > > Mich Talebzadeh, > Distinguished Technologist,

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

Apologies that should read ... release 3.5.0 (RC4) plus .. Mich Talebzadeh, Distinguished Technologist, Solutions Architect & Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

Hi, Can you please confirm that this cut is release 3.4.0 plus the resolved Jira https://issues.apache.org/jira/browse/SPARK-44805 which was already fixed yesterday? Nothing else I believe? Thanks Mich view my Linkedin profile

[VOTE] Release Apache Spark 3.5.0 (RC5)

Please vote on releasing the following candidate(RC5) as Apache Spark version 3.5.0. The vote is open until 11:59pm Pacific time Sep 11th and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.5.0 [ ] -1 Do not release this

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

This issue is not a regression and yet we fail the vote? Couldn't this issue have been fixed in 3.5.1? Sorry I am new, so maybe this is how it works? On Sat, 9 Sep 2023, 02:29 Dongjoon Hyun, wrote: > Sorry but I'm -1 because there exists a late-arrival correctness patch > although it's not a

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

@Dongjoon Hyun Thank you for reporting this and for your prompt response. The vote has failed. I'll cut RC5 tonight, PST time. Dongjoon Hyun 于2023年9月8日周五 15:57写道： > Sorry but I'm -1 because there exists a late-arrival correctness patch > although it's not a regression. > > -

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1 Thank you for driving the release! On Fri, Sep 8, 2023 at 10:12 AM Jungtaek Lim wrote: > +1 (non-binding) > > Thanks for driving this release! > > On Fri, Sep 8, 2023 at 11:29 AM Holden Karau wrote: > >> +1 pip installing seems to function :) >> >> On Thu, Sep 7, 2023 at 7:22 PM Yuming

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

Sorry but I'm -1 because there exists a late-arrival correctness patch although it's not a regression. - https://issues.apache.org/jira/browse/SPARK-44805 "Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true" - https://github.com/apache/spark/pull/42850 -

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1 (non-binding) Thanks for driving this release! On Fri, Sep 8, 2023 at 11:29 AM Holden Karau wrote: > +1 pip installing seems to function :) > > On Thu, Sep 7, 2023 at 7:22 PM Yuming Wang wrote: > >> +1. >> >> On Thu, Sep 7, 2023 at 10:33 PM yangjie01 >> wrote: >> >>> +1 >>> >>> >>> >>>

Re: Elasticsearch support for Spark 3.x

@Alfie Davidson : Awesome, it worked with "“org.elasticsearch.spark.sql”" But as soon as I switched to *elasticsearch-spark-20_2.12, *"es" also worked. On Fri, Sep 8, 2023 at 12:45 PM Dipayan Dev wrote: > > Let me try that and get back. Just wondering, if there a change in the > way we pass

Re: Elasticsearch support for Spark 3.x

Let me try that and get back. Just wondering, if there a change in the way we pass the format in connector from Spark 2 to 3? On Fri, 8 Sep 2023 at 12:35 PM, Alfie Davidson wrote: > I am pretty certain you need to change the write.format from “es” to > “org.elasticsearch.spark.sql” > > Sent

Re: Elasticsearch support for Spark 3.x

I am pretty certain you need to change the write.format from “es” to “org.elasticsearch.spark.sql”Sent from my iPhoneOn 8 Sep 2023, at 03:10, Dipayan Dev wrote:++ DevOn Thu, 7 Sep 2023 at 10:22 PM, Dipayan Dev wrote:Hi, Can you please elaborate your last response? I

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1 pip installing seems to function :) On Thu, Sep 7, 2023 at 7:22 PM Yuming Wang wrote: > +1. > > On Thu, Sep 7, 2023 at 10:33 PM yangjie01 > wrote: > >> +1 >> >> >> >> *发件人**: *Gengliang Wang >> *日期**: *2023年9月7日 星期四 12:53 >> *收件人**: *Yuanjian Li >> *抄送**: *Xiao Li ,

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1. On Thu, Sep 7, 2023 at 10:33 PM yangjie01 wrote: > +1 > > > > *发件人**: *Gengliang Wang > *日期**: *2023年9月7日 星期四 12:53 > *收件人**: *Yuanjian Li > *抄送**: *Xiao Li , "her...@databricks.com.invalid" > , Spark dev list > *主题**: *Re: [VOTE] Release Apache Spark 3.5.0 (RC4) > > > > +1 > > > > On

Re: Making spark plan UI interactive

I really appreciate the idea. Another inspiration could be Datadog with its line graph and run logs below. Any way to graphically understand the application breakpoint can be great. Em qua., 6 de set. de 2023 08:04, Santosh Pingale escreveu: > Hey community > > Spark UI with the plan

Re: Elasticsearch support for Spark 3.x

++ Dev On Thu, 7 Sep 2023 at 10:22 PM, Dipayan Dev wrote: > Hi, > > Can you please elaborate your last response? I don’t have any external > dependencies added, and just updated the Spark version as mentioned below. > > Can someone help me with this? > > On Fri, 1 Sep 2023 at 5:58 PM, Koert

<    5   6   7   8   9   10   11   12   13   14   >