Re: [ANNOUNCE] Apache Spark 3.2.3 released

Thanks, Chao! 发件人: Maxim Gekk 日期: 2022年11月30日 星期三 19:40 收件人: Jungtaek Lim 抄送: Wenchen Fan , Chao Sun , dev , user 主题: Re: [ANNOUNCE] Apache Spark 3.2.3 released Thank you, Chao! On Wed, Nov 30, 2022 at 12:42 PM Jungtaek Lim mailto:kabhwan.opensou...@gmail.com>> wrote: Thanks Chao for

Re: [ANNOUNCE] Apache Spark 3.2.3 released

Thank you, Chao! On Wed, Nov 30, 2022 at 8:16 AM Yang,Jie(INF) wrote: > Thanks, Chao! > > > > *发件人**: *Maxim Gekk > *日期**: *2022年11月30日 星期三 19:40 > *收件人**: *Jungtaek Lim > *抄送**: *Wenchen Fan , Chao Sun , > dev , user > *主题**: *Re: [ANNOUNCE] Apache Spark 3.2.3 released > > > > Thank you,

Re: [ANNOUNCE] Apache Spark 3.2.3 released

Thanks Chao for driving the release! On Wed, Nov 30, 2022 at 9:24 AM Dongjoon Hyun wrote: > Thank you, Chao! > > On Wed, Nov 30, 2022 at 8:16 AM Yang,Jie(INF) wrote: > >> Thanks, Chao! >> >> >> >> *发件人**: *Maxim Gekk >> *日期**: *2022年11月30日 星期三 19:40 >> *收件人**: *Jungtaek Lim >> *抄送**:

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

Thanks for all the clarifications and details Jerry, Jungtaek :-) This looks like an exciting improvement to Structured Streaming - looking forward to it becoming part of Apache Spark ! Regards, Mridul On Mon, Nov 28, 2022 at 8:40 PM Jerry Peng wrote: > Hi all, > > I will add my two cents.

Re: [ANNOUNCE] Apache Spark 3.2.3 released

Thanks, Chao! On Wed, Nov 30, 2022 at 1:33 AM Chao Sun wrote: > We are happy to announce the availability of Apache Spark 3.2.3! > > Spark 3.2.3 is a maintenance release containing stability fixes. This > release is based on the branch-3.2 maintenance branch of Spark. We strongly > recommend

Contributions needed: 4 higher order functions

Hi all, There are four higher order functions in our backlog: - https://issues.apache.org/jira/browse/SPARK-41235 - https://issues.apache.org/jira/browse/SPARK-41234 - https://issues.apache.org/jira/browse/SPARK-41233 - https://issues.apache.org/jira/browse/SPARK-41232 Would be a great chance

Re: [ANNOUNCE] Apache Spark 3.2.3 released

Thanks Chao for driving the release! On Wed, Nov 30, 2022 at 6:03 PM Wenchen Fan wrote: > Thanks, Chao! > > On Wed, Nov 30, 2022 at 1:33 AM Chao Sun wrote: > >> We are happy to announce the availability of Apache Spark 3.2.3! >> >> Spark 3.2.3 is a maintenance release containing stability

[VOTE][SPIP] Asynchronous Offset Management in Structured Streaming

Hi all, I'd like to start the vote for SPIP: Asynchronous Offset Management in Structured Streaming. The high level summary of the SPIP is that we propose a couple of improvements on offset management in microbatch execution to lower down processing latency, which would help for certain types of

Re: [VOTE][SPIP] Asynchronous Offset Management in Structured Streaming

+1 On Wed, Nov 30, 2022 at 5:59 PM Jungtaek Lim wrote: > Starting with +1 from me. > > On Thu, Dec 1, 2022 at 10:54 AM Jungtaek Lim > wrote: > >> Hi all, >> >> I'd like to start the vote for SPIP: Asynchronous Offset Management in >> Structured Streaming. >> >> The high level summary of the

Re: [VOTE][SPIP] Asynchronous Offset Management in Structured Streaming

+1 Regards, Mridul On Wed, Nov 30, 2022 at 8:55 PM Xingbo Jiang wrote: > +1 > > On Wed, Nov 30, 2022 at 5:59 PM Jungtaek Lim > wrote: > >> Starting with +1 from me. >> >> On Thu, Dec 1, 2022 at 10:54 AM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Hi all, >>> >>> I'd like to

Re: [VOTE][SPIP] Asynchronous Offset Management in Structured Streaming

+1 On Thu, 1 Dec 2022 at 12:39, Mridul Muralidharan wrote: > > +1 > > Regards, > Mridul > > On Wed, Nov 30, 2022 at 8:55 PM Xingbo Jiang > wrote: > >> +1 >> >> On Wed, Nov 30, 2022 at 5:59 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Starting with +1 from me. >>> >>> On

Re: [VOTE][SPIP] Asynchronous Offset Management in Structured Streaming

+1 On Wed, Nov 30, 2022 at 8:04 PM Hyukjin Kwon wrote: > +1 > > On Thu, 1 Dec 2022 at 12:39, Mridul Muralidharan wrote: > >> >> +1 >> >> Regards, >> Mridul >> >> On Wed, Nov 30, 2022 at 8:55 PM Xingbo Jiang >> wrote: >> >>> +1 >>> >>> On Wed, Nov 30, 2022 at 5:59 PM Jungtaek Lim < >>>

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

+1 On Thu, 1 Dec 2022 at 08:10, Shixiong Zhu wrote: > +1 > > This is exciting. I agree with Jerry that this SPIP and continuous > processing are orthogonal. This SPIP itself would be a great improvement > and impact most Structured Streaming users. > > Best Regards, > Shixiong > > > On Wed, Nov

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

Thanks all for the support! Great to see we drive the discussion for Structured Streaming and have sufficient support. We would like to move forward with the vote thread. Please also participate in the vote. Thanks again! On Thu, Dec 1, 2022 at 10:04 AM Wenchen Fan wrote: > +1 to improve the

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

+1 to improve the widely used micro-batch mode first. On Thu, Dec 1, 2022 at 8:49 AM Hyukjin Kwon wrote: > +1 > > On Thu, 1 Dec 2022 at 08:10, Shixiong Zhu wrote: > >> +1 >> >> This is exciting. I agree with Jerry that this SPIP and continuous >> processing are orthogonal. This SPIP itself

Re: [VOTE][SPIP] Asynchronous Offset Management in Structured Streaming

Starting with +1 from me. On Thu, Dec 1, 2022 at 10:54 AM Jungtaek Lim wrote: > Hi all, > > I'd like to start the vote for SPIP: Asynchronous Offset Management in > Structured Streaming. > > The high level summary of the SPIP is that we propose a couple of > improvements on offset management in

Re: [ANNOUNCE] Apache Spark 3.2.3 released

Thanks, Chao! On Wed, Nov 30, 2022 at 9:58 AM huaxin gao wrote: > > Thanks Chao for driving the release! > > On Wed, Nov 30, 2022 at 9:24 AM Dongjoon Hyun wrote: >> >> Thank you, Chao! >> >> On Wed, Nov 30, 2022 at 8:16 AM Yang,Jie(INF) wrote: >>> >>> Thanks, Chao! >>> >>> >>> >>> 发件人: Maxim

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

+1 This is exciting. I agree with Jerry that this SPIP and continuous processing are orthogonal. This SPIP itself would be a great improvement and impact most Structured Streaming users. Best Regards, Shixiong On Wed, Nov 30, 2022 at 6:57 AM Mridul Muralidharan wrote: > > Thanks for all the

Re: Syndicate Apache Spark Twitter to Mastodon?

Hello, Does any long-term statistics about number of developers who moved to mastodon and activity use exists? I believe the most devs are still using Twitter. чт, 1 дек. 2022 г., 01:35 Holden Karau : > Do we want to start syndicating Apache Spark Twitter to a Mastodon > instance. It seems

Re: Syndicate Apache Spark Twitter to Mastodon?

My personal opinion, one of the most features of Twiiter that it is not federated and is good platform for annonces and so on. So it means "it would be good to reach our users where they are" means stay in twitter(most companies who use Spark/Databricks are in Twitter) For Federated features, I

Re: Syndicate Apache Spark Twitter to Mastodon?

I agree that there is probably a majority still on twitter, but it would be a syndication (e.g. we'd keep both). As to the # of devs it's hard to say since: 1) It's a federated service 2) Figuring out if an account is a dev or not is hard But, for example, There seems to be roughly an aggregate

Syndicate Apache Spark Twitter to Mastodon?

Do we want to start syndicating Apache Spark Twitter to a Mastodon instance. It seems like a lot of software dev folks are moving over there and it would be good to reach our users where they are. Any objections / concerns? Any thoughts on which server we should pick if we do this? -- Twitter:

Re: [ANNOUNCE] Apache Spark 3.2.3 released

Thank you, Chao! On Wed, Nov 30, 2022 at 12:42 PM Jungtaek Lim wrote: > Thanks Chao for driving the release! > > On Wed, Nov 30, 2022 at 6:03 PM Wenchen Fan wrote: > >> Thanks, Chao! >> >> On Wed, Nov 30, 2022 at 1:33 AM Chao Sun wrote: >> >>> We are happy to announce the availability of

Custom resolution rules that grow query plans

Hello, I wish to write a custom logical plan rule that modifies the output schema and grows the logical plan. The purpose of the rule is roughly to apply a projection on top of DatasourceV2Relation depending on some condition: case class MyRule extends Rule[LogicalPlan] { override def