[jira] [Created] (FLINK-35690) Release Testing: Verify FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-35690:
-

 Summary: Release Testing: Verify FLIP-459: Support Flink hybrid 
shuffle integration with Apache Celeborn
 Key: FLINK-35690
 URL: https://issues.apache.org/jira/browse/FLINK-35690
 Project: Flink
  Issue Type: Sub-task
Reporter: Yuxin Tan
 Fix For: 1.20.0


Follow up the test for https://issues.apache.org/jira/browse/FLINK-35533

In Flink 1.20,  we proposed integrating Flink's Hybrid Shuffle with Apache 
Celeborn through a pluggable remote tier interface. To verify this feature, you 
should reference these main two steps.

1. Implement Celeborn tier.
 * Implement a new tier factory and tier for Celeborn, including these APIs, 
including TierFactory/TierMasterAgent/TierProducerAgent/TierConsumerAgent.
 * The implementations should support granular data management at the Segment 
level for both client and server sides.

2. Use the implemented tier to shuffle data.
 * Compile Flink and Celeborn.
 * Deploy Celeborn service
 ** Deploy a new Celeborn service with the new compiled packages. You can 
reference the doc (https://celeborn.apache.org/docs/latest/) to deploy the 
cluster. 
 * Add the compiled flink plugin jar (celeborn-client-flink-xxx.jar) to Flink 
classpaths.
 * Configure the options to enable the feature.
 ** Configure the option 
taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class to the 
new Celeborn tier classes. Except for this option, the following options should 
also be added.

 
{code:java}
execution.batch-shuffle-mode: ALL_EXCHANGES_HYBRID_FULL 
celeborn.master.endpoints: 
celeborn.client.shuffle.partition.type: MAP\{code}
 * Run some test examples(e.g., WordCount) to verify the feature.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35658) Hybrid shuffle external tier can not work with UnknownInputChannel

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-35658:
-

 Summary: Hybrid shuffle external tier can not work with 
UnknownInputChannel
 Key: FLINK-35658
 URL: https://issues.apache.org/jira/browse/FLINK-35658
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.20.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


Currently, the hybrid shuffle can not work with UnknownInputChannel, we should 
fix it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi, Venkatakrishnan

The current proposal is designed to initialize a single remote
tier, such as Celeborn, if the remote tier factory is configured.
However, this is a temporary approach because the tier interfaces
are not stable. Our aim is to eventually support multiple storage
tiers: memory tier, disk tier, and remote tier. The remote tier
could be cloud storage or a remote shuffle service like Celeborn.

Initially, we don't expect to need both a cloud storage and a
remote shuffle service as two remote tiers concurrently due to
their large capacities. However, we're open and receptive to
adding both tiers concurrently if future discussions suggest it's
necessary. The architecture is designed for flexibility, allowing
each tier to function independently and be easily added to the
tiered storage with minimal modifications.

Best,
Yuxin


Venkatakrishnan Sowrirajan  于2024年6月17日周一 15:30写道：

> Yuxin,
>
> One question, in the current proposal is it limited to only one tiered
> storage implementation for eg: celeborn? Is it possible to have multiple
> tiered storages like a separate cloud storage and Celeborn or some such?
>
>
> On Thu, Jun 6, 2024, 9:46 AM Jeyhun Karimov  wrote:
>
> > Hi Yuxin,
> >
> > +1 for this proposal.
> > This change will greatly alleviate the pressure on local storage
> resources
> > (especially when there is limited local storage)
> > particularly in the context of cloud-native environments.
> >
> > Regards,
> > Jeyhun
> >
> > On Thu, Jun 6, 2024 at 1:20 PM Yuxin Tan  wrote:
> >
> > > Hi all,
> > >
> > > Thanks for all the feedback and suggestions so far.
> > >
> > > If there is no further comment, we will open the voting thread
> tomorrow.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Yuxin Tan  于2024年6月6日周四 15:40写道：
> > >
> > > > Thanks Zhu for the suggestion.
> > > > I have updated the description of the option.
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Zhu Zhu  于2024年6月6日周四 14:59写道：
> > > >
> > > >> +1
> > > >>
> > > >> Maybe explain in the description of
> > > >>
> > `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
> > > >> that it only accepts Celeborn as the remote shuffle tier at this
> > moment?
> > > >>
> > > >> Thanks,
> > > >> Zhu
> > > >>
> > > >> Junrui Lee  于2024年6月6日周四 13:49写道：
> > > >>
> > > >> > Thanks Yuxin for your answer. +1 for this proposal.
> > > >> >
> > > >> > Best,
> > > >> > Junrui.
> > > >> >
> > > >> > Yuxin Tan  于2024年6月6日周四 13:42写道：
> > > >> >
> > > >> > > Thanks Junrui for your question.
> > > >> > >
> > > >> > > > I wonder if the current interface design support the
> > > >> > > future adaptation for batch job recovery
> > > >> > >
> > > >> > > I noticed that FLIP-383 supports batch job recovery by
> introducing
> > > >> > > some new APIs. These APIs can also be added to the Tier-related
> > > >> > > interfaces to facilitate the feature. Since these modifications
> > are
> > > >> not
> > > >> > > directly related to the current integration tasks and the
> > > integration
> > > >> > > does not conflict with the batch job recovery, I propose that
> this
> > > >> FLIP
> > > >> > > doesn't involve these particular changes. Moreover, considering
> > that
> > > >> > > the Tier interfaces are not public currently, it is also
> feasible
> > to
> > > >> add
> > > >> > > the interfaces directly if necessary.
> > > >> > > WDYT?
> > > >> > >
> > > >> > > Best,
> > > >> > > Yuxin
> > > >> > >
> > > >> > >
> > > >> > > Junrui Lee  于2024年6月6日周四 11:02写道：
> > > >> > >
> > > >> > > > Thanks Yuxin for driving this proposal!
> > > >> > > >
> > > >> > > > I have a question about the public interface compatibility in
> > the
> > > >> > context
> > > >> > >

[RESULT][VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi, all

I am happy to say that FLIP-459 Support Flink
hybrid shuffle integration with Apache Celeborn [1]
has been accepted.

There are 11 votes, of which 4 are binding[2].

Xintong Song (binding)
Zhu Zhu (binding)
weijie guo (binding)
Jim Hughes (non-binding)
Jeyhun Karimov (non-binding)
Ahmed Hamdy (non-binding)
Venkatakrishnan Sowrirajan (non-binding)
Junrui Lee (non-binding)
Muhammet Orazov (non-binding)
Rui Fan (binding)
Yuepeng Pan (non-binding)

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
[2] https://lists.apache.org/thread/l64ykk3n8c2gc40gjbowt0ozs0x0jmqm

Best,
Yuxin

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi all,

Thanks for all your votes, I hereby close the vote and I'll announce
the results in a separate email.

Best,
Yuxin


Zhu Zhu  于2024年6月12日周三 11:15写道：

> +1 (binding)
>
> Thanks,
> Zhu
>
> Yuepeng Pan  于2024年6月11日周二 17:04写道：
>
> > +1 (non-binding)
> >
> > Best regards,
> > Yuepeng Pan
> >
> > At 2024-06-11 16:34:12, "Rui Fan" <1996fan...@gmail.com> wrote:
> > >+1（binding）
> > >
> > >Best,
> > >Rui
> > >
> > >On Tue, Jun 11, 2024 at 4:14 PM Muhammet Orazov
> > > wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> Thanks Yuxin for driving this!
> > >>
> > >> Best,
> > >> Muhammet
> > >>
> > >>
> > >> On 2024-06-07 08:02, Yuxin Tan wrote:
> > >> > Hi everyone,
> > >> >
> > >> > Thanks for all the feedback about the FLIP-459 Support Flink
> > >> > hybrid shuffle integration with Apache Celeborn[1].
> > >> > The discussion thread is here [2].
> > >> >
> > >> > I'd like to start a vote for it. The vote will be open for at least
> > >> > 72 hours unless there is an objection or insufficient votes.
> > >> >
> > >> > [1]
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > >> > [2]
> https://lists.apache.org/thread/gy7sm7qrf7yrv1rl5f4vtk5fo463ts33
> > >> >
> > >> > Best,
> > >> > Yuxin
> > >>
> >
>

[VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi everyone,

Thanks for all the feedback about the FLIP-459 Support Flink
hybrid shuffle integration with Apache Celeborn[1].
The discussion thread is here [2].

I'd like to start a vote for it. The vote will be open for at least
72 hours unless there is an objection or insufficient votes.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
[2] https://lists.apache.org/thread/gy7sm7qrf7yrv1rl5f4vtk5fo463ts33

Best,
Yuxin

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi all,

Thanks for all the feedback and suggestions so far.

If there is no further comment, we will open the voting thread tomorrow.

Best,
Yuxin


Yuxin Tan  于2024年6月6日周四 15:40写道：

> Thanks Zhu for the suggestion.
> I have updated the description of the option.
>
> Best,
> Yuxin
>
>
> Zhu Zhu  于2024年6月6日周四 14:59写道：
>
>> +1
>>
>> Maybe explain in the description of
>> `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
>> that it only accepts Celeborn as the remote shuffle tier at this moment?
>>
>> Thanks,
>> Zhu
>>
>> Junrui Lee  于2024年6月6日周四 13:49写道：
>>
>> > Thanks Yuxin for your answer. +1 for this proposal.
>> >
>> > Best,
>> > Junrui.
>> >
>> > Yuxin Tan  于2024年6月6日周四 13:42写道：
>> >
>> > > Thanks Junrui for your question.
>> > >
>> > > > I wonder if the current interface design support the
>> > > future adaptation for batch job recovery
>> > >
>> > > I noticed that FLIP-383 supports batch job recovery by introducing
>> > > some new APIs. These APIs can also be added to the Tier-related
>> > > interfaces to facilitate the feature. Since these modifications are
>> not
>> > > directly related to the current integration tasks and the integration
>> > > does not conflict with the batch job recovery, I propose that this
>> FLIP
>> > > doesn't involve these particular changes. Moreover, considering that
>> > > the Tier interfaces are not public currently, it is also feasible to
>> add
>> > > the interfaces directly if necessary.
>> > > WDYT?
>> > >
>> > > Best,
>> > > Yuxin
>> > >
>> > >
>> > > Junrui Lee  于2024年6月6日周四 11:02写道：
>> > >
>> > > > Thanks Yuxin for driving this proposal!
>> > > >
>> > > > I have a question about the public interface compatibility in the
>> > context
>> > > > of FLIP-459. As we've supported batch job recovery from jobMaster
>> > > failures
>> > > > in FLIP-383 which will be released in Flink 1.20. I wonder if the
>> > current
>> > > > interface design support the future adaptation for batch job
>> recovery?
>> > > >
>> > > > Looking forward to your feedback.
>> > > >
>> > > > Best,
>> > > > Junrui.
>> > > >
>> > > > weijie guo  于2024年6月5日周三 10:13写道：
>> > > >
>> > > > > Thanks Yuxin for the proposal!
>> > > > >
>> > > > > When we first proposed Hybrid Shuffle, I wanted to support
>> pluggable
>> > > > > storage tier in the future. However, limited by the architecture
>> of
>> > the
>> > > > > legacy Hybrid Shuffle at that time, this idea has not been
>> realized.
>> > > The
>> > > > > new architecture abstracts the tier nicely, and now it's time to
>> > > > introduce
>> > > > > support for external storage.
>> > > > >
>> > > > > Big +1 for this one!
>> > > > >
>> > > > > Best regards,
>> > > > >
>> > > > > Weijie
>> > > > >
>> > > > >
>> > > > > rexxiong  于2024年6月5日周三 00:08写道：
>> > > > >
>> > > > > > Thanks Yuxin for the proposal. +1,  as a member of the Apache
>> > > Celeborn
>> > > > > > community, I am very excited about the integration of Flink's
>> > Hybrid
>> > > > > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks
>> good
>> > to
>> > > > > me. I
>> > > > > > am looking forward to this integration.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Jiashu Xiong
>> > > > > >
>> > > > > > Ethan Feng  于2024年6月4日周二 16:47写道：
>> > > > > >
>> > > > > > > +1 for this proposal.
>> > > > > > >
>> > > > > > > After internally reviewing the prototype of CIP-6, this would
>> > > improve
>> > > > > > > performance and stability for Flink users using Celeborn.
>> > > > > > >
>> > > > > > > Expect to see this feature 

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Thanks Zhu for the suggestion.
I have updated the description of the option.

Best,
Yuxin


Zhu Zhu  于2024年6月6日周四 14:59写道：

> +1
>
> Maybe explain in the description of
> `taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class`
> that it only accepts Celeborn as the remote shuffle tier at this moment?
>
> Thanks,
> Zhu
>
> Junrui Lee  于2024年6月6日周四 13:49写道：
>
> > Thanks Yuxin for your answer. +1 for this proposal.
> >
> > Best,
> > Junrui.
> >
> > Yuxin Tan  于2024年6月6日周四 13:42写道：
> >
> > > Thanks Junrui for your question.
> > >
> > > > I wonder if the current interface design support the
> > > future adaptation for batch job recovery
> > >
> > > I noticed that FLIP-383 supports batch job recovery by introducing
> > > some new APIs. These APIs can also be added to the Tier-related
> > > interfaces to facilitate the feature. Since these modifications are not
> > > directly related to the current integration tasks and the integration
> > > does not conflict with the batch job recovery, I propose that this FLIP
> > > doesn't involve these particular changes. Moreover, considering that
> > > the Tier interfaces are not public currently, it is also feasible to
> add
> > > the interfaces directly if necessary.
> > > WDYT?
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Junrui Lee  于2024年6月6日周四 11:02写道：
> > >
> > > > Thanks Yuxin for driving this proposal!
> > > >
> > > > I have a question about the public interface compatibility in the
> > context
> > > > of FLIP-459. As we've supported batch job recovery from jobMaster
> > > failures
> > > > in FLIP-383 which will be released in Flink 1.20. I wonder if the
> > current
> > > > interface design support the future adaptation for batch job
> recovery?
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > > Best,
> > > > Junrui.
> > > >
> > > > weijie guo  于2024年6月5日周三 10:13写道：
> > > >
> > > > > Thanks Yuxin for the proposal!
> > > > >
> > > > > When we first proposed Hybrid Shuffle, I wanted to support
> pluggable
> > > > > storage tier in the future. However, limited by the architecture of
> > the
> > > > > legacy Hybrid Shuffle at that time, this idea has not been
> realized.
> > > The
> > > > > new architecture abstracts the tier nicely, and now it's time to
> > > > introduce
> > > > > support for external storage.
> > > > >
> > > > > Big +1 for this one!
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Weijie
> > > > >
> > > > >
> > > > > rexxiong  于2024年6月5日周三 00:08写道：
> > > > >
> > > > > > Thanks Yuxin for the proposal. +1,  as a member of the Apache
> > > Celeborn
> > > > > > community, I am very excited about the integration of Flink's
> > Hybrid
> > > > > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks
> good
> > to
> > > > > me. I
> > > > > > am looking forward to this integration.
> > > > > >
> > > > > > Thanks,
> > > > > > Jiashu Xiong
> > > > > >
> > > > > > Ethan Feng  于2024年6月4日周二 16:47写道：
> > > > > >
> > > > > > > +1 for this proposal.
> > > > > > >
> > > > > > > After internally reviewing the prototype of CIP-6, this would
> > > improve
> > > > > > > performance and stability for Flink users using Celeborn.
> > > > > > >
> > > > > > > Expect to see this feature come out to the community.
> > > > > > >
> > > > > > > As I come from the Celeborn community, I hope more users can
> try
> > to
> > > > > > > use Celeborn when there are Flink batch jobs.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Ethan Feng
> > > > > > >
> > > > > > > Yuxin Tan  于2024年6月4日周二 16:34写道：
> > > > > > > >
> > > > > > > > Hi, Venkatakrishnan,
> > > > > > > >
> > > > > > > > Th

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Thanks Junrui for your question.

> I wonder if the current interface design support the
future adaptation for batch job recovery

I noticed that FLIP-383 supports batch job recovery by introducing
some new APIs. These APIs can also be added to the Tier-related
interfaces to facilitate the feature. Since these modifications are not
directly related to the current integration tasks and the integration
does not conflict with the batch job recovery, I propose that this FLIP
doesn't involve these particular changes. Moreover, considering that
the Tier interfaces are not public currently, it is also feasible to add
the interfaces directly if necessary.
WDYT?

Best,
Yuxin


Junrui Lee  于2024年6月6日周四 11:02写道：

> Thanks Yuxin for driving this proposal!
>
> I have a question about the public interface compatibility in the context
> of FLIP-459. As we've supported batch job recovery from jobMaster failures
> in FLIP-383 which will be released in Flink 1.20. I wonder if the current
> interface design support the future adaptation for batch job recovery?
>
> Looking forward to your feedback.
>
> Best,
> Junrui.
>
> weijie guo  于2024年6月5日周三 10:13写道：
>
> > Thanks Yuxin for the proposal!
> >
> > When we first proposed Hybrid Shuffle, I wanted to support pluggable
> > storage tier in the future. However, limited by the architecture of the
> > legacy Hybrid Shuffle at that time, this idea has not been realized. The
> > new architecture abstracts the tier nicely, and now it's time to
> introduce
> > support for external storage.
> >
> > Big +1 for this one!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > rexxiong  于2024年6月5日周三 00:08写道：
> >
> > > Thanks Yuxin for the proposal. +1,  as a member of the Apache Celeborn
> > > community, I am very excited about the integration of Flink's Hybrid
> > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks good to
> > me. I
> > > am looking forward to this integration.
> > >
> > > Thanks,
> > > Jiashu Xiong
> > >
> > > Ethan Feng  于2024年6月4日周二 16:47写道：
> > >
> > > > +1 for this proposal.
> > > >
> > > > After internally reviewing the prototype of CIP-6, this would improve
> > > > performance and stability for Flink users using Celeborn.
> > > >
> > > > Expect to see this feature come out to the community.
> > > >
> > > > As I come from the Celeborn community, I hope more users can try to
> > > > use Celeborn when there are Flink batch jobs.
> > > >
> > > > Thanks,
> > > > Ethan Feng
> > > >
> > > > Yuxin Tan  于2024年6月4日周二 16:34写道：
> > > > >
> > > > > Hi, Venkatakrishnan,
> > > > >
> > > > > Thanks for joining the discussion. We appreciate your interest
> > > > > in contributing to the work. Once the FLIP and CIP proposals
> > > > > have been approved, we will create some JIRA tickets in Flink
> > > > > and Celeborn projects. Please feel free to take a look at the
> > > > > tickets and select any that resonate with your interests.
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > > >
> > > > > Venkatakrishnan Sowrirajan  于2024年5月31日周五
> 23:11写道：
> > > > >
> > > > > > Thanks for this FLIP. We are also interested in
> > learning/contributing
> > > > to
> > > > > > the hybrid shuffle integration with celeborn for batch
> executions.
> > > > > >
> > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan 
> > > > wrote:
> > > > > >
> > > > > > > Hi, Xintong,
> > > > > > >
> > > > > > > >  I think we can also publish the prototype codes so the
> > > > > > > community can better understand and help with it.
> > > > > > >
> > > > > > > Ok, I agree on the point. I will prepare and publish the code
> > > > > > > recently.
> > > > > > >
> > > > > > > Rui,
> > > > > > >
> > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > >
> > > > > > > Thanks for the reminder. I've updated the images.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > &g

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Yuxin Tan]

Congratulations, Rui!

Best,
Yuxin


Xuannan Su  于2024年6月6日周四 09:58写道：

> Congratulations!
>
> Best regards,
> Xuannan
>
> On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu  wrote:
> >
> > Congratulations, Rui !
> >
> > On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee 
> wrote:
> >
> > > Congratulations, Rui!
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > Lijie Wang  于2024年6月6日周四 09:11写道：
> > >
> > > > Congratulations, Rui!
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Rodrigo Meneses  于2024年6月5日周三 21:35写道：
> > > >
> > > > > All the best
> > > > >
> > > > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng 
> > > > wrote:
> > > > >
> > > > > > Congratulations, Rui!
> > > > > >
> > > > > > Regards,
> > > > > > Xiangyu Feng
> > > > > >
> > > > > > Feng Jin  于2024年6月5日周三 20:42写道：
> > > > > >
> > > > > > > Congratulations, Rui!
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng Jin
> > > > > > >
> > > > > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei  >
> > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Rui!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yanfei
> > > > > > > >
> > > > > > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > > > > > >
> > > > > > > > > Congrats, Rui!
> > > > > > > > >
> > > > > > > > > Luke
> > > > > > > > >
> > > > > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun <
> > > jiabao...@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congrats, Rui. Well-deserved!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Jiabao
> > > > > > > > > >
> > > > > > > > > > Zhanghao Chen  于2024年6月5日周三
> > > > 19:29写道：
> > > > > > > > > >
> > > > > > > > > > > Congrats, Rui!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Zhanghao Chen
> > > > > > > > > > > 
> > > > > > > > > > > From: Piotr Nowojski 
> > > > > > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > > > > > To: dev ; rui fan <
> > > > 1996fan...@gmail.com>
> > > > > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan
> Rui
> > > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > On behalf of the PMC, I'm very happy to announce
> another
> > > new
> > > > > > Apache
> > > > > > > > Flink
> > > > > > > > > > > PMC Member - Fan Rui.
> > > > > > > > > > >
> > > > > > > > > > > Rui has been active in the community since August 2019.
> > > > During
> > > > > > this
> > > > > > > > time
> > > > > > > > > > he
> > > > > > > > > > > has contributed a lot of new features. Among others:
> > > > > > > > > > >   - Decoupling Autoscaler from Kubernetes Operator, and
> > > > > > supporting
> > > > > > > > > > > Standalone Autoscaler
> > > > > > > > > > >   - Improvements to checkpointing, flamegraphs, restart
> > > > > > strategies,
> > > > > > > > > > > watermark alignment, network shuffles
> > > > > > > > > > >   - Optimizing the memory and CPU usage of large
> operators,
> > > > > > greatly
> > > > > > > > > > > reducing the risk and probability of TaskManager OOM
> > > > > > > > > > >
> > > > > > > > > > > He reviewed a significant amount of PRs and has been
> active
> > > > > both
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > mailing lists and in Jira helping to both maintain and
> grow
> > > > > > Apache
> > > > > > > > > > Flink's
> > > > > > > > > > > community. He is also our current Flink 1.20 release
> > > manager.
> > > > > > > > > > >
> > > > > > > > > > > In the last 12 months, Rui has been the most active
> > > > contributor
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > Flink Kubernetes Operator project, while being the 2nd
> most
> > > > > > active
> > > > > > > > Flink
> > > > > > > > > > > contributor at the same time.
> > > > > > > > > > >
> > > > > > > > > > > Please join me in welcoming and congratulating Fan Rui!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
>

[jira] [Created] (FLINK-35533) FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-35533:
-

 Summary: FLIP-459: Support Flink hybrid shuffle integration with 
Apache Celeborn
 Key: FLINK-35533
 URL: https://issues.apache.org/jira/browse/FLINK-35533
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Affects Versions: 1.20.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


This is the umbrella jira for 
[FLIP-459|https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi, Venkatakrishnan,

Thanks for joining the discussion. We appreciate your interest
in contributing to the work. Once the FLIP and CIP proposals
have been approved, we will create some JIRA tickets in Flink
and Celeborn projects. Please feel free to take a look at the
tickets and select any that resonate with your interests.

Best,
Yuxin


Venkatakrishnan Sowrirajan  于2024年5月31日周五 23:11写道：

> Thanks for this FLIP. We are also interested in learning/contributing to
> the hybrid shuffle integration with celeborn for batch executions.
>
> On Tue, May 28, 2024, 7:07 PM Yuxin Tan  wrote:
>
> > Hi, Xintong,
> >
> > >  I think we can also publish the prototype codes so the
> > community can better understand and help with it.
> >
> > Ok, I agree on the point. I will prepare and publish the code
> > recently.
> >
> > Rui,
> >
> > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> >
> > Thanks for the reminder. I've updated the images.
> >
> >
> > Best,
> > Yuxin
> >
> >
> > Rui Fan <1996fan...@gmail.com> 于2024年5月29日周三 09:33写道：
> >
> > > Thanks Yuxin for driving this proposal!
> > >
> > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > >
> > > [1]
> > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > >
> > > Best,
> > > Rui
> > >
> > > On Wed, May 29, 2024 at 9:03 AM Xintong Song 
> > > wrote:
> > >
> > > > +1 for this proposal.
> > > >
> > > > We have been prototyping this feature internally at Alibaba for a
> > couple
> > > of
> > > > months. Yuxin, I think we can also publish the prototype codes so the
> > > > community can better understand and help with it.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan 
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to start a discussion on FLIP-459 Support Flink hybrid
> > > > shuffle
> > > > > integration with
> > > > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions
> between
> > > > > memory, disk, and
> > > > > remote storage to improve performance and job stability.
> > Concurrently,
> > > > > Apache Celeborn
> > > > > provides a stable, performant, scalable remote shuffle service.
> This
> > > > > integration proposal is to
> > > > > harness the benefits from both hybrid shuffle and Celeborn
> > > > simultaneously.
> > > > >
> > > > > Note that this proposal has two parts.
> > > > > 1. The Flink-side modifications are in FLIP-459[1].
> > > > > 2. The Celeborn-side changes are in CIP-6[2].
> > > > >
> > > > > Looking forward to everyone's feedback and suggestions. Thank you!
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

[Yuxin Tan]

Congratulations, Weijie!

Best,
Yuxin


Yuepeng Pan  于2024年6月4日周二 14:57写道：

> Congratulations !
>
>
> Best,
> Yuepeng Pan
>
> At 2024-06-04 14:45:45, "Xintong Song"  wrote:
> >Hi everyone,
> >
> >On behalf of the PMC, I'm very happy to announce that Weijie Guo has
> joined
> >the Flink PMC!
> >
> >Weijie has been an active member of the Apache Flink community for many
> >years. He has made significant contributions in many components, including
> >runtime, shuffle, sdk, connectors, etc. He has driven / participated in
> >many FLIPs, authored and reviewed hundreds of PRs, been consistently
> active
> >on mailing lists, and also helped with release management of 1.20 and
> >several other bugfix releases.
> >
> >Congratulations and welcome Weijie!
> >
> >Best,
> >
> >Xintong (on behalf of the Flink PMC)
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi, Xintong,

>  I think we can also publish the prototype codes so the
community can better understand and help with it.

Ok, I agree on the point. I will prepare and publish the code
recently.

Rui,

> Kindly reminder: the image of CIP-6[1] cannot be loaded.

Thanks for the reminder. I've updated the images.


Best,
Yuxin


Rui Fan <1996fan...@gmail.com> 于2024年5月29日周三 09:33写道：

> Thanks Yuxin for driving this proposal!
>
> Kindly reminder: the image of CIP-6[1] cannot be loaded.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
>
> Best,
> Rui
>
> On Wed, May 29, 2024 at 9:03 AM Xintong Song 
> wrote:
>
> > +1 for this proposal.
> >
> > We have been prototyping this feature internally at Alibaba for a couple
> of
> > months. Yuxin, I think we can also publish the prototype codes so the
> > community can better understand and help with it.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan 
> wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a discussion on FLIP-459 Support Flink hybrid
> > shuffle
> > > integration with
> > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> > > memory, disk, and
> > > remote storage to improve performance and job stability. Concurrently,
> > > Apache Celeborn
> > > provides a stable, performant, scalable remote shuffle service. This
> > > integration proposal is to
> > > harness the benefits from both hybrid shuffle and Celeborn
> > simultaneously.
> > >
> > > Note that this proposal has two parts.
> > > 1. The Flink-side modifications are in FLIP-459[1].
> > > 2. The Celeborn-side changes are in CIP-6[2].
> > >
> > > Looking forward to everyone's feedback and suggestions. Thank you!
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > >
> > > Best,
> > > Yuxin
> > >
> >
>

[DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi all,

I would like to start a discussion on FLIP-459 Support Flink hybrid shuffle
integration with
Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
memory, disk, and
remote storage to improve performance and job stability. Concurrently,
Apache Celeborn
provides a stable, performant, scalable remote shuffle service. This
integration proposal is to
harness the benefits from both hybrid shuffle and Celeborn simultaneously.

Note that this proposal has two parts.
1. The Flink-side modifications are in FLIP-459[1].
2. The Celeborn-side changes are in CIP-6[2].

Looking forward to everyone's feedback and suggestions. Thank you!

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
[2]
https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn

Best,
Yuxin

Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0

[Yuxin Tan]

Thanks for the effort, Xuannan.

+1 for the proposal.

Best,
Yuxin


Xintong Song  于2024年4月29日周一 15:40写道：

> Thanks for driving this effort, Xuannan.
>
> +1 for the proposed changes.
>
> Just one suggestion: Some of the proposed changes involve not solely
> changing the configuration options, but are bound to changing / removal of
> certain features. E.g., the removal of hash-blocking shuffle and legacy
> hybrid shuffle mode, and the behavior change of overdraft network buffers.
> Therefore, it might be nicer to provide an implementation plan with a list
> of related tasks in the FLIP. This should not block the FLIP though.
>
> Best,
>
> Xintong
>
>
>
> On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su  wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion on FLIP-450: Improve Runtime
> > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have
> > revisited all runtime configurations and identified several
> > improvements to enhance user-friendliness and maintainability. In this
> > FLIP, we aim to refine the runtime configuration.
> >
> > Looking forward to everyone's feedback and suggestions. Thank you!
> >
> > Best regards,
> > Xuannan
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0
> >
>

[jira] [Created] (FLINK-35214) Update result partition id for remote input channel when unknown input channel is updated

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-35214:
-

 Summary: Update result partition id for remote input channel when 
unknown input channel is updated
 Key: FLINK-35214
 URL: https://issues.apache.org/jira/browse/FLINK-35214
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.20.0
Reporter: Yuxin Tan


In [FLINK-29768|https://issues.apache.org/jira/browse/FLINK-29768], the result 
partition in the local input channel has been updated to support speculation. 
It is necessary to similarly update the result partition ID in the remote input 
channel.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35169) Recycle buffers to freeSegments before releasing data buffer for sort accumulator

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-35169:
-

 Summary: Recycle buffers to freeSegments before releasing data 
buffer for sort accumulator
 Key: FLINK-35169
 URL: https://issues.apache.org/jira/browse/FLINK-35169
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.20.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


When using sortBufferAccumulator, we should recycle the buffers to freeSegments 
before releasing the data buffer. The reason is that when getting buffers from 
the DataBuffer, it may require more buffers than the current quantity available 
in freeSegments. Consequently, to ensure adequate buffers from DataBuffer, the 
flushed and recycled buffers should also be added to freeSegments for reuse.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Zakelly Lan  于2024年4月17日周三 16:51写道：

> +1 binding
>
>
> Best,
> Zakelly
>
> On Wed, Apr 17, 2024 at 2:05 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> > +1(binding)
> >
> > Best,
> > Rui
> >
> > On Wed, Apr 17, 2024 at 1:02 PM Xuannan Su 
> wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for all the feedback about the FLIP-442: General Improvement to
> > > Configuration for Flink 2.0 [1] [2].
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours(excluding weekends,until APR 22, 12:00AM GMT) unless there is an
> > > objection or an insufficient number of votes.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-442%3A+General+Improvement+to+Configuration+for+Flink+2.0
> > > [2] https://lists.apache.org/thread/15k0stwyoknhxvd643ctwjw3fd17pqwk
> > >
> > >
> > > Best regards,
> > > Xuannan
> > >
> >
>

[jira] [Created] (FLINK-35130) Simplify AvailabilityNotifierImpl to support speculative scheduler and improve performance

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-35130:
-

 Summary: Simplify AvailabilityNotifierImpl to support speculative 
scheduler and improve performance
 Key: FLINK-35130
 URL: https://issues.apache.org/jira/browse/FLINK-35130
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Affects Versions: 1.20.0
Reporter: Yuxin Tan


The AvailabilityNotifierImpl in SingleInputGate has maps storing the channel 
ids. But the map key is the result partition id, which will change according to 
the different attempt numbers when speculation is enabled.  This can be 
resolved by using `inputChannels` to get channel and the map key of 
inputChannels will not vary with the attempts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

[Yuxin Tan]

Congratulations, Jing!

Best,
Yuxin


Danny Cranmer  于2024年4月15日周一 20:26写道：

> Congrats Jing!
>
> Best Regards,
> Danny
>
> On Mon, Apr 15, 2024 at 11:51 AM Swapnal Varma 
> wrote:
>
> > Congratulations, Jing!
> >
> > Best,
> > Swapnal
> >
> > On Mon, 15 Apr 2024, 15:14 Jacky Lau,  wrote:
> >
> > > Congratulations, Jing!
> > >
> > > Best,
> > > Jacky Lau
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

[Yuxin Tan]

Congratulations, Lincoln!

Best,
Yuxin


Xuannan Su  于2024年4月16日周二 10:26写道：

> Congratulations, Lincoln!
>
> Best regards,
> Xuannan
>
> On Tue, Apr 16, 2024 at 10:04 AM Hang Ruan  wrote:
> >
> > Congratulations, Lincoln!
> >
> > Best,
> > Hang
> >
> > yh z  于2024年4月16日周二 09:14写道：
> >
> > > Congratulations, Lincoln!
> > >
> > > Best,
> > > Yunhong (Swuferhong)
> > >
> > >
> > > Swapnal Varma  于2024年4月15日周一 18:50写道：
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > Best,
> > > > Swapnal
> > > >
> > > >
> > > > On Mon, 15 Apr 2024, 15:16 Jacky Lau,  wrote:
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > Best,
> > > > > Jacky Lau
> > > > >
> > > > > Jinzhong Li  于2024年4月15日周一 15:45写道：
> > > > >
> > > > > > Congratulations, Lincoln!
> > > > > >
> > > > > > Best,
> > > > > > Jinzhong Li
> > > > > >
> > > > > > On Mon, Apr 15, 2024 at 2:56 PM Hangxiang Yu <
> master...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Congratulations, Lincoln!
> > > > > > >
> > > > > > > On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan <
> > > zakelly@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Lincoln!
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zakelly
> > > > > > > >
> > > > > > > > On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky
> > > > > >  > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations, Lincoln!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ferenc
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Friday, April 12th, 2024 at 15:54,
> > > > > lorenzo.affe...@ververica.com
> > > > > > > > .INVALID
> > > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Huge congrats! Well done!
> > > > > > > > > > On Apr 12, 2024 at 13:56 +0200, Ron liu
> ron9@gmail.com,
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Ron
> > > > > > > > > > >
> > > > > > > > > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > > > > > > > > >
> > > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Junrui
> > > > > > > > > > > >
> > > > > > > > > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五
> > > > 18:29写道：
> > > > > > > > > > > >
> > > > > > > > > > > > > > Congratulations, Lincoln!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best Regards
> > > > > > > > > > > > > > Aleksandr
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best,
> > > > > > > Hangxiang.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

[Yuxin Tan]

Congratulations, Zakelly!

Best,
Yuxin


Xuannan Su  于2024年4月16日周二 10:30写道：

> Congratulations Zakelly!
>
> Best regards,
> Xuannan
>
> On Mon, Apr 15, 2024 at 4:31 PM Jing Ge 
> wrote:
> >
> > Congratulations Zakelly!
> >
> > Best regards,
> > Jing
> >
> > On Mon, Apr 15, 2024 at 4:26 PM Xia Sun  wrote:
> >
> > > Congratulations Zakelly!
> > >
> > >  Best,
> > >  Xia
> > >
> > > Leonard Xu  于2024年4月15日周一 16:16写道：
> > >
> > > > Congratulations Zakelly!
> > > >
> > > >
> > > > Best,
> > > > Leonard
> > > > > 2024年4月15日 下午3:56，Samrat Deb  写道：
> > > > >
> > > > > Congratulations Zakelly!
> > > >
> > > >
> > >
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

[Yuxin Tan]

Congratulations! Thanks for the efforts.


Best,
Yuxin


Samrat Deb  于2024年3月21日周四 20:28写道：

> Congratulations !
>
> Bests
> Samrat
>
> On Thu, 21 Mar 2024 at 5:52 PM, Ahmed Hamdy  wrote:
>
> > Congratulations, great work and great news.
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Thu, 21 Mar 2024 at 11:41, Benchao Li  wrote:
> >
> > > Congratulations, and thanks for the great work!
> > >
> > > Yuan Mei  于2024年3月21日周四 18:31写道：
> > > >
> > > > Thanks for driving these efforts!
> > > >
> > > > Congratulations
> > > >
> > > > Best
> > > > Yuan
> > > >
> > > > On Thu, Mar 21, 2024 at 4:35 PM Yu Li  wrote:
> > > >
> > > > > Congratulations and look forward to its further development!
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > > On Thu, 21 Mar 2024 at 15:54, ConradJam 
> wrote:
> > > > > >
> > > > > > Congrattulations!
> > > > > >
> > > > > > Leonard Xu  于2024年3月20日周三 21:36写道：
> > > > > >
> > > > > > > Hi devs and users,
> > > > > > >
> > > > > > > We are thrilled to announce that the donation of Flink CDC as a
> > > > > > > sub-project of Apache Flink has completed. We invite you to
> > explore
> > > > > the new
> > > > > > > resources available:
> > > > > > >
> > > > > > > - GitHub Repository: https://github.com/apache/flink-cdc
> > > > > > > - Flink CDC Documentation:
> > > > > > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > > > > > >
> > > > > > > After Flink community accepted this donation[1], we have
> > completed
> > > > > > > software copyright signing, code repo migration, code cleanup,
> > > website
> > > > > > > migration, CI migration and github issues migration etc.
> > > > > > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > > > > Qingsheng
> > > > > > > Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> contributors
> > > for
> > > > > their
> > > > > > > contributions and help during this process！
> > > > > > >
> > > > > > >
> > > > > > > For all previous contributors: The contribution process has
> > > slightly
> > > > > > > changed to align with the main Flink project. To report bugs or
> > > > > suggest new
> > > > > > > features, please open tickets
> > > > > > > Apache Jira (https://issues.apache.org/jira).  Note that we
> will
> > > no
> > > > > > > longer accept GitHub issues for these purposes.
> > > > > > >
> > > > > > >
> > > > > > > Welcome to explore the new repository and documentation. Your
> > > feedback
> > > > > and
> > > > > > > contributions are invaluable as we continue to improve Flink
> CDC.
> > > > > > >
> > > > > > > Thanks everyone for your support and happy exploring Flink CDC!
> > > > > > >
> > > > > > > Best，
> > > > > > > Leonard
> > > > > > > [1]
> > > https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Best
> > > > > >
> > > > > > ConradJam
> > > > >
> > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
> >
>

[jira] [Created] (FLINK-34544) The release check bug in tiered memory manager of hybrid shuffle

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-34544:
-

 Summary: The release check bug in tiered memory manager of hybrid 
shuffle
 Key: FLINK-34544
 URL: https://issues.apache.org/jira/browse/FLINK-34544
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.19.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


The `TieredStorageMemoryManagerImpl` has a `numRequestedBuffers` check when 
releasing resources. However, this check is performed in the task thread, while 
the buffer recycle may occur in the Netty thread. As a result, it may 
incorrectly throw an exception when the release is too quick for the vertex, 
which has almost no data.
We should fix it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-331: Support EndOfStreamTrigger and isOutputOnlyAfterEndOfStream operator attribute to optimize task deployment

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Hang Ruan  于2024年2月5日周一 11:22写道：

>  +1 (non-binding)
>
> Best,
> Hang
>
> Dong Lin  于2024年2月5日周一 11:08写道：
>
> > Thanks for the FLIP.
> >
> > +1 (binding)
> >
> > Best,
> > Dong
> >
> > On Wed, Jan 31, 2024 at 11:41 AM Xuannan Su 
> wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for all the feedback about the FLIP-331: Support
> > > EndOfStreamTrigger and isOutputOnlyAfterEndOfStream operator attribute
> > > to optimize task deployment [1] [2].
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours(excluding weekends,until Feb 5, 12:00AM GMT) unless there is an
> > > objection or an insufficient number of votes.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-331%3A+Support+EndOfStreamTrigger+and+isOutputOnlyAfterEndOfStream+operator+attribute+to+optimize+task+deployment
> > > [2] https://lists.apache.org/thread/qq39rmg3f23ysx5m094s4c4cq0m4tdj5
> > >
> > >
> > > Best,
> > > Xuannan
> > >
> >
>

Re: [VOTE] Accept Flink CDC into Apache Flink

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Márton Balassi  于2024年1月9日周二 17:25写道：

> +1 (binding)
>
> On Tue, Jan 9, 2024 at 10:15 AM Leonard Xu  wrote:
>
> > +1(binding)
> >
> > Best,
> > Leonard
> >
> > > 2024年1月9日 下午5:08，Yangze Guo  写道：
> > >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Tue, Jan 9, 2024 at 5:06 PM Robert Metzger 
> > wrote:
> > >>
> > >> +1 (binding)
> > >>
> > >>
> > >> On Tue, Jan 9, 2024 at 9:54 AM Guowei Ma 
> wrote:
> > >>
> > >>> +1 (binding)
> > >>> Best,
> > >>> Guowei
> > >>>
> > >>>
> > >>> On Tue, Jan 9, 2024 at 4:49 PM Rui Fan <1996fan...@gmail.com> wrote:
> > >>>
> >  +1 (non-binding)
> > 
> >  Best,
> >  Rui
> > 
> >  On Tue, Jan 9, 2024 at 4:41 PM Hang Ruan 
> > wrote:
> > 
> > > +1 (non-binding)
> > >
> > > Best,
> > > Hang
> > >
> > > gongzhongqiang  于2024年1月9日周二 16:25写道：
> > >
> > >> +1 non-binding
> > >>
> > >> Best,
> > >> Zhongqiang
> > >>
> > >> Leonard Xu  于2024年1月9日周二 15:05写道：
> > >>
> > >>> Hello all,
> > >>>
> > >>> This is the official vote whether to accept the Flink CDC code
> > >> contribution
> > >>> to Apache Flink.
> > >>>
> > >>> The current Flink CDC code, documentation, and website can be
> > >>> found here:
> > >>> code: https://github.com/ververica/flink-cdc-connectors <
> > >>> https://github.com/ververica/flink-cdc-connectors>
> > >>> docs: https://ververica.github.io/flink-cdc-connectors/ <
> > >>> https://ververica.github.io/flink-cdc-connectors/>
> > >>>
> > >>> This vote should capture whether the Apache Flink community is
> > > interested
> > >>> in accepting, maintaining, and evolving Flink CDC.
> > >>>
> > >>> Regarding my original proposal[1] in the dev mailing list, I
> firmly
> > >> believe
> > >>> that this initiative aligns perfectly with Flink. For the Flink
> > >> community,
> > >>> it represents an opportunity to bolster Flink's competitive edge
> in
> > >>> streaming
> > >>> data integration, fostering the robust growth and prosperity of
> the
> > >> Apache
> > >>> Flink
> > >>> ecosystem. For the Flink CDC project, becoming a sub-project of
> >  Apache
> > >>> Flink
> > >>> means becoming an integral part of a neutral open-source
> community,
> > >>> capable of
> > >>> attracting a more diverse pool of contributors.
> > >>>
> > >>> All Flink CDC maintainers are dedicated to continuously
> > >>> contributing
> >  to
> > >>> achieve
> > >>> seamless integration with Flink. Additionally, PMC members like
> > >>> Jark,
> > >>> Qingsheng,
> > >>> and I are willing to infacilitate the expansion of contributors
> and
> > >>> committers to
> > >>> effectively maintain this new sub-project.
> > >>>
> > >>> This is a "Adoption of a new Codebase" vote as per the Flink
> bylaws
> > > [2].
> > >>> Only PMC votes are binding. The vote will be open at least 7 days
> > >>> (excluding weekends), meaning until Thursday January 18 12:00
> UTC,
> > >>> or
> > >>> until we
> > >>> achieve the 2/3rd majority. We will follow the instructions in
> the
> > > Flink
> > >>> Bylaws
> > >>> in the case of insufficient active binding voters:
> > >>>
> >  1. Wait until the minimum length of the voting passes.
> >  2. Publicly reach out via personal email to the remaining
> binding
> > >> voters
> > >>> in the
> > >>> voting mail thread for at least 2 attempts with at least 7 days
> >  between
> > >>> two attempts.
> >  3. If the binding voter being contacted still failed to respond
> >  after
> > >>> all the attempts,
> > >>> the binding voter will be considered as inactive for the purpose
> of
> > > this
> > >>> particular voting.
> > >>>
> > >>> Welcome voting !
> > >>>
> > >>> Best,
> > >>> Leonard
> > >>> [1]
> > >>> https://lists.apache.org/thread/o7klnbsotmmql999bnwmdgo56b6kxx9l
> > >>> [2]
> > >>>
> > >>
> > >
> > 
> > >>>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> > >>
> > >
> > 
> > >>>
> >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov

[Yuxin Tan]

Congratulations, Alex!

Best,
Yuxin


Lijie Wang  于2024年1月3日周三 14:04写道：

> Congratulations Alex !
>
> Best,
> Lijie
>
> Romit Mahanta  于2024年1月3日周三 13:41写道：
>
> > Happy New Year & congratulations Alex!
> >
> > Best,
> >
> > R
> >
> > On Tue, 2 Jan, 2024, 5:45 pm Maximilian Michels,  wrote:
> >
> > > Happy New Year everyone,
> > >
> > > I'd like to start the year off by announcing Alexander Fedulov as a
> > > new Flink committer.
> > >
> > > Alex has been active in the Flink community since 2019. He has
> > > contributed more than 100 commits to Flink, its Kubernetes operator,
> > > and various connectors [1][2].
> > >
> > > Especially noteworthy are his contributions on deprecating and
> > > migrating the old Source API functions and test harnesses, the
> > > enhancement to flame graphs, the dynamic rescale time computation in
> > > Flink Autoscaling, as well as all the small enhancements Alex has
> > > contributed which make a huge difference.
> > >
> > > Beyond code contributions, Alex has been an active community member
> > > with his activity on the mailing lists [3][4], as well as various
> > > talks and blog posts about Apache Flink [5][6].
> > >
> > > Congratulations Alex! The Flink community is proud to have you.
> > >
> > > Best,
> > > The Flink PMC
> > >
> > > [1]
> > >
> https://github.com/search?type=commits=author%3Aafedulov+org%3Aapache
> > > [2]
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-28229?jql=status%20in%20(Resolved%2C%20Closed)%20AND%20assignee%20in%20(afedulov)%20ORDER%20BY%20resolved%20DESC%2C%20created%20DESC
> > > [3]
> https://lists.apache.org/list?dev@flink.apache.org:lte=100M:Fedulov
> > > [4]
> https://lists.apache.org/list?u...@flink.apache.org:lte=100M:Fedulov
> > > [5]
> > >
> >
> https://flink.apache.org/2020/01/15/advanced-flink-application-patterns-vol.1-case-study-of-a-fraud-detection-system/
> > > [6]
> > >
> >
> https://www.ververica.com/blog/presenting-our-streaming-concepts-introduction-to-flink-video-series
> > >
> >
>

Re: [VOTE] FLIP-382: Unify the Provision of Diverse Metadata for Context-like APIs

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


weijie guo  于2023年12月15日周五 10:05写道：

> +1(binding)
>
> Best regards,
>
> Weijie
>
>
> Wencong Liu  于2023年12月15日周五 09:13写道：
>
> > Hi dev,
> >
> > I'd like to start a vote on FLIP-382.
> >
> > Discussion thread:
> > https://lists.apache.org/thread/3mgsc31odtpmzzl32s4oqbhlhxd0mn6b
> > FLIP:
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs
> >
> > Best regards,
> > Wencong Liu
>

Re: [VOTE] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

[Yuxin Tan]

Thanks Wencong for driving the Flip.

+1(non-binding)

Best,
Yuxin


Wencong Liu  于2023年12月15日周五 09:15写道：

> Hi dev,
>
> I'd like to start a vote on FLIP-380.
>
> Discussion thread:
> https://lists.apache.org/thread/nn7myj7vsvytbkdrnbvj5h0homsjrn1h
> FLIP:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream
>
> Best regards,
> Wencong Liu

Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an alternative for Flink's current Azure CI infrastructure

[Yuxin Tan]

+1 (non binding)
Thanks for the effort.

Best,
Yuxin


Samrat Deb  于2023年12月12日周二 15:25写道：

> +1 (non binding)
> Thanks for driving
>
> On Tue, 12 Dec 2023 at 11:59 AM, Sergey Nuyanzin 
> wrote:
>
> > +1 (binding)
> >
> > Thanks for driving this
> >
> > On Tue, Dec 12, 2023, 07:22 Rui Fan <1996fan...@gmail.com> wrote:
> >
> > > +1(binding)
> > >
> > > Best,
> > > Rui
> > >
> > > On Tue, Dec 12, 2023 at 11:58 AM weijie guo  >
> > > wrote:
> > >
> > > > Thanks Matthias for this efforts.
> > > >
> > > > +1(binding)
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > Weijie
> > > >
> > > >
> > > > Matthias Pohl  于2023年12月11日周一
> 21:51写道：
> > > >
> > > > > Hi everyone,
> > > > > I'd like to start a vote on FLIP-396 [1]. It covers enabling GitHub
> > > > Actions
> > > > > (GHA) in Apache Flink. This means that GHA workflows will run aside
> > > from
> > > > > the usual Azure CI workflows in a trial phase (which ends earliest
> > with
> > > > the
> > > > > release of Flink 1.19). Azure CI will still serve as the project's
> > > ground
> > > > > of truth until the community decides in a final vote to switch to
> GHA
> > > or
> > > > > stick to Azure CI.
> > > > >
> > > > > The related discussion thread can be found in [2].
> > > > >
> > > > > The vote will remain open for at least 72 hours and only concluded
> if
> > > > there
> > > > > are no objections and enough (i.e. at least 3) binding votes.
> > > > >
> > > > > Matthias
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure
> > > > > [2]
> https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb
> > > > >
> > > > > --
> > > > >
> > > > > [image: Aiven] 
> > > > >
> > > > > *Matthias Pohl*
> > > > > Opensource Software Engineer, *Aiven*
> > > > > matthias.p...@aiven.io|  +49 170 9869525
> > > > > aiven.io    |   <
> > > > https://www.facebook.com/aivencloud
> > > > > >
> > > > >      <
> > > > > https://twitter.com/aiven_io>
> > > > > *Aiven Deutschland GmbH*
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

[Yuxin Tan]

Thanks Wencong for driving this FLIP.

+1 from my side. It appears to significantly improve the handling
of full-window data within the DataStream API. However, I do
have a small question regarding the current limitation to batch
processing: does this stem from performance-related considerations?
Additionally, is there any possibility that support for streaming
in the future?

In addition, some format of the section `API implementation` is
not right (some lines have exceeded the text box), maybe we
can update and fix it.

Best,
Yuxin


weijie guo  于2023年12月12日周二 15:09写道：

> Thanks Wencong for driving this!
>
> I believe this is a useful feature, so +1 from my side.
>
> I only have one minor question about the exchange mode of `xxxPartition`
> method. Does this means the window operator must be connected to the
> upstream operator in forward edge (otherwise the concept of mapPartition is
> a bit far-fetched).
>
> Best regards,
>
> Weijie
>
>
> Wencong Liu  于2023年12月1日周五 14:04写道：
>
> > Hi devs,
> >
> > I'm excited to propose a new FLIP[1] aimed at enhancing the DataStream
> API
> >
> > to support full window processing on non-keyed streams. This feature
> > addresses
> > the current limitation where non-keyed DataStreams cannot accumulate
> > records
> > per subtask for collective processing at the end of input.
> >
> > Key proposals include:
> >
> >
> > 1. Introduction of PartitionWindowedStream allowing non-keyed DataStreams
> > to
> > be transformed for full window processing per subtask.
> >
> > 2. Addition of four new APIs - mapPartition, sortPartition, aggregate,
> and
> > reduce
> > - to enable powerful operations on PartitionWindowedStream.
> >
> > This initiative seeks to fill the gap left by the deprecation of the
> > DataSet API,
> > marrying its partition processing strengths with the dynamic capabilities
> > of the DataStream API.
> >
> > Looking forward to your feedback on this FLIP.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream
> >
> > Best regards,
> > Wencong Liu
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

[Yuxin Tan]

Cool, +1 for contributing CDC Connectors to Apache Flink.

Best,
Yuxin


Jing Ge  于2023年12月7日周四 15:43写道：

> Awesome! +1
>
> Best regards,
> Jing
>
> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin 
> wrote:
>
> > thanks for working on this and driving it
> >
> > +1
> >
> > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin  wrote:
> >
> > > This is incredibly exciting news, a big +1 for this.
> > >
> > > Thank you for the fantastic work on Flink CDC. We have created
> thousands
> > of
> > > real-time integration jobs using Flink CDC connectors.
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
> gongzhongqi...@apache.org
> > >
> > > wrote:
> > >
> > > > It's very exciting to hear the news.
> > > > +1 for adding CDC Connectors  to Apache Flink !
> > > >
> > > >
> > > > Best,
> > > > Zhongqiang
> > > >
> > > > Leonard Xu  于2023年12月7日周四 11:25写道：
> > > >
> > > > > Dear Flink devs,
> > > > >
> > > > >
> > > > > As you may have heard, we at Alibaba (Ververica) are planning to
> > donate
> > > > CDC Connectors for the Apache Flink project
> > > > > *[1]* to the Apache Flink community.
> > > > >
> > > > >
> > > > >
> > > > > CDC Connectors for Apache Flink comprise a collection of source
> > > > connectors designed specifically for Apache Flink. These connectors
> > > > > *[2]*
> > > > >  enable the ingestion of changes from various databases using
> Change
> > > > Data Capture (CDC), most of these CDC connectors are powered by
> > Debezium
> > > > > *[3]*
> > > > > . They support both the DataStream API and the Table/SQL API,
> > > > facilitating the reading of database snapshots and continuous reading
> > of
> > > > transaction logs with exactly-once processing, even in the event of
> > > > failures.
> > > > >
> > > > >
> > > > >
> > > > > Additionally, in the latest version 3.0, we have introduced many
> > > > long-awaited features. Starting from CDC version 3.0, we've built a
> > > > Streaming ELT Framework available for streaming data integration.
> This
> > > > framework allows users to write their data synchronization logic in a
> > > > simple YAML file, which will automatically be translated into a Flink
> > > > DataStreaming job. It emphasizes optimizing the task submission
> process
> > > and
> > > > offers advanced functionalities such as whole database
> synchronization,
> > > > merging sharded tables, and schema evolution
> > > > > *[4]*.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > I believe this initiative is a perfect match for both sides. For
> the
> > > > Flink community, it presents an opportunity to enhance Flink's
> > > competitive
> > > > advantage in streaming data integration, promoting the healthy growth
> > and
> > > > prosperity of the Apache Flink ecosystem. For the CDC Connectors
> > project,
> > > > becoming a sub-project of Apache Flink means being part of a neutral
> > > > open-source community, which can attract a more diverse pool of
> > > > contributors.
> > > > >
> > > > >
> > > > > Please note that the aforementioned points represent only some of
> our
> > > > motivations and vision for this donation. Specific future operations
> > need
> > > > to be further discussed in this thread. For example, the sub-project
> > name
> > > > after the donation; we hope to name it Flink-CDC
> > > > > aiming to streaming data intergration through Apache Flink,
> > > > > following the naming convention of Flink-ML; And this project is
> > > managed
> > > > by a total of 8 maintainers, including 3 Flink PMC members and 1
> Flink
> > > > Committer. The remaining 4 maintainers are also highly active
> > > contributors
> > > > to the Flink community, donating this project to the Flink community
> > > > implies that their permissions might be reduced. Therefore, we may
> need
> > > to
> > > > bring up this topic for further discussion within the Flink PMC.
> > > > Additionally, we need to discuss how to migrate existing users and
> > > > documents. We have a user group of nearly 10,000 people and a
> > > multi-version
> > > > documentation site need to migrate. We also need to plan for the
> > > migration
> > > > of CI/CD processes and other specifics.
> > > > >
> > > > >
> > > > >
> > > > > While there are many intricate details that require implementation,
> > we
> > > > are committed to progressing and finalizing this donation process.
> > > > >
> > > > >
> > > > >
> > > > > Despite being Flink’s most active ecological project (as evaluated
> by
> > > > GitHub metrics), it also boasts a significant user base. However, I
> > > believe
> > > > it's essential to commence discussions on future operations only
> after
> > > the
> > > > community reaches a consensus on whether they desire this donation.
> > > > >
> > > > >
> > > > > Really looking forward to hear what you think!
> > > > >
> > > > >
> > > > >
> > > > > Best,
> > > > > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > > > >
> > > > > [1] 

[jira] [Created] (FLINK-33746) More precise dynamic selection of Hybrid Shuffle or AQE

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33746:
-

 Summary: More precise dynamic selection of Hybrid Shuffle or AQE
 Key: FLINK-33746
 URL: https://issues.apache.org/jira/browse/FLINK-33746
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination, Runtime / Network
Affects Versions: 1.19.0
Reporter: Yuxin Tan


We can even adopt more precise and intelligent strategies to select between 
Hybrid Shuffle and AQE. For instance, the choice could be made based on the 
edge type between tasks, or we could leverage historical job performance data 
and other metrics to inform our decision. Such tailored strategies would enable 
us to utilize each feature where it is most beneficial.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33745) Dynamically choose hybrid shuffle or AQE in a job level

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33745:
-

 Summary: Dynamically choose hybrid shuffle or AQE in a job level
 Key: FLINK-33745
 URL: https://issues.apache.org/jira/browse/FLINK-33745
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Affects Versions: 1.19.0
Reporter: Yuxin Tan


To enhance the initial integration of Hybrid Shuffle with Adaptive Query 
Execution (AQE), we could implement a coarse-grained mode selection strategy. 
For instance, we can opt for either Hybrid Shuffle or AQE at the granularity 
level of an entire job. This approach would allow us to better align the two 
features in the early stages of adoption.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33744) Hybrid shuffle avoids restarting the whole job when failover

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33744:
-

 Summary: Hybrid shuffle avoids restarting the whole job when 
failover
 Key: FLINK-33744
 URL: https://issues.apache.org/jira/browse/FLINK-33744
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Affects Versions: 1.19.0
Reporter: Yuxin Tan


If Hybrid shuffle is enabled, the whole job will be restarted when failover. 
This is a critical issue for large-scale jobs. We should improve the logic and 
avoid restarting the whole job when failover.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33743) Support consuming multiple subpartitions on a single channel

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33743:
-

 Summary: Support consuming multiple subpartitions on a single 
channel
 Key: FLINK-33743
 URL: https://issues.apache.org/jira/browse/FLINK-33743
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: Yuxin Tan


At present, a downstream channel is limited to consuming data from a single 
subpartition, a constraint that can lead to increased memory consumption. 
Addressing this issue is also a critical step in ensuring that Hybrid Shuffle 
functions effectively with Adaptive Query Execution (AQE). 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33742) Hybrid Shuffle should work well with Adaptive Query Execution

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33742:
-

 Summary: Hybrid Shuffle should work well with Adaptive Query 
Execution
 Key: FLINK-33742
 URL: https://issues.apache.org/jira/browse/FLINK-33742
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination, Runtime / Network
Affects Versions: 1.19.0
Reporter: Yuxin Tan


At present, Hybrid Shuffle and Adaptive Query Execution (AQE), which includes 
features such as Dynamic Partition Pruning (DPP), Runtime Filter, and Adaptive 
Batch Scheduler, are not fully compatible. While they can be used concurrently 
at the same time, the activation of AQE inhibits the key capability of Hybrid 
Shuffle to perform simultaneous reading and writing. This limitation arises 
because AQE dictates that downstream tasks may only initiate once upstream 
tasks have finished, a requirement that is inconsistent with the simultaneous 
read-write process facilitated by Hybrid Shuffle.

To harness the full potential of Hybrid Shuffle and AQE, it is essential to 
refine their integration. By doing so, we can capitalize on each feature's 
distinct advantages and enhance overall system performance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Yuxin Tan]

Ok, Thanks for the update and the explanations.

Best,
Yuxin


Matthias Pohl  于2023年11月29日周三 15:43写道：

> >
> > According to the Flip, the new tests will support arm env.
> > I believe that's good news for arm users. I have a minor
> > question here. Will it be a blocker before migrating the new
> > tests? If not,  If not, when can we expect arm environment
> > support to be implemented? Thanks.
>
>
> Thanks for your feedback, everyone.
>
> About the ARM support. I want to underline that this FLIP is not about
> migrating to GitHub Actions but to start a trial run in the Apache Flink
> repository. That would allow us to come up with a proper decision whether
> GitHub Actions is what we want. I admit that the title is a bit
> "clickbaity". I updated the FLIP's title and its Motivation to make things
> clear.
>
> The FLIP suggests starting a trial period until 1.19 is released to try
> things out. A proper decision on whether we want to migrate would be made
> at the end of the 1.19 release cycle.
>
> About the ARM support: This related content of the FLIP is entirely based
> on documentation from Apache INFRAs side. INFRA seems to offer this ARM
> support for their ephemeral runners. The ephemeral runners are in the
> testing stage, i.e. these runners are still experimental. Apache INFRA asks
> Apache projects to join this test.
>
> Whether the ARM support is actually possible to achieve within Flink is
> something we have to figure out as part of the trial run. One conclusion of
> the trial run could be that we still move ahead with GHA but don't use arm
> machines due to some blocking issues.
>
> Matthias
>
>
>
> On Wed, Nov 29, 2023 at 4:46 AM Yuxin Tan  wrote:
>
> > Hi, Matthias,
> >
> > Thanks for driving this.
> > +1 from my side.
> >
> > According to the Flip, the new tests will support arm env.
> > I believe that's good news for arm users. I have a minor
> > question here. Will it be a blocker before migrating the new
> > tests? If not,  If not, when can we expect arm environment
> > support to be implemented? Thanks.
> >
> > Best,
> > Yuxin
> >
> >
> > Márton Balassi  于2023年11月29日周三 03:09写道：
> >
> > > Thanks, Matthias. Big +1 from me.
> > >
> > > On Tue, Nov 28, 2023 at 5:30 PM Matthias Pohl
> > >  wrote:
> > >
> > > > Thanks for the pointer. I'm planning to join that meeting.
> > > >
> > > > On Tue, Nov 28, 2023 at 4:16 PM Etienne Chauchot <
> echauc...@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > FYI there is the ASF infra roundtable soon. One of the subjects for
> > > this
> > > > > session is GitHub Actions. It could be worth passing by:
> > > > >
> > > > > December 6th, 2023 at 1700 UTC on the #Roundtablechannel on Slack.
> > > > >
> > > > > For information about theroundtables, and about how to join,
> > > > > see:https://infra.apache.org/roundtable.html
> > > > > <https://infra.apache.org/roundtable.html>
> > > > >
> > > > > Best
> > > > >
> > > > > Etienne
> > > > >
> > > > > Le 24/11/2023 à 14:16, Maximilian Michels a écrit :
> > > > > > Thanks for reviving the efforts here Matthias! +1 for the
> > transition
> > > > > > to GitHub Actions.
> > > > > >
> > > > > > As for ASF Infra Jenkins, it works fine. Jenkins is extremely
> > > > > > feature-rich. Not sure about the spare capacity though. I know
> that
> > > > > > for Apache Beam, Google donated a bunch of servers to get
> > additional
> > > > > > build capacity.
> > > > > >
> > > > > > -Max
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
> > > > > >   wrote:
> > > > > >> Btw. even though we've been focusing on GitHub Actions with this
> > > FLIP,
> > > > > I'm
> > > > > >> curious whether somebody has experience with Apache Infra's
> > Jenkins
> > > > > >> deployment. The discussion I found about Jenkins [1] is quite
> > > > out-dated
> > > > > >> (2014). I haven't worked with it myself but could imagine that
> > there
> > > > are
> > > > > >>

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Yuxin Tan]

Hi, Matthias,

Thanks for driving this.
+1 from my side.

According to the Flip, the new tests will support arm env.
I believe that's good news for arm users. I have a minor
question here. Will it be a blocker before migrating the new
tests? If not,  If not, when can we expect arm environment
support to be implemented? Thanks.

Best,
Yuxin


Márton Balassi  于2023年11月29日周三 03:09写道：

> Thanks, Matthias. Big +1 from me.
>
> On Tue, Nov 28, 2023 at 5:30 PM Matthias Pohl
>  wrote:
>
> > Thanks for the pointer. I'm planning to join that meeting.
> >
> > On Tue, Nov 28, 2023 at 4:16 PM Etienne Chauchot 
> > wrote:
> >
> > > Hi all,
> > >
> > > FYI there is the ASF infra roundtable soon. One of the subjects for
> this
> > > session is GitHub Actions. It could be worth passing by:
> > >
> > > December 6th, 2023 at 1700 UTC on the #Roundtablechannel on Slack.
> > >
> > > For information about theroundtables, and about how to join,
> > > see:https://infra.apache.org/roundtable.html
> > > 
> > >
> > > Best
> > >
> > > Etienne
> > >
> > > Le 24/11/2023 à 14:16, Maximilian Michels a écrit :
> > > > Thanks for reviving the efforts here Matthias! +1 for the transition
> > > > to GitHub Actions.
> > > >
> > > > As for ASF Infra Jenkins, it works fine. Jenkins is extremely
> > > > feature-rich. Not sure about the spare capacity though. I know that
> > > > for Apache Beam, Google donated a bunch of servers to get additional
> > > > build capacity.
> > > >
> > > > -Max
> > > >
> > > >
> > > > On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
> > > >   wrote:
> > > >> Btw. even though we've been focusing on GitHub Actions with this
> FLIP,
> > > I'm
> > > >> curious whether somebody has experience with Apache Infra's Jenkins
> > > >> deployment. The discussion I found about Jenkins [1] is quite
> > out-dated
> > > >> (2014). I haven't worked with it myself but could imagine that there
> > are
> > > >> some features provided through plugins which are missing in GitHub
> > > Actions.
> > > >>
> > > >> [1]https://lists.apache.org/thread/vs81xdhn3q777r7x9k7wd4dyl9kvoqn4
> > > >>
> > > >> On Tue, Nov 21, 2023 at 4:19 PM Matthias Pohl<
> matthias.p...@aiven.io>
> > > >> wrote:
> > > >>
> > > >>> That's a valid point. I updated the FLIP accordingly:
> > > >>>
> > >  Currently, the secrets (e.g. for S3 access tokens) are maintained
> by
> > >  certain PMC members with access to the corresponding configuration
> > in
> > > the
> > >  Azure CI project. This responsibility will be moved to Apache
> Infra.
> > > They
> > >  are in charge of handling secrets in the Apache organization. As a
> > >  consequence, updating secrets is becoming a bit more complicated.
> > > This can
> > >  be still considered an improvement from a legal standpoint because
> > the
> > >  responsibility is transferred from an individual company (i.e.
> > > Ververica
> > >  who's the maintainer of the Azure CI project) to the Apache
> > > Foundation.
> > > >>>
> > > >>> On Tue, Nov 21, 2023 at 3:37 PM Martijn Visser<
> > > martijnvis...@apache.org>
> > > >>> wrote:
> > > >>>
> > >  Hi Matthias,
> > > 
> > >  Thanks for the write-up and for the efforts on this. I really hope
> > >  that we can move away from Azure towards GHA for a better
> > integration
> > >  as well (directly seeing if a PR can be merged due to CI passing
> for
> > >  example).
> > > 
> > >  The one thing I'm missing in the FLIP is how we would setup the
> > >  secrets for the nightly runs (for the S3 tests, potential tests
> with
> > >  external services etc). My guess is we need to provide the secret
> to
> > >  ASF Infra and then we would be able to refer to them in a
> pipeline?
> > > 
> > >  Best regards,
> > > 
> > >  Martijn
> > > 
> > >  On Tue, Nov 21, 2023 at 3:05 PM Matthias Pohl
> > >    wrote:
> > > > I realized that I mixed up FLIP IDs. FLIP-395 is already reserved
> > > [1]. I
> > > > switched to FLIP-396 [2] for the sake of consistency. 8)
> > > >
> > > > [1]
> > https://lists.apache.org/thread/wjd3nbvg6nt93lb0sd52f0lzls6559tv
> > > > [2]
> > > >
> > > 
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Migration+to+GitHub+Actions
> > > > On Tue, Nov 21, 2023 at 2:58 PM Matthias Pohl<
> > matthias.p...@aiven.io
> > > >
> > > > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> The Flink community discussed migrating from Azure CI to GitHub
> > >  Actions
> > > >> quite some time ago [1]. The efforts around that stalled due to
> > >  limitations
> > > >> around self-hosted runner support from Apache Infra’s side.
> There
> > >  were some
> > > >> recent developments on that topic. Apache Infra is experimenting
> > > with
> > > >> ephemeral runners now which might enable us to move ahead with
> > > GitHub
> > > >> Actions.
> > > 

Re: [VOTE] Release 1.17.2, release candidate #1

[Yuxin Tan]

+1 (non-binding)

- Verified signatures and hashsums
- Verified there are no binaries in the source archive
- Built Flink from sources
- Reviewed the web PR

Best,
Yuxin


Feng Jin  于2023年11月24日周五 02:20写道：

> +1(non-binding)
>
> - verified signatures and hashsums
> - Verified there are no binaries in the source archive
> - Built Flink from sources
> - Started local standalone cluster, submitted sql job using sql-client
>
> Best,
> Feng
>
>
> On Thu, Nov 23, 2023 at 11:08 PM Hang Ruan  wrote:
>
> > +1 (non-binding)
> >
> > - verified signatures
> > - verified hashsums
> > - Verified there are no binaries in the source archive
> > - reviewed the web PR
> > - built Flink from sources
> >
> > Best,
> > Hang
> >
> > Jiabao Sun  于2023年11月23日周四 22:09写道：
> >
> > > Thanks for driving this release.
> > >
> > > +1(non-binding)
> > >
> > > - Checked the tag in git
> > > - Verified signatures and hashsums
> > > - Verified there are no binaries in the source archive
> > > - Built Flink from sources
> > >
> > > Best,
> > > Jiabao
> > >
> > >
> > > > 2023年11月21日 20:46，Matthias Pohl  写道：
> > > >
> > > > +1 (binding)
> > > >
> > > > * Downloaded artifacts
> > > > * Built Flink from sources
> > > > * Verified SHA512 checksums & GPG signatures
> > > > * Compared checkout with provided sources
> > > > * Verified pom file versions
> > > > * Went over NOTICE/pom file changes without finding anything
> suspicious
> > > > * Deployed standalone session cluster and ran WordCount example in
> > batch
> > > > and streaming: Nothing suspicious in log files found
> > > > * Verified Java version of uploaded binaries
> > > >
> > > > Thanks Yun Tang for taking care of it.
> > > >
> > > > On Thu, Nov 16, 2023 at 7:02 AM Rui Fan <1996fan...@gmail.com>
> wrote:
> > > >
> > > >> +1 (non-binding)
> > > >>
> > > >> - Verified signatures
> > > >> - Reviewed the flink-web PR, left a couple of comments
> > > >> - The source archives do not contain any binaries
> > > >> - Build the source with Maven 3 and java8 (Checked the license as
> > well)
> > > >> - bin/start-cluster.sh with java8, it works fine and no any
> unexpected
> > > LOG
> > > >> - Ran demo, it's fine:  bin/flink run
> > > >> examples/streaming/StateMachineExample.jar
> > > >>
> > > >> Best,
> > > >> Rui
> > > >>
> > > >> On Mon, Nov 13, 2023 at 4:04 PM Yun Tang  wrote:
> > > >>
> > > >>> Hi everyone,
> > > >>>
> > > >>> Please review and vote on the release candidate #1 for the version
> > > >> 1.17.2,
> > > >>>
> > > >>> as follows:
> > > >>>
> > > >>> [ ] +1, Approve the release
> > > >>>
> > > >>> [ ] -1, Do not approve the release (please provide specific
> comments)
> > > >>>
> > > >>>
> > > >>> The complete staging area is available for your review, which
> > includes:
> > > >>> * JIRA release notes [1],
> > > >>>
> > > >>> * the official Apache source release and binary convenience
> releases
> > to
> > > >> be
> > > >>> deployed to dist.apache.org [2], which are signed with the key
> with
> > > >>> fingerprint 2E0E1AB5D39D55E608071FB9F795C02A4D2482B3 [3],
> > > >>>
> > > >>> * all artifacts to be deployed to the Maven Central Repository [4],
> > > >>>
> > > >>> * source code tag "release-1.17.2-rc1" [5],
> > > >>>
> > > >>> * website pull request listing the new release and adding
> > announcement
> > > >>> blog post [6].
> > > >>>
> > > >>>
> > > >>> The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > >>> approval, with at least 3 PMC affirmative votes.
> > > >>>
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353260
> > > >>>
> > > >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.17.2-rc1/
> > > >>>
> > > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > >>>
> > > >>> [4]
> > > >>>
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1669/
> > > >>>
> > > >>> [5]
> https://github.com/apache/flink/releases/tag/release-1.17.2-rc1
> > > >>>
> > > >>> [6] https://github.com/apache/flink-web/pull/696
> > > >>>
> > > >>> Thanks,
> > > >>> Release Manager
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: [NOTICE] flink-table-planner module has been migrated to JUnit5

[Yuxin Tan]

Thanks Jiabao for the quick response.

Best,
Yuxin


Jiabao Sun  于2023年11月23日周四 11:25写道：

> Thanks Yuxin for you reminder,
>
> I will complete it today.
>
> Best,
> Jiabao
>
>
> > 2023年11月22日 16:26，Yuxin Tan  写道：
> >
> > Hi, Leonard, jiabao
> >
> > Thanks for migrating the test classes.
> >
> > I have noticed that many test classes (e.g.,
> FlinkCalciteCatalogReaderTest)
> > in flink-table (including flink-table-planner) have been updated to use
> > AssertJ.
> > However, their dependent test classes still utilize org.junit.Test or
> > org.junit.Before,
> > etc., whereas they should be using org.junit.jupiter.api.Test, etc. These
> > classes are mostly from the issue FLINK-26582[1]. Maybe we need to fix
> > these classes as well.
> >
> > [1] https://github.com/apache/flink/pull/19039
> >
> > Best,
> > Yuxin
> >
> >
> > Jane Chan  于2023年11月22日周三 15:32写道：
> >
> >> Thanks Leonard and Jiabao for your tremendous efforts in this regard!
> >>
> >> Best,
> >> Jane
> >>
> >> On Wed, Nov 22, 2023 at 2:42 PM Leonard Xu  wrote:
> >>
> >>> Hi, all
> >>>
> >>> There are a lot of JUnit4 tests in flink-table-planner module, but now
> >>> Jiabao and I have just migrated all tests to JUnit5 through
> >> FLINK-29451[1].
> >>>
> >>> Please use JUnit5 tests when submitting new PRs for flink-table-planner
> >>> module. The opened PRs for flink-table-planner module need to rebase to
> >> the
> >>> latest master.
> >>>
> >>> Thanks Jiabao for the huge efforts.
> >>>
> >>>
> >>> Best,
> >>> Leonard
> >>>
> >>> [1]https://issues.apache.org/jira/browse/FLINK-29541
> >>
>
>

Re: [NOTICE] flink-table-planner module has been migrated to JUnit5

[Yuxin Tan]

Hi, Leonard, jiabao

Thanks for migrating the test classes.

I have noticed that many test classes (e.g., FlinkCalciteCatalogReaderTest)
in flink-table (including flink-table-planner) have been updated to use
AssertJ.
However, their dependent test classes still utilize org.junit.Test or
org.junit.Before,
etc., whereas they should be using org.junit.jupiter.api.Test, etc. These
classes are mostly from the issue FLINK-26582[1]. Maybe we need to fix
these classes as well.

[1] https://github.com/apache/flink/pull/19039

Best,
Yuxin


Jane Chan  于2023年11月22日周三 15:32写道：

> Thanks Leonard and Jiabao for your tremendous efforts in this regard!
>
> Best,
> Jane
>
> On Wed, Nov 22, 2023 at 2:42 PM Leonard Xu  wrote:
>
> > Hi, all
> >
> > There are a lot of JUnit4 tests in flink-table-planner module, but now
> > Jiabao and I have just migrated all tests to JUnit5 through
> FLINK-29451[1].
> >
> > Please use JUnit5 tests when submitting new PRs for flink-table-planner
> > module. The opened PRs for flink-table-planner module need to rebase to
> the
> > latest master.
> >
> > Thanks Jiabao for the huge efforts.
> >
> >
> > Best,
> > Leonard
> >
> > [1]https://issues.apache.org/jira/browse/FLINK-29541
>

[jira] [Created] (FLINK-33612) The table plan of hybrid shuffle may introduce additional blocking edges occasionally

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33612:
-

 Summary: The table plan of hybrid shuffle may introduce additional 
blocking edges occasionally
 Key: FLINK-33612
 URL: https://issues.apache.org/jira/browse/FLINK-33612
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.19.0
Reporter: Yuxin Tan


To enhance the performance of hybrid shuffle, it is imperative to address the 
inconsistency between hybrid shuffle mode and blocking shuffle mode in certain 
query plans of TPC-DS (such as q88.sql, q14a.sql, q14b.sql, etc). 
In hybrid shuffle mode, these plans introduce additional blocking shuffle edges 
and result in increased shuffle times, potentially impacting overall 
efficiency. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33582) Update flink-shaded version

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33582:
-

 Summary: Update flink-shaded version
 Key: FLINK-33582
 URL: https://issues.apache.org/jira/browse/FLINK-33582
 Project: Flink
  Issue Type: Bug
  Components: BuildSystem / Shaded
Affects Versions: 1.17.2
Reporter: Yuxin Tan


This is a follow-up task for https://issues.apache.org/jira/browse/FLINK-33417. 

After flink-shaded 16.2 is released, we should update the flink-shaded version 
for Flink 1.17 to resolve the issue thoroughly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-shaded 16.2, release candidate #1

[Yuxin Tan]

Thanks weijie for driving the new release!

+1 (non-binding)

- Built from source code succeeded
- Verified signatures
- Verified hashsums
- Reviewed the web PR

Best,
Yuxin


weijie guo  于2023年11月13日周一 15:57写道：

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version
> 16.2, as follows:
>
> [ ] +1, Approve the release
>
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
>
>
>
> The complete staging area is available for your review, which includes:
>
> * JIRA release notes [1],
>
> * the official Apache source release to be deployed to dist.apache.org
>  [2], which are signed with the key with fingerprint
> 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [3],
>
> * all artifacts to be deployed to the Maven Central Repository [4],
>
> * source code tag "release-16.2-rc1" [5],
>
> * website pull request listing the new release [6].
>
>
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
>
>
> Thanks,
>
> Release Manager
>
>
>
> [1] https://issues.apache.org/jira/projects/FLINK/versions/12353810
>
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-16.2-rc1
>
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1671/
>
> [5] https://github.com/apache/flink-shaded/releases/tag/release-16.2-rc1
>
> [6] https://github.com/apache/flink-web/pull/697
>

Re: [VOTE] FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects

[Yuxin Tan]

+1(non-binding)

Best,
Yuxin


Yuepeng Pan  于2023年11月10日周五 18:31写道：

> +1(non-binding)
>
> Best,
> Roc
>
> On 2023/11/10 03:58:10 Junrui Lee wrote:
> > Hi everyone,
> >
> > Thank you to everyone for the feedback on FLIP-381: Deprecate
> configuration
> > getters/setters that return/set complex Java objects[1] which has been
> > discussed in this thread [2].
> >
> > I would like to start a vote for it. The vote will be open for at least
> 72
> > hours (excluding weekends) unless there is an objection or not enough
> votes.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992
> > [2]https://lists.apache.org/thread/y5owjkfxq3xs9lmpdbl6d6jmqdgbjqxo
> >
>

Re: [DISCUSS] Release flink-shaded 16.2

[Yuxin Tan]

Hi, Weijie,

Thank you for volunteering to be a release manager.

Currently, no objections are received. If there are no objections, we
will release flink-shaded 16.2 next week.

Best,
Yuxin


weijie guo  于2023年11月8日周三 11:11写道：

> Thanks Yuxin for driving this!
>
> I am willing to help release this. :)
>
> Best regards,
>
> Weijie
>
>
> Yuxin Tan  于2023年11月6日周一 15:45写道：
>
>> Hi, all,
>>
>> I would like to discuss creating a new 16.2 patch release for
>> flink-shaded[1].
>>
>> In our ARM environment, we recently encountered a critical bug
>> (FLINK-33417[2])
>> while using flink-shaded 16.1 in flink 1.17. The issue has been discussed
>> in the
>> discussion[3]. Fortunately, we could resolve this bug by updating the
>> netty version.
>>
>> To address this issue comprehensively, we need to release a new minor
>> version for
>> flink-shaded. As a result, Flink 1.17 should update its dependent
>> flink-shaded version
>> accordingly. However, there is no need for Flink 1.18+ to update the
>> flink-shaded
>> version since these newer versions already depend on flink-shaded 17.0+
>> and have
>> updated their netty version, thereby eliminating the existence of this
>> bug.
>>
>> Considering the low probability of this issue occurring, I do not
>> recommend
>> considering it as a blocker for the Flink 1.17 release. Furthermore, the
>> external connector has been decoupled from flink-shaded in FLINK-33190[4],
>> and the compatibility of the newer netty version 4.1.91 has been verified
>> in the new
>> Flink 1.18+(which is dependent on flink-shaded 17.0 and it is dependent
>> on
>> netty 4.1.91). Hence, we believe this update will not cause any
>> compatibility issues.
>>
>> If there are no objections, we would greatly appreciate it if a committer
>> could
>> volunteer as the release manager. Thank you.
>>
>> [1] https://github.com/apache/flink-shaded
>> [2] https://issues.apache.org/jira/browse/FLINK-33417
>> [3] https://lists.apache.org/thread/y1c8545bcsx2836d9pgfdzj65knvw7kb
>> [4] https://issues.apache.org/jira/browse/FLINK-33190
>>
>> Best,
>> Yuxin
>>
>

[jira] [Created] (FLINK-33493) Elasticsearch connector ElasticsearchWriterITCase test failed

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33493:
-

 Summary: Elasticsearch connector ElasticsearchWriterITCase test 
failed
 Key: FLINK-33493
 URL: https://issues.apache.org/jira/browse/FLINK-33493
 Project: Flink
  Issue Type: Bug
  Components: Connectors / ElasticSearch
Reporter: Yuxin Tan


When I ran tests, the test failed. The failed reason is

{code:java}
Error:  
/home/runner/work/flink-connector-elasticsearch/flink-connector-elasticsearch/flink-connector-elasticsearch-base/src/test/java/org/apache/flink/connector/elasticsearch/sink/ElasticsearchWriterITCase.java:[197,46]
 cannot find symbol
  symbol:   method 
mock(org.apache.flink.metrics.MetricGroup,org.apache.flink.metrics.groups.OperatorIOMetricGroup)
  location: class 
org.apache.flink.runtime.metrics.groups.InternalSinkWriterMetricGroup
Error:  
/home/runner/work/flink-connector-elasticsearch/flink-connector-elasticsearch/flink-connector-elasticsearch-base/src/test/java/org/apache/flink/connector/elasticsearch/sink/ElasticsearchWriterITCase.java:[273,46]
 cannot find symbol
  symbol:   method mock(org.apache.flink.metrics.MetricGroup)
{code}

https://github.com/apache/flink-connector-elasticsearch/actions/runs/6809899863/job/18517273714?pr=77#step:13:134.

ElasticsearchWriterITCase called Flink "InternalSinkWriterMetricGroup#mock", 
and it is renamed in https://github.com/apache/flink/pull/23541 (in Flink 
1.19). So the test failed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] Release flink-shaded 16.2

[Yuxin Tan]

Hi, all,

I would like to discuss creating a new 16.2 patch release for
flink-shaded[1].

In our ARM environment, we recently encountered a critical bug
(FLINK-33417[2])
while using flink-shaded 16.1 in flink 1.17. The issue has been discussed
in the
discussion[3]. Fortunately, we could resolve this bug by updating the netty
version.

To address this issue comprehensively, we need to release a new minor
version for
flink-shaded. As a result, Flink 1.17 should update its dependent
flink-shaded version
accordingly. However, there is no need for Flink 1.18+ to update the
flink-shaded
version since these newer versions already depend on flink-shaded 17.0+ and
have
updated their netty version, thereby eliminating the existence of this bug.

Considering the low probability of this issue occurring, I do not recommend
considering it as a blocker for the Flink 1.17 release. Furthermore, the
external connector has been decoupled from flink-shaded in FLINK-33190[4],
and the compatibility of the newer netty version 4.1.91 has been verified
in the new
Flink 1.18+(which is dependent on flink-shaded 17.0 and it is dependent on
netty 4.1.91). Hence, we believe this update will not cause any
compatibility issues.

If there are no objections, we would greatly appreciate it if a committer
could
volunteer as the release manager. Thank you.

[1] https://github.com/apache/flink-shaded
[2] https://issues.apache.org/jira/browse/FLINK-33417
[3] https://lists.apache.org/thread/y1c8545bcsx2836d9pgfdzj65knvw7kb
[4] https://issues.apache.org/jira/browse/FLINK-33190

Best,
Yuxin

Re: [DISCUSS] FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects

[Yuxin Tan]

Thanks Junrui for driving the proposal.

+1 for this proposal. I believe this change will enhance the usability of
Flink configuration for both users and developers, while also ensuring
consistency across various types of configurations.

Best,
Yuxin


Lijie Wang  于2023年11月3日周五 10:59写道：

> Thanks Junrui for driving this.
>
> Making configurations simple and consistent has great benefits for both
> users and devs. +1 for the proposal.
>
> Best,
> Lijie
>
> weijie guo  于2023年11月2日周四 16:49写道：
>
> > Thanks Junrui for driving this proposal!
> >
> > I believe this is helpful for the new Process Function API. Because we
> > don't need to move some related class/components from flink-core to a
> pure
> > API module (maybe, called flink-core-api) after this. Even though the
> FLIP
> > related to new API is in preparation atm, I still want to emphasize our
> > goal is that user application should no longer depend on these stuff. So
> > I'm + 1 for this proposal.
> >
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Zhu Zhu  于2023年11月2日周四 16:00写道：
> >
> > > Thanks Junrui for creating the FLIP and kicking off this discussion.
> > >
> > > The community has been constantly striving to unify and simplify the
> > > configuration layer of Flink. Some progress has already been made,
> > > such as FLINK-29379. However, the compatibility of public interfaces
> > > poses an obstacle to completing the task. The release of Flink 2.0
> > > presents a great opportunity to accomplish this goal.
> > >
> > > +1 for the proposal.
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Rui Fan <1996fan...@gmail.com> 于2023年11月2日周四 10:27写道：
> > >
> > > > Thanks Junrui for driving this proposal!
> > > >
> > > > ConfigOption is easy to use for flink users, easy to manage options
> > > > for flink platform maintainers, and easy to maintain for flink
> > developers
> > > > and flink community.
> > > >
> > > > So big +1 for this proposal!
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Thu, Nov 2, 2023 at 10:10 AM Junrui Lee 
> > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I would like to start a discussion on FLIP-381: Deprecate
> > configuration
> > > > > getters/setters that return/set complex Java objects[1].
> > > > >
> > > > > Currently, the job configuration in FLINK is spread out across
> > > different
> > > > > components, which leads to inconsistencies and confusion. To
> address
> > > this
> > > > > issue, it is necessary to migrate non-ConfigOption complex Java
> > objects
> > > > to
> > > > > use ConfigOption and adopt a single Configuration object to host
> all
> > > the
> > > > > configuration.
> > > > > However, there is a significant blocker in implementing this
> > solution.
> > > > > These complex Java objects in StreamExecutionEnvironment,
> > > > CheckpointConfig,
> > > > > and ExecutionConfig have already been exposed through the public
> API,
> > > > > making it challenging to modify the existing implementation.
> > > > >
> > > > > Therefore, I propose to deprecate these Java objects and their
> > > > > corresponding getter/setter interfaces, ultimately removing them in
> > > > > FLINK-2.0.
> > > > >
> > > > > Your feedback and thoughts on this proposal are highly appreciated.
> > > > >
> > > > > Best regards,
> > > > > Junrui Lee
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-33417) Update netty version to 4.1.83 for flink-shaded

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33417:
-

 Summary: Update netty version to 4.1.83 for flink-shaded
 Key: FLINK-33417
 URL: https://issues.apache.org/jira/browse/FLINK-33417
 Project: Flink
  Issue Type: Bug
  Components: BuildSystem / Shaded
Reporter: Yuxin Tan


In our ARM environment, we encounter a compile error when
using Flink 1.17.

Flink 1.17 depends on flink-shaded 16.1, which uses netty 4.1.82.
However, flink-shaded 16.1 fails to compile in the ARM
environment. As a result, we are unable to compile Flink 1.17
due to this issue.

We have tested compiling flink-shaded using netty 4.1.83 or
a later version in ARM env, and it can compile successfully.

Taking into consideration the previous discussions regarding
compatibility and the dependency of external connectors on
this version, I propose addressing the bug by only updating
flink-shaded's netty to a minor version (e.g., 4.1.83) rather than
backporting FLINK-32032. 

To implement the update, maybe a new release of flink-shaded 16.2 needs to be 
released.

The discussion details is at 
https://lists.apache.org/thread/y1c8545bcsx2836d9pgfdzj65knvw7kb.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33370) Simplify validateAndParseHostsString in Elasticsearch connecter's configuration

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33370:
-

 Summary: Simplify validateAndParseHostsString in Elasticsearch 
connecter's configuration
 Key: FLINK-33370
 URL: https://issues.apache.org/jira/browse/FLINK-33370
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / ElasticSearch
Reporter: Yuxin Tan
Assignee: Yuxin Tan


Currently, the validateAndParseHostsString method exists in each configuration 
file(repeated for 3 times), but the method logic is exactly the same. We can 
simplify it  by introducing a common util.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release 1.18.0, release candidate #3

[Yuxin Tan]

+1(non-binding)

- Verified checksum
- Build from source code
- Verified signature
- Started a local cluster and run Streaming & Batch wordcount job, the
result is expected
- Verified web PR

Best,
Yuxin


Qingsheng Ren  于2023年10月24日周二 11:19写道：

> +1 (binding)
>
> - Verified checksums and signatures
> - Built from source with Java 8
> - Started a standalone cluster and submitted a Flink SQL job that read and
> wrote with Kafka connector and CSV / JSON format
> - Reviewed web PR and release note
>
> Best,
> Qingsheng
>
> On Mon, Oct 23, 2023 at 10:40 PM Leonard Xu  wrote:
>
> > +1 (binding)
> >
> > - verified signatures
> > - verified hashsums
> > - built from source code succeeded
> > - checked all dependency artifacts are 1.18
> > - started SQL Client, used MySQL CDC connector to read changelog from
> > database , the result is expected
> > - reviewed the web PR, left minor comments
> > - reviewed the release notes PR, left minor comments
> >
> >
> > Best,
> > Leonard
> >
> > > 2023年10月21日 下午7:28，Rui Fan <1996fan...@gmail.com> 写道：
> > >
> > > +1(non-binding)
> > >
> > > - Downloaded artifacts from dist[1]
> > > - Verified SHA512 checksums
> > > - Verified GPG signatures
> > > - Build the source with java-1.8 and verified the licenses together
> > > - Verified web PR
> > >
> > > [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc3/
> > >
> > > Best,
> > > Rui
> > >
> > > On Fri, Oct 20, 2023 at 10:31 PM Martijn Visser <
> > martijnvis...@apache.org>
> > > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> - Validated hashes
> > >> - Verified signature
> > >> - Verified that no binaries exist in the source archive
> > >> - Build the source with Maven
> > >> - Verified licenses
> > >> - Verified web PR
> > >> - Started a cluster and the Flink SQL client, successfully read and
> > >> wrote with the Kafka connector to Confluent Cloud with AVRO and Schema
> > >> Registry enabled
> > >>
> > >> On Fri, Oct 20, 2023 at 2:55 PM Matthias Pohl
> > >>  wrote:
> > >>>
> > >>> +1 (binding)
> > >>>
> > >>> * Downloaded artifacts
> > >>> * Built Flink from sources
> > >>> * Verified SHA512 checksums GPG signatures
> > >>> * Compared checkout with provided sources
> > >>> * Verified pom file versions
> > >>> * Verified that there are no pom/NOTICE file changes since RC1
> > >>> * Deployed standalone session cluster and ran WordCount example in
> > batch
> > >>> and streaming: Nothing suspicious in log files found
> > >>>
> > >>> On Thu, Oct 19, 2023 at 3:00 PM Piotr Nowojski  >
> > >> wrote:
> > >>>
> >  +1 (binding)
> > 
> >  Best,
> >  Piotrek
> > 
> >  czw., 19 paź 2023 o 09:55 Yun Tang  napisał(a):
> > 
> > > +1 (non-binding)
> > >
> > >
> > >  *   Build from source code
> > >  *   Verify the pre-built jar packages were built with JDK8
> > >  *   Verify FLIP-291 with a standalone cluster, and it works fine
> > >> with
> > > StateMachine example.
> > >  *   Checked the signature
> > >  *   Viewed the PRs.
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Cheng Pan 
> > > Sent: Thursday, October 19, 2023 14:38
> > > To: dev@flink.apache.org 
> > > Subject: RE: [VOTE] Release 1.18.0, release candidate #3
> > >
> > > +1 (non-binding)
> > >
> > > We(the Apache Kyuubi community), verified that the Kyuubi Flink
> > >> engine
> > > works well[1] with Flink 1.18.0 RC3.
> > >
> > > [1] https://github.com/apache/kyuubi/pull/5465
> > >
> > > Thanks,
> > > Cheng Pan
> > >
> > >
> > > On 2023/10/19 00:26:24 Jing Ge wrote:
> > >> Hi everyone,
> > >>
> > >> Please review and vote on the release candidate #3 for the version
> > >> 1.18.0, as follows:
> > >> [ ] +1, Approve the release
> > >> [ ] -1, Do not approve the release (please provide specific
> > >> comments)
> > >>
> > >> The complete staging area is available for your review, which
> > >> includes:
> > >>
> > >> * JIRA release notes [1], and the pull request adding release note
> > >> for
> > >> users [2]
> > >> * the official Apache source release and binary convenience
> > >> releases to
> > > be
> > >> deployed to dist.apache.org [3], which are signed with the key
> > >> with
> > >> fingerprint 96AE0E32CBE6E0753CE6 [4],
> > >> * all artifacts to be deployed to the Maven Central Repository
> [5],
> > >> * source code tag "release-1.18.0-rc3" [6],
> > >> * website pull request listing the new release and adding
> > >> announcement
> > > blog
> > >> post [7].
> > >>
> > >> The vote will be open for at least 72 hours. It is adopted by
> > >> majority
> > >> approval, with at least 3 PMC affirmative votes.
> > >>
> > >> Best regards,
> > >> Konstantin, Sergey, Qingsheng, and Jing
> > >>
> > >> [1]
> > >>
> > >
> > 
> > >>
> >
> 

Re: Flink and Flink shaded dependency

[Yuxin Tan]

Hi, Matthias

Thanks a lot for joining the discussion.

> Generally, netty upgrades are a source of instability based on past
experiences. But considering that we have newer versions (4.1.91.Final)
tested with flink-shaded 17.0 in Flink 1.18, it should be alright bumping
netty in flink-shaded 16.2 to netty 4.1.83.Final.

Agree with your consideration, this also is the reason why we chose
4.1.83.Final.

> If no other objections are shared, the next step would be to create a Jira
issue, do the change in flink-shaded, release flink-shaded 16.2 and update
the flink-shaded version in the 1.17 branch.

If there are no objections, I'm ready to create a Jira ticket and prepare
the
PR for the change. However, it may need assistance from a committer for
the subsequent steps. It would be very appreciated if someone could take the
releasing task. Thanks

Best,
Yuxin


Matthias Pohl  于2023年10月20日周五 20:30写道：

>  IHi Yuxin,
> That would be the way to go for flink-shaded and Flink 1.17.
>
> Generally, netty upgrades are a source of instability based on past
> experiences. But considering that we have newer versions (4.1.91.Final)
> tested with flink-shaded 17.0 in Flink 1.18, it should be alright bumping
> netty in flink-shaded 16.2 to netty 4.1.83.Final.
>
> One argument against such an upgrade would be that the Flink community is
> not supporting arm officially (i.e. there's no CI setup). In this sense I
> want to wait whether other's have objections against doing the upgrade.
>
> If no other objections are shared, the next step would be to create a Jira
> issue, do the change in flink-shaded, release flink-shaded 16.2 and update
> the flink-shaded version in the 1.17 branch.
>
> Matthias
>
> On Mon, Oct 16, 2023 at 5:07 AM Yuxin Tan  wrote:
>
> > Hi, devs,
> >
> > I would like to revive the discussion again.
> >
> > In our ARM environment, we encounter a compile error when
> > using Flink 1.17.
> >
> > Flink 1.17 depends on flink-shaded 16.1, which uses netty 4.1.82.
> > However, flink-shaded 16.1 fails to compile in the ARM
> > environment. As a result, we are unable to compile Flink 1.17
> > due to this issue.
> >
> > We have tested compiling flink-shaded using netty 4.1.83 or
> > a later version in ARM env, and it can compile successfully.
> >
> > I believe this is a critical bug that needs to be addressed first.
> >
> > Taking into consideration the previous discussions regarding
> > compatibility and the dependency of external connectors on
> > this version, I propose addressing the bug by only updating
> > flink-shaded's netty to a minor version (e.g., 4.1.83) rather than
> > backporting FLINK-32032. This approach can avoid introducing
> > other changes, such as updating to a higher netty version (4.1.91),
> > potential guava version alterations, Curator version modifications,
> > and so on.
> >
> > WDYT?
> >
> >
> > Best,
> > Yuxin
> >
> >
> > Jing Ge  于2023年10月5日周四 14:39写道：
> >
> > > Hi Chesnay,
> > >
> > > Thanks for joining this discussion and sharing your thoughts!
> > >
> > >
> > > > Connectors shouldn't depend on flink-shaded.
> > > >
> > >
> > > Perfect! We are on the same page. If you could read through the
> > discussion,
> > > you would realize that, currently, there are many connectors depend on
> > > flink-shaded.
> > >
> > >
> > > > Connectors are small enough in scope that depending directly on
> > > > guava/jackson/etc. is a fine approach, and they have plenty of other
> > > > dependencies that they need to manage anyway; let's treat these the
> > same
> > > > way.
> > > >
> > >
> > > It is even better, if we could do that. Jira tickest[1] are created.
> > >
> > >
> > > > As for class-loading, there has been a long-standing goal of each
> > > > connector being loaded in their own classloader. That still is the
> > north
> > > > star and the only reasonable way to ensure that multiple connectors
> can
> > > > be safely used with SQL.
> > > >
> > >
> > > What is the current status? Do we have any Jira ticket for that?
> > >
> > > Best regards,
> > > Jing
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-33190
> > >
> > > On Wed, Oct 4, 2023 at 4:43 PM Chesnay Schepler 
> > > wrote:
> > >
> > > > There is no "monolithic" flink-shaded dependency.
> > > > Connectors shouldn't depend on anythi

Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

[Yuxin Tan]

Congratulations, Ron!

Best,
Yuxin


Junrui Lee  于2023年10月16日周一 10:24写道：

> Congratulations Ron !
>
> Best,
> Junrui
>
> Yun Tang  于2023年10月16日周一 10:22写道：
>
> > Congratulations, Ron!
> >
> > Best
> > Yun Tang
> > 
> > From: yu zelin 
> > Sent: Monday, October 16, 2023 10:16
> > To: dev@flink.apache.org 
> > Cc: ron9@gmail.com 
> > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu
> >
> > Congratulations!
> >
> > Best,
> > Yu Zelin
> >
> > > 2023年10月16日 09:56，Jark Wu  写道：
> > >
> > > Hi, everyone
> > >
> > > On behalf of the PMC, I'm very happy to announce Ron Liu as a new Flink
> > > Committer.
> > >
> > > Ron has been continuously contributing to the Flink project for many
> > years,
> > > authored and reviewed a lot of codes. He mainly works on Flink SQL
> parts
> > > and drove several important FLIPs, e.g., USING JAR (FLIP-214), Operator
> > > Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a great
> > > knowledge of the Batch SQL and improved a lot of batch performance in
> the
> > > past several releases. He is also quite active in mailing lists,
> > > participating in discussions and answering user questions.
> > >
> > > Please join me in congratulating Ron Liu for becoming a Flink
> Committer!
> > >
> > > Best,
> > > Jark Wu (on behalf of the Flink PMC)
> >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan

[Yuxin Tan]

Congratulations Jane!

Best,
Yuxin


xiangyu feng  于2023年10月16日周一 10:27写道：

> Congratulations Jane!
>
> Best,
> Xiangyu
>
> Xuannan Su  于2023年10月16日周一 10:25写道：
>
> > Congratulations Jane!
> >
> > Best,
> > Xuannan
> >
> > On Mon, Oct 16, 2023 at 10:21 AM Yun Tang  wrote:
> > >
> > > Congratulations, Jane!
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Rui Fan <1996fan...@gmail.com>
> > > Sent: Monday, October 16, 2023 10:16
> > > To: dev@flink.apache.org 
> > > Cc: qingyue@gmail.com 
> > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan
> > >
> > > Congratulations Jane!
> > >
> > > Best,
> > > Rui
> > >
> > > On Mon, Oct 16, 2023 at 10:15 AM yu zelin 
> wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Yu Zelin
> > > >
> > > > > 2023年10月16日 09:58，Jark Wu  写道：
> > > > >
> > > > > Hi, everyone
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce Jane Chan as a new
> > Flink
> > > > > Committer.
> > > > >
> > > > > Jane started code contribution in Jan 2021 and has been active in
> the
> > > > Flink
> > > > > community since. She authored more than 60 PRs and reviewed more
> > than 40
> > > > > PRs. Her contribution mainly revolves around Flink SQL, including
> > Plan
> > > > > Advice (FLIP-280), operator-level state TTL (FLIP-292), and ALTER
> > TABLE
> > > > > statements (FLINK-21634). Jane participated deeply in development
> > > > > discussions and also helped answer user question emails. Jane was
> > also a
> > > > > core contributor of Flink Table Store (now Paimon) when the project
> > was
> > > > in
> > > > > the early days.
> > > > >
> > > > > Please join me in congratulating Jane Chan for becoming a Flink
> > > > Committer!
> > > > >
> > > > > Best,
> > > > > Jark Wu (on behalf of the Flink PMC)
> > > >
> > > >
> >
>

Re: Flink and Flink shaded dependency

[Yuxin Tan]

Hi, devs,

I would like to revive the discussion again.

In our ARM environment, we encounter a compile error when
using Flink 1.17.

Flink 1.17 depends on flink-shaded 16.1, which uses netty 4.1.82.
However, flink-shaded 16.1 fails to compile in the ARM
environment. As a result, we are unable to compile Flink 1.17
due to this issue.

We have tested compiling flink-shaded using netty 4.1.83 or
a later version in ARM env, and it can compile successfully.

I believe this is a critical bug that needs to be addressed first.

Taking into consideration the previous discussions regarding
compatibility and the dependency of external connectors on
this version, I propose addressing the bug by only updating
flink-shaded's netty to a minor version (e.g., 4.1.83) rather than
backporting FLINK-32032. This approach can avoid introducing
other changes, such as updating to a higher netty version (4.1.91),
potential guava version alterations, Curator version modifications,
and so on.

WDYT?


Best,
Yuxin


Jing Ge  于2023年10月5日周四 14:39写道：

> Hi Chesnay,
>
> Thanks for joining this discussion and sharing your thoughts!
>
>
> > Connectors shouldn't depend on flink-shaded.
> >
>
> Perfect! We are on the same page. If you could read through the discussion,
> you would realize that, currently, there are many connectors depend on
> flink-shaded.
>
>
> > Connectors are small enough in scope that depending directly on
> > guava/jackson/etc. is a fine approach, and they have plenty of other
> > dependencies that they need to manage anyway; let's treat these the same
> > way.
> >
>
> It is even better, if we could do that. Jira tickest[1] are created.
>
>
> > As for class-loading, there has been a long-standing goal of each
> > connector being loaded in their own classloader. That still is the north
> > star and the only reasonable way to ensure that multiple connectors can
> > be safely used with SQL.
> >
>
> What is the current status? Do we have any Jira ticket for that?
>
> Best regards,
> Jing
>
> [1] https://issues.apache.org/jira/browse/FLINK-33190
>
> On Wed, Oct 4, 2023 at 4:43 PM Chesnay Schepler 
> wrote:
>
> > There is no "monolithic" flink-shaded dependency.
> > Connectors shouldn't depend on anything that Flink provides, but be
> > self-contained as Martijn pointed out.
> >
> >
>
> > Connectors shouldn't depend on flink-shaded.
> >
>
>
> > The overhead and/or risks of doing/supporting that right now far
> > outweigh the benefits.
> > ( Because we either have to encode the full version for all dependencies
> > into the package, or accept the risk of minor/patch dependency clashes)
> >
>
>
> > Connectors are small enough in scope that depending directly on
> > guava/jackson/etc. is a fine approach, and they have plenty of other
> > dependencies that they need to manage anyway; let's treat these the same
> > way.
> >
>
>
> > Naturally this is also an argument against flink-shaded-connectors; on
> > top of that we already experience repo creep and managing releases is
> > difficult enough as-is.
> >
> >
>
> > As for class-loading, there has been a long-standing goal of each
> > connector being loaded in their own classloader. That still is the north
> > star and the only reasonable way to ensure that multiple connectors can
> > be safely used with SQL.
> >
>
>
> > On 02/10/2023 18:32, Jing Ge wrote:
> > > Hi Sergey,
> > >
> > > Thanks for sharing your thoughts. It could somehow help but didn't get
> to
> > > the root of this issue.
> > >
> > > According to the documentation, Flink shaded is used to provide a
> single
> > > instance of a shaded dependency across sub-modules in Flink repo.
> Shaded
> > > namespaces should be used where shaded dependencies are configured.
> After
> > > connectors have been externalized, it ends up with more repos depending
> > on
> > > one shaded jar, e.g. guava. This is a "monolithic" dependency setup
> that
> > > makes it difficult to change the root(flink-shade), because any changes
> > of
> > > the root have to be propagated to all downstream repos. Even worse is
> > that
> > > not every downstream repo is known while modifying the root.
> > >
> > > Since all externalized connectors have their own repos and are not
> > > sub-modules of Flink anymore, I would suggest the following upgrade:
> > >
> > > 1. Connectors should use their own classloader instead of Flink's
> > > classloader. This will break the monolithic dependency. Connectors and
> > > Flink can use different versions of flink-shaded.
> > > 2. [optional] It would be even better that all connector repos depend
> on
> > > their own individual shaded repo, e.g. flink-connector-shaded.
> > flink-shaded
> > > should only be used by Flink.
> > >
> > > WDYT?
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > > On Thu, Sep 14, 2023 at 11:28 PM Sergey Nuyanzin 
> > > wrote:
> > >
> > >> Yes, that's a reasonable question, thanks for raising it.
> > >>
> > >> I think this is not only about flink-shaded, rather about dependencies
> > in

Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

[Yuxin Tan]

+1(non-binding)

Best,
Yuxin


Zhanghao Chen  于2023年10月13日周五 10:54写道：

> +1 (non-binding)
>
> Best,
> Zhanghao Chen
> 
> From: Junrui Lee 
> Sent: Friday, October 13, 2023 10:12
> To: dev@flink.apache.org 
> Subject: [VOTE] FLIP-366: Support standard YAML for FLINK configuration
>
> Hi all,
>
> Thank you to everyone for the feedback on FLIP-366[1]: Support standard
> YAML for FLINK configuration in the discussion thread [2].
> I would like to start a vote for it. The vote will be open for at least 72
> hours (excluding weekends, unless there is an objection or an insufficient
> number of votes).
>
> Thanks,
> Junrui
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
> [2]https://lists.apache.org/thread/qfhcm7h8r5xkv38rtxwkghkrcxg0q7k5
>

Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

[Yuxin Tan]

Hi, Junrui

+1 for the proposal.
Thanks for your effort.

Best,
Yuxin


Samrat Deb  于2023年9月22日周五 13:23写道：

> Hello Junrui,
>
> +1 for the proposal.
>
>
> Bests,
> Samrat
>
> On Fri, Sep 22, 2023 at 10:18 AM Shammon FY  wrote:
>
> > +1 for the proposal, thanks for driving.
> >
> > Bet,
> > Shammon FY
> >
> > On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo  wrote:
> >
> > > Thanks for driving this, +1 for the proposal.
> > >
> > > Best,
> > > Yangze Guo
> > >
> > >
> > > On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang 
> > > wrote:
> > > >
> > > > Hi Junrui,
> > > >
> > > > +1 for this proposal, thanks for driving.
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > ConradJam  于2023年9月22日周五 10:07写道：
> > > >
> > > > > +1 Support for standard YAML format facilitates specification
> > > > >
> > > > > Jing Ge  于2023年9月22日周五 02:23写道：
> > > > >
> > > > > > Hi Junrui,
> > > > > >
> > > > > > +1 for following the standard. Thanks for your effort!
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee 
> > > wrote:
> > > > > >
> > > > > > > Hi Jane,
> > > > > > >
> > > > > > > Thank you for your valuable feedback and suggestions.
> > > > > > > I agree with your point about differentiating between
> > > > > "flink-config.yaml"
> > > > > > > and "flink-conf.yaml" to determine the standard syntax at a
> > glance.
> > > > > > >
> > > > > > > While I understand your suggestion of using
> > > "flink-conf-default.yaml"
> > > > > to
> > > > > > > represent the default YAML file for Flink 1.x, I have been
> > > considering
> > > > > > > the option of using "flink-configuration.yaml" as the file name
> > > for the
> > > > > > > new configuration file.
> > > > > > > This name "flink-configuration.yaml" provides a clear
> distinction
> > > > > between
> > > > > > > the new and old configuration files based on their names, and
> it
> > > does
> > > > > not
> > > > > > > introduce any additional semantics. Moreover, this name
> > > > > > > "flink-configuration.yaml" can continue to be used in future
> > > versions
> > > > > > > FLINK-2.0.
> > > > > > >
> > > > > > > WDYT? If we can reach a consensus on this, I will update the
> FLIP
> > > > > > > documentation
> > > > > > > accordingly.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Junrui
> > > > > > >
> > > > > > > Jane Chan  于2023年9月20日周三 23:38写道：
> > > > > > >
> > > > > > > > Hi Junrui,
> > > > > > > >
> > > > > > > > Thanks for driving this FLIP. +1 for adoption of the standard
> > > YAML
> > > > > > > syntax.
> > > > > > > > I just have one minor suggestion. It's a little bit
> challenging
> > > to
> > > > > > > > differentiate between `flink-config.yaml` and
> `flink-conf.yaml`
> > > to
> > > > > > > > determine which one uses the standard syntax at a glance. How
> > > about
> > > > > > > > using `flink-conf-default.yaml` to represent the default yaml
> > > file
> > > > > for
> > > > > > > > Flink 1.x?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jane
> > > > > > > >
> > > > > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee <
> > jrlee@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi devs,
> > > > > > > > >
> > > > > > > > > I would like to start a discussion about FLIP-366:
> > > > > > > > > Support standard YAML for FLINK configuration[1]
> > > > > > > > >
> > > > > > > > > The current flink-conf.yaml parser in FLINK is not a
> standard
> > > YAML
> > > > > > > > parser,
> > > > > > > > > which has some shortcomings.
> > > > > > > > > Firstly, it does not support nested structure configuration
> > > items
> > > > > and
> > > > > > > > only
> > > > > > > > > supports key-value pairs, resulting in poor readability.
> > > Secondly,
> > > > > if
> > > > > > > the
> > > > > > > > > value is a collection type, such as a List or Map, users
> are
> > > > > required
> > > > > > > to
> > > > > > > > > write the value in a FLINK-specific pattern, which is
> > > inconvenient
> > > > > to
> > > > > > > > use.
> > > > > > > > > Additionally, the parser of FLINK has some differences in
> > > syntax
> > > > > > > compared
> > > > > > > > > to the standard YAML parser, such as the syntax for parsing
> > > > > comments
> > > > > > > and
> > > > > > > > > null values. These inconsistencies can cause confusion for
> > > users,
> > > > > as
> > > > > > > seen
> > > > > > > > > in FLINK-15358 and FLINK-32740.
> > > > > > > > >
> > > > > > > > > By supporting standard YAML, these issues can be resolved,
> > and
> > > > > users
> > > > > > > can
> > > > > > > > > create a Flink configuration file using third-party tools
> and
> > > > > > leverage
> > > > > > > > > some advanced YAML features. Therefore, we propose to
> support
> > > > > > standard
> > > > > > > > > YAML for FLINK configuration.
> > > > > > > > >
> > > > > > > > > You can find more details in the FLIP-366[1]. Looking
> forward
> > > to
> > > > > your
> > > > > > > > > feedback.
> > > > > > > > >
> > > > > > > > > [1]
> > 

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, Zakelly,

No benchmark tests currently are affected by this issue. We
may add benchmarks to guard it later. Thanks.

Best,
Yuxin


Zakelly Lan  于2023年9月21日周四 11:56写道：

> Hi Jing,
>
> Sure, I will run the benchmark with this fix.
>
> Hi Yunxin,
>
> I'm not familiar with the hybrid shuffle. Is there any specific
> benchmark test that may be affected by this issue? I will pay special
> attention to it.
> Thanks.
>
>
> Best,
> Zakelly
>
> On Thu, Sep 21, 2023 at 10:08 AM Yuxin Tan  wrote:
> >
> > Hi, Jing, Qingsheng,
> >
> > Thanks a lot.
> > The fix has been backported.
> >
> > Best,
> > Yuxin
> >
> >
> > Jing Ge  于2023年9月21日周四 00:42写道：
> >
> > > Hi Lijie,
> > >
> > > Thanks for reaching out. Please backport it to release-1.18.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang 
> > > wrote:
> > >
> > > > Hi community and release managers:
> > > >
> > > > We found a critical bug[1] of the rest client a few days ago, which
> may
> > > > cause the inode to be used up. Now the fix-PR[2] is ready for
> merging, I
> > > > hope to backport it to release-1.18.
> > > >
> > > > Please let me know if you have any concerns. Thanks.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-32974
> > > > [2] https://github.com/apache/flink/pull/23363
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Zakelly Lan  于2023年9月19日周二 17:26写道：
> > > >
> > > > > Hi Yuan and Jing,
> > > > >
> > > > > Thank you for sharing your thoughts. I completely agree that it is
> our
> > > > > top priority to ensure that there are no regressions from the last
> > > > > commit the previous benchmark pipeline covered to the final commit
> of
> > > > > this release. I will try to get this result first.
> > > > >
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Tue, Sep 19, 2023 at 4:55 PM Jing Ge  >
> > > > > wrote:
> > > > > >
> > > > > > Hi
> > > > > >
> > > > > > Thanks Zakelly and Yuan for your effort and update. Since we
> changed
> > > > the
> > > > > > hardware, IMHO, if we are able to reach a consensus in the
> community
> > > > that
> > > > > > there is no regression with the benchmarks, we could consider
> > > releasing
> > > > > rc1
> > > > > > without waiting for the new baseline scores which might take
> days.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei <
> yuanmei.w...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hey Zakelly,
> > > > > > >
> > > > > > > Thanks very much for the efforts to re-build the entire
> benchmark
> > > > > > > environment.
> > > > > > >
> > > > > > > As long as we have
> > > > > > > 1) the pipeline set up and ready (no need for the entire portal
> > > > ready),
> > > > > > > 2) get benchmark comparison numbers (comparing with the commit
> just
> > > > > before
> > > > > > > the benchmark pipeline is down) and
> > > > > > > 3) confirmed no-regression, it should be good enough.
> > > > > > >
> > > > > > > Thanks again!
> > > > > > >
> > > > > > > Best
> > > > > > > Yuan
> > > > > > >
> > > > > > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan <
> zakelly@gmail.com
> > > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I am working on rebuilding the benchmark pipeline and it's
> almost
> > > > > > > > done. However, due to the change in machines for
> benchmarking, I
> > > > will
> > > > > > > > need a few more days 

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, Jing, Qingsheng,

Thanks a lot.
The fix has been backported.

Best,
Yuxin


Jing Ge  于2023年9月21日周四 00:42写道：

> Hi Lijie,
>
> Thanks for reaching out. Please backport it to release-1.18.
>
> Best regards,
> Jing
>
> On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang 
> wrote:
>
> > Hi community and release managers:
> >
> > We found a critical bug[1] of the rest client a few days ago, which may
> > cause the inode to be used up. Now the fix-PR[2] is ready for merging, I
> > hope to backport it to release-1.18.
> >
> > Please let me know if you have any concerns. Thanks.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-32974
> > [2] https://github.com/apache/flink/pull/23363
> >
> > Best,
> > Lijie
> >
> > Zakelly Lan  于2023年9月19日周二 17:26写道：
> >
> > > Hi Yuan and Jing,
> > >
> > > Thank you for sharing your thoughts. I completely agree that it is our
> > > top priority to ensure that there are no regressions from the last
> > > commit the previous benchmark pipeline covered to the final commit of
> > > this release. I will try to get this result first.
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> > > wrote:
> > > >
> > > > Hi
> > > >
> > > > Thanks Zakelly and Yuan for your effort and update. Since we changed
> > the
> > > > hardware, IMHO, if we are able to reach a consensus in the community
> > that
> > > > there is no regression with the benchmarks, we could consider
> releasing
> > > rc1
> > > > without waiting for the new baseline scores which might take days.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> > > wrote:
> > > >
> > > > > Hey Zakelly,
> > > > >
> > > > > Thanks very much for the efforts to re-build the entire benchmark
> > > > > environment.
> > > > >
> > > > > As long as we have
> > > > > 1) the pipeline set up and ready (no need for the entire portal
> > ready),
> > > > > 2) get benchmark comparison numbers (comparing with the commit just
> > > before
> > > > > the benchmark pipeline is down) and
> > > > > 3) confirmed no-regression, it should be good enough.
> > > > >
> > > > > Thanks again!
> > > > >
> > > > > Best
> > > > > Yuan
> > > > >
> > > > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan  >
> > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > > > done. However, due to the change in machines for benchmarking, I
> > will
> > > > > > need a few more days to run tests and gather the baseline scores
> > for
> > > > > > further comparison. Once the pipeline is fully ready, we will
> > proceed
> > > > > > with the performance test for release 1.18.0.
> > > > > >
> > > > > > Please let me know if you have any concerns. Thank you all for
> your
> > > > > > patience.
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge
>  > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> > > currently
> > > > > > for
> > > > > > > preview only to facilitate the integrated testing since the
> > > benchmark
> > > > > > tests
> > > > > > > are not available yet[1] and the release announcement is still
> > > under
> > > > > > > review. The RC1 will be released after all benchmarks tests are
> > > passed.
> > > > > > The
> > > > > > > related voting process will be triggered once the announcement
> is
> > > > > ready.
> > > > > > > The RC0 has all the artifacts that we would typically have for
> a
> > > > > release,
> > > > > > > except for the release note and the website pull request for
> the
> > > > > release
> > > > > > > announcement.
> > > > > > >
> > > > > > > The following contents are available for your review:
> > > > > > >
> > > > > > > - The preview source release and binary convenience releases
> [2],
> > > which
> > > > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6
> > [3].
> > > > > > > - all artifacts that would normally be deployed to the Maven
> > > > > > > Central Repository [4].
> > > > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > > > >
> > > > > > > Your help testing the release will be greatly appreciated! And
> > > we'll
> > > > > > > create the rc1 release and the voting thread as soon as all the
> > > efforts
> > > > > > are
> > > > > > > finished.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > > > [2]
> > https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > [4]
> > > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > > > [5]
> > > https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Qingsheng, Sergei, Konstantin and Jing
> > > > > >
> > > > >
> 

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing this up.

When testing the release candidate #0 for the batch scenario, I found an
issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by
1.18 and may significantly impact the performance of shuffle writing.

The fix tickit FLINK-33044[1] has been reviewed and approved. To prevent
any performance issues, I would like to backport the fix to version 1.18
unless there are any objections.

Glad to hear your suggestions.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin


Zakelly Lan  于2023年9月19日周二 17:26写道：

> Hi Yuan and Jing,
>
> Thank you for sharing your thoughts. I completely agree that it is our
> top priority to ensure that there are no regressions from the last
> commit the previous benchmark pipeline covered to the final commit of
> this release. I will try to get this result first.
>
>
> Best,
> Zakelly
>
> On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> wrote:
> >
> > Hi
> >
> > Thanks Zakelly and Yuan for your effort and update. Since we changed the
> > hardware, IMHO, if we are able to reach a consensus in the community that
> > there is no regression with the benchmarks, we could consider releasing
> rc1
> > without waiting for the new baseline scores which might take days.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> wrote:
> >
> > > Hey Zakelly,
> > >
> > > Thanks very much for the efforts to re-build the entire benchmark
> > > environment.
> > >
> > > As long as we have
> > > 1) the pipeline set up and ready (no need for the entire portal ready),
> > > 2) get benchmark comparison numbers (comparing with the commit just
> before
> > > the benchmark pipeline is down) and
> > > 3) confirmed no-regression, it should be good enough.
> > >
> > > Thanks again!
> > >
> > > Best
> > > Yuan
> > >
> > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > done. However, due to the change in machines for benchmarking, I will
> > > > need a few more days to run tests and gather the baseline scores for
> > > > further comparison. Once the pipeline is fully ready, we will proceed
> > > > with the performance test for release 1.18.0.
> > > >
> > > > Please let me know if you have any concerns. Thank you all for your
> > > > patience.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> currently
> > > > for
> > > > > preview only to facilitate the integrated testing since the
> benchmark
> > > > tests
> > > > > are not available yet[1] and the release announcement is still
> under
> > > > > review. The RC1 will be released after all benchmarks tests are
> passed.
> > > > The
> > > > > related voting process will be triggered once the announcement is
> > > ready.
> > > > > The RC0 has all the artifacts that we would typically have for a
> > > release,
> > > > > except for the release note and the website pull request for the
> > > release
> > > > > announcement.
> > > > >
> > > > > The following contents are available for your review:
> > > > >
> > > > > - The preview source release and binary convenience releases [2],
> which
> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > > - all artifacts that would normally be deployed to the Maven
> > > > > Central Repository [4].
> > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > >
> > > > > Your help testing the release will be greatly appreciated! And
> we'll
> > > > > create the rc1 release and the voting thread as soon as all the
> efforts
> > > > are
> > > > > finished.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > >
> > > > > Best regards,
> > > > > Qingsheng, Sergei, Konstantin and Jing
> > > >
> > >
>

Re: [DISCUSS] Backport fix to release 1.18

[Yuxin Tan]

Hi, Jing

Thanks for the update. The delay is indeed long.

Please ignore it and join the discussion in the original thread.

Best,
Yuxin


Jing Ge  于2023年9月20日周三 22:01写道：

> Hi folks,
>
> Please ignore this email. It seems the dev mail server has some issues
> that emails have been sent but will be received by dev@flink.apache.org
> with a big delay at [1]. Participants who have been sent to or cc directly
> should get the email instantly. Let's consolidate the discussion in this
> thread [1].
>
> Best regards,
> Jing
>
>
> [1] https://lists.apache.org/thread/5x28rp3zct4p603hm4zdwx6kfr101w38
>
> On Wed, Sep 20, 2023 at 11:43 AM Yuxin Tan  wrote:
>
>>
>> Hi, dear community and release managers,
>>
>> Thanks for bringing up the new release candidate for 1.18.
>>
>> When testing for the batch scenario, I found an issue of frequent
>> flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and
>> may significantly impact the performance of shuffle writing.
>>
>> The fix ticket FLINK-33044[1] has been reviewed and approved. To
>> prevent the performance issues, I would like to backport the fix to
>> release
>> 1.18 unless there are any objections.
>>
>> Glad to hear your suggestions.
>>
>> In addition to this, sorry to start a new email thread to discuss it.
>> I attempted many times to send this email within the existing thread
>> titled "[ANNOUNCE] Release 1.18.0, release candidate #0", but the mail
>> failed to be delivered, that thread looks broken. It is very strange.
>> So I start the new discussion thread.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-33044
>>
>> Best,
>> Yuxin
>>
>

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing this up.

When testing the release candidate #0 for the batch scenario, I found an
issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by
1.18 and may significantly impact the performance of shuffle writing.

The fix ticket FLINK-33044[1] has been reviewed and approved. To prevent
any performance issues, I would like to backport the fix to version 1.18
unless there are any objections.

Glad to hear your suggestions.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin


Zakelly Lan  于2023年9月19日周二 17:26写道：

> Hi Yuan and Jing,
>
> Thank you for sharing your thoughts. I completely agree that it is our
> top priority to ensure that there are no regressions from the last
> commit the previous benchmark pipeline covered to the final commit of
> this release. I will try to get this result first.
>
>
> Best,
> Zakelly
>
> On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> wrote:
> >
> > Hi
> >
> > Thanks Zakelly and Yuan for your effort and update. Since we changed the
> > hardware, IMHO, if we are able to reach a consensus in the community that
> > there is no regression with the benchmarks, we could consider releasing
> rc1
> > without waiting for the new baseline scores which might take days.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> wrote:
> >
> > > Hey Zakelly,
> > >
> > > Thanks very much for the efforts to re-build the entire benchmark
> > > environment.
> > >
> > > As long as we have
> > > 1) the pipeline set up and ready (no need for the entire portal ready),
> > > 2) get benchmark comparison numbers (comparing with the commit just
> before
> > > the benchmark pipeline is down) and
> > > 3) confirmed no-regression, it should be good enough.
> > >
> > > Thanks again!
> > >
> > > Best
> > > Yuan
> > >
> > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > done. However, due to the change in machines for benchmarking, I will
> > > > need a few more days to run tests and gather the baseline scores for
> > > > further comparison. Once the pipeline is fully ready, we will proceed
> > > > with the performance test for release 1.18.0.
> > > >
> > > > Please let me know if you have any concerns. Thank you all for your
> > > > patience.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> currently
> > > > for
> > > > > preview only to facilitate the integrated testing since the
> benchmark
> > > > tests
> > > > > are not available yet[1] and the release announcement is still
> under
> > > > > review. The RC1 will be released after all benchmarks tests are
> passed.
> > > > The
> > > > > related voting process will be triggered once the announcement is
> > > ready.
> > > > > The RC0 has all the artifacts that we would typically have for a
> > > release,
> > > > > except for the release note and the website pull request for the
> > > release
> > > > > announcement.
> > > > >
> > > > > The following contents are available for your review:
> > > > >
> > > > > - The preview source release and binary convenience releases [2],
> which
> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > > - all artifacts that would normally be deployed to the Maven
> > > > > Central Repository [4].
> > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > >
> > > > > Your help testing the release will be greatly appreciated! And
> we'll
> > > > > create the rc1 release and the voting thread as soon as all the
> efforts
> > > > are
> > > > > finished.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > >
> > > > > Best regards,
> > > > > Qingsheng, Sergei, Konstantin and Jing
> > > >
> > >
>

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing this up.

When testing the release candidate #0 for the batch scenario, I found an
issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by
1.18 and may significantly impact the performance of shuffle writing.

The fix tickit FLINK-33044[1] has been reviewed and approved. To prevent
any performance issues, I would like to backport the fix to version 1.18
unless there are any objections.

Glad to hear your suggestions.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin


Zakelly Lan  于2023年9月19日周二 17:26写道：

> Hi Yuan and Jing,
>
> Thank you for sharing your thoughts. I completely agree that it is our
> top priority to ensure that there are no regressions from the last
> commit the previous benchmark pipeline covered to the final commit of
> this release. I will try to get this result first.
>
>
> Best,
> Zakelly
>
> On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> wrote:
> >
> > Hi
> >
> > Thanks Zakelly and Yuan for your effort and update. Since we changed the
> > hardware, IMHO, if we are able to reach a consensus in the community that
> > there is no regression with the benchmarks, we could consider releasing
> rc1
> > without waiting for the new baseline scores which might take days.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> wrote:
> >
> > > Hey Zakelly,
> > >
> > > Thanks very much for the efforts to re-build the entire benchmark
> > > environment.
> > >
> > > As long as we have
> > > 1) the pipeline set up and ready (no need for the entire portal ready),
> > > 2) get benchmark comparison numbers (comparing with the commit just
> before
> > > the benchmark pipeline is down) and
> > > 3) confirmed no-regression, it should be good enough.
> > >
> > > Thanks again!
> > >
> > > Best
> > > Yuan
> > >
> > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > done. However, due to the change in machines for benchmarking, I will
> > > > need a few more days to run tests and gather the baseline scores for
> > > > further comparison. Once the pipeline is fully ready, we will proceed
> > > > with the performance test for release 1.18.0.
> > > >
> > > > Please let me know if you have any concerns. Thank you all for your
> > > > patience.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> currently
> > > > for
> > > > > preview only to facilitate the integrated testing since the
> benchmark
> > > > tests
> > > > > are not available yet[1] and the release announcement is still
> under
> > > > > review. The RC1 will be released after all benchmarks tests are
> passed.
> > > > The
> > > > > related voting process will be triggered once the announcement is
> > > ready.
> > > > > The RC0 has all the artifacts that we would typically have for a
> > > release,
> > > > > except for the release note and the website pull request for the
> > > release
> > > > > announcement.
> > > > >
> > > > > The following contents are available for your review:
> > > > >
> > > > > - The preview source release and binary convenience releases [2],
> which
> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > > - all artifacts that would normally be deployed to the Maven
> > > > > Central Repository [4].
> > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > >
> > > > > Your help testing the release will be greatly appreciated! And
> we'll
> > > > > create the rc1 release and the voting thread as soon as all the
> efforts
> > > > are
> > > > > finished.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > >
> > > > > Best regards,
> > > > > Qingsheng, Sergei, Konstantin and Jing
> > > >
> > >
>

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing this up.

When testing the release candidate #0 for the batch scenario, I found an
issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by
1.18 and may significantly impact the performance of shuffle writing.

The fix tickit FLINK-33044[1] has been reviewed and approved. To prevent
any performance issues, I would like to backport the fix to version 1.18
unless there are any objections.

Glad to hear your suggestions.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin


Zakelly Lan  于2023年9月19日周二 17:26写道：

> Hi Yuan and Jing,
>
> Thank you for sharing your thoughts. I completely agree that it is our
> top priority to ensure that there are no regressions from the last
> commit the previous benchmark pipeline covered to the final commit of
> this release. I will try to get this result first.
>
>
> Best,
> Zakelly
>
> On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> wrote:
> >
> > Hi
> >
> > Thanks Zakelly and Yuan for your effort and update. Since we changed the
> > hardware, IMHO, if we are able to reach a consensus in the community that
> > there is no regression with the benchmarks, we could consider releasing
> rc1
> > without waiting for the new baseline scores which might take days.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> wrote:
> >
> > > Hey Zakelly,
> > >
> > > Thanks very much for the efforts to re-build the entire benchmark
> > > environment.
> > >
> > > As long as we have
> > > 1) the pipeline set up and ready (no need for the entire portal ready),
> > > 2) get benchmark comparison numbers (comparing with the commit just
> before
> > > the benchmark pipeline is down) and
> > > 3) confirmed no-regression, it should be good enough.
> > >
> > > Thanks again!
> > >
> > > Best
> > > Yuan
> > >
> > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > done. However, due to the change in machines for benchmarking, I will
> > > > need a few more days to run tests and gather the baseline scores for
> > > > further comparison. Once the pipeline is fully ready, we will proceed
> > > > with the performance test for release 1.18.0.
> > > >
> > > > Please let me know if you have any concerns. Thank you all for your
> > > > patience.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> currently
> > > > for
> > > > > preview only to facilitate the integrated testing since the
> benchmark
> > > > tests
> > > > > are not available yet[1] and the release announcement is still
> under
> > > > > review. The RC1 will be released after all benchmarks tests are
> passed.
> > > > The
> > > > > related voting process will be triggered once the announcement is
> > > ready.
> > > > > The RC0 has all the artifacts that we would typically have for a
> > > release,
> > > > > except for the release note and the website pull request for the
> > > release
> > > > > announcement.
> > > > >
> > > > > The following contents are available for your review:
> > > > >
> > > > > - The preview source release and binary convenience releases [2],
> which
> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > > - all artifacts that would normally be deployed to the Maven
> > > > > Central Repository [4].
> > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > >
> > > > > Your help testing the release will be greatly appreciated! And
> we'll
> > > > > create the rc1 release and the voting thread as soon as all the
> efforts
> > > > are
> > > > > finished.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > >
> > > > > Best regards,
> > > > > Qingsheng, Sergei, Konstantin and Jing
> > > >
> > >
>

[DISCUSS] Backport fix to release 1.18

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing up the new release candidate for 1.18.

When testing for the batch scenario, I found an issue of frequent
flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and
may significantly impact the performance of shuffle writing.

The fix ticket FLINK-33044[1] has been reviewed and approved. To
prevent the performance issues, I would like to backport the fix to release
1.18 unless there are any objections.

Glad to hear your suggestions.

In addition to this, sorry to start a new email thread to discuss it.
I attempted many times to send this email within the existing thread
titled "[ANNOUNCE] Release 1.18.0, release candidate #0", but the mail
failed to be delivered, that thread looks broken. It is very strange.
So I start the new discussion thread.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuxin Tan]

Hi, dear community and release managers,

Thanks for bringing this up.

When testing the release candidate #0 for the batch scenario, I found an
issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by
1.18 and may significantly impact the performance of shuffle writing.

The fix ticket FLINK-33044[1] has been reviewed and approved. To prevent
any performance issues, I would like to backport the fix to version 1.18
unless there are any objections.

Glad to hear your suggestions.

[1] https://issues.apache.org/jira/browse/FLINK-33044

Best,
Yuxin

Zakelly Lan  于2023年9月19日周二 17:26写道：

> Hi Yuan and Jing,
>
> Thank you for sharing your thoughts. I completely agree that it is our
> top priority to ensure that there are no regressions from the last
> commit the previous benchmark pipeline covered to the final commit of
> this release. I will try to get this result first.
>
>
> Best,
> Zakelly
>
> On Tue, Sep 19, 2023 at 4:55 PM Jing Ge 
> wrote:
> >
> > Hi
> >
> > Thanks Zakelly and Yuan for your effort and update. Since we changed the
> > hardware, IMHO, if we are able to reach a consensus in the community that
> > there is no regression with the benchmarks, we could consider releasing
> rc1
> > without waiting for the new baseline scores which might take days.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei 
> wrote:
> >
> > > Hey Zakelly,
> > >
> > > Thanks very much for the efforts to re-build the entire benchmark
> > > environment.
> > >
> > > As long as we have
> > > 1) the pipeline set up and ready (no need for the entire portal ready),
> > > 2) get benchmark comparison numbers (comparing with the commit just
> before
> > > the benchmark pipeline is down) and
> > > 3) confirmed no-regression, it should be good enough.
> > >
> > > Thanks again!
> > >
> > > Best
> > > Yuan
> > >
> > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I am working on rebuilding the benchmark pipeline and it's almost
> > > > done. However, due to the change in machines for benchmarking, I will
> > > > need a few more days to run tests and gather the baseline scores for
> > > > further comparison. Once the pipeline is fully ready, we will proceed
> > > > with the performance test for release 1.18.0.
> > > >
> > > > Please let me know if you have any concerns. Thank you all for your
> > > > patience.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > The RC0 for Apache Flink 1.18.0 has been created. This RC is
> currently
> > > > for
> > > > > preview only to facilitate the integrated testing since the
> benchmark
> > > > tests
> > > > > are not available yet[1] and the release announcement is still
> under
> > > > > review. The RC1 will be released after all benchmarks tests are
> passed.
> > > > The
> > > > > related voting process will be triggered once the announcement is
> > > ready.
> > > > > The RC0 has all the artifacts that we would typically have for a
> > > release,
> > > > > except for the release note and the website pull request for the
> > > release
> > > > > announcement.
> > > > >
> > > > > The following contents are available for your review:
> > > > >
> > > > > - The preview source release and binary convenience releases [2],
> which
> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > > > > - all artifacts that would normally be deployed to the Maven
> > > > > Central Repository [4].
> > > > > - source code tag "release-1.18.0-rc0" [5]
> > > > >
> > > > > Your help testing the release will be greatly appreciated! And
> we'll
> > > > > create the rc1 release and the voting thread as soon as all the
> efforts
> > > > are
> > > > > finished.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> > > > >
> > > > > Best regards,
> > > > > Qingsheng, Sergei, Konstantin and Jing
> > > >
> > >
>

Re: Re: [DISCUSS] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

[Yuxin Tan]

Hi, Dong,
Thanks for your efforts.

+1 to this proposal,
I believe this will improve the performance in some mixture circumstances
of bounded and unbounded workloads.

Best,
Yuxin


Xintong Song  于2023年9月18日周一 10:56写道：

> Thanks for addressing my comments, Dong.
>
> LGTM.
>
> Best,
>
> Xintong
>
>
>
> On Sat, Sep 16, 2023 at 3:34 PM Wencong Liu  wrote:
>
> > Hi Dong & Jinhao,
> >
> > Thanks for your clarification! +1
> >
> > Best regards,
> > Wencong
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2023-09-15 11:26:16, "Dong Lin"  wrote:
> > >Hi Wencong,
> > >
> > >Thanks for your comments! Please see my reply inline.
> > >
> > >On Thu, Sep 14, 2023 at 12:30 PM Wencong Liu 
> > wrote:
> > >
> > >> Dear Dong,
> > >>
> > >> I have thoroughly reviewed the proposal for FLIP-331 and believe it
> > would
> > >> be
> > >> a valuable addition to Flink. However, I do have a few questions that
> I
> > >> would
> > >> like to discuss:
> > >>
> > >>
> > >> 1. The FLIP-331 proposed the EndOfStreamWindows that is implemented by
> > >> TimeWindow with maxTimestamp = (Long.MAX_VALUE - 1), which naturally
> > >> supports WindowedStream and AllWindowedStream to process all records
> > >> belonging to a key in a 'global' window under both STREAMING and BATCH
> > >> runtime execution mode.
> > >>
> > >>
> > >> However, besides coGroup and keyBy().aggregate(), other operators on
> > >> WindowedStream and AllWindowedStream, such as join/reduce, etc,
> > currently
> > >> are still implemented based on WindowOperator.
> > >>
> > >>
> > >> In fact, these operators can also be implemented without using
> > >> WindowOperator
> > >> to prevent additional WindowAssigner#assignWindows or
> > >> triggerContext#onElement
> > >> invocation cost. Will there be plans to support these operators in the
> > >> future?
> > >>
> > >
> > >You are right. The EndOfStreamWindows proposed in this FLIP can
> > potentially
> > >benefit any DataStream API that takes WindowAssigner as parameters. This
> > >can involve more operations than aggregate and co-group.
> > >
> > >And yes, we have plans to take advantage of this API to optimize these
> > >operators in the future. This FLIP focuses on the introduction of the
> > >public APIs and uses aggregate/co-group as the first two examples to
> > >show-case the performance benefits.
> > >
> > >I have added a "Analysis of APIs affected by this FLIP" to list the
> > >DataStream APIs that can benefit from this FLIP. Would this answer your
> > >question?
> > >
> > >
> > >>
> > >> 2. When using EndOfStreamWindows, upstream operators no longer support
> > >> checkpointing. This limit may be too strict, especially when dealing
> > with
> > >> bounded data in streaming runtime execution mode, where checkpointing
> > >> can still be useful.
> > >>
> > >
> > >I am not sure we have a good way to support checkpoint while still
> > >achieving the performance improves targeted by this FLIP.
> > >
> > >The issue here is that if we support checkpoint, then we can not take
> > >advantage of algorithms (e.g. sorting inputs using ExternalSorter) that
> > are
> > >not compatible with checkpoints. These algorithms (which do not support
> > >checkpoint) are the main reasons why batch mode currently significantly
> > >outperforms stream mode in doing aggregation/cogroup etc.
> > >
> > >In most cases where the user does not care about processing latency, it
> is
> > >generally preferred to use batch mode to perform aggregation operations
> > >(which should be 10X faster than the existing stream mode performance)
> > >instead of doing checkpoint.
> > >
> > >Also note that we can still let operators perform failover in the same
> as
> > >the existing batch mode execution, where the intermediate results
> > (produced
> > >by one operator) can be persisted in shuffle service and downstream
> > >operators can re-read those data from shuffle service after failover.
> > >
> > >
> > >>
> > >> 3. The proposal mentions that if a transformation has isOutputOnEOF ==
> > >> true, the
> > >> operator as well as its upstream operators will be executed in 'batch
> > >> mode' with
> > >> checkpointing disabled. I would like to understand the specific
> > >> implications of this
> > >> 'batch mode' and if there are any other changes associated with it?
> > >
> > >
> > >Good point. We should explicitly mention the changes. I have updated the
> > >FLIP to clarify this.
> > >
> > >More specifically, the checkpoint is disabled when these operators are
> > >running, such that these operators can do operations not compatible with
> > >checkpoints (e.g. sorting inputs). And operators should re-read the data
> > >from the upstream blocking edge or sources after failover.
> > >
> > >Would this answer your question?
> > >
> > >
> > >>
> > >> Additionally, I am curious to know if this 'batch mode' conflicts with
> > the
> > >> 'mix mode'
> > >>
> > >> described in FLIP-327. While the coGroup and keyBy().aggregate()
> > operators
> > 

[jira] [Created] (FLINK-33103) Hybrid shuffle ITCase supports the new mode

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33103:
-

 Summary: Hybrid shuffle ITCase supports the new mode
 Key: FLINK-33103
 URL: https://issues.apache.org/jira/browse/FLINK-33103
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Affects Versions: 1.18.1
Reporter: Yuxin Tan


Currently, the Hybrid shuffle ITCase only supports the legacy mode. The new 
mode should also be verified, so we should improve it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33088) Fix NullPointerException in RemoteTierConsumerAgent of tiered storage

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33088:
-

 Summary: Fix NullPointerException in RemoteTierConsumerAgent of 
tiered storage
 Key: FLINK-33088
 URL: https://issues.apache.org/jira/browse/FLINK-33088
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


Currently, when getting a buffer from RemoteTierConsumerAgent of tiered 
storage, a NullPointerException may be thrown, we should fix it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-357: Deprecate Iteration API of DataStream

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Xintong Song  于2023年9月14日周四 17:14写道：

> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Thu, Sep 14, 2023 at 3:48 PM Jing Ge 
> wrote:
>
> > +1(binding)
> >
> > Best regards,
> > Jing
> >
> > On Thu, Sep 14, 2023 at 7:31 AM Dong Lin  wrote:
> >
> > > Thanks Wencong for the FLIP.
> > >
> > > +1 (binding)
> > >
> > > On Thu, Sep 14, 2023 at 12:36 PM Wencong Liu 
> > wrote:
> > >
> > > > Hi dev,
> > > >
> > > >
> > > > I'd like to start a vote on FLIP-357.
> > > >
> > > >
> > > > Discussion thread:
> > > > https://lists.apache.org/thread/shf77phc0wzlbj06jsfj3nclxnm2mrv5
> > > > FLIP:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-357%3A+Deprecate+Iteration+API+of+DataStream
> > > >
> > > >
> > > > Best regards,
> > > > Wencong Liu
> > >
> >
>

[jira] [Created] (FLINK-33044) Reduce the frequency of triggering flush for the disk tier of the tiered storage

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-33044:
-

 Summary: Reduce the frequency of triggering flush for the disk 
tier of the tiered storage
 Key: FLINK-33044
 URL: https://issues.apache.org/jira/browse/FLINK-33044
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


The disk cache of tiered storage will flush at the end of each subpartition's 
segment, which is too frequent and is bad for performance. We should improve it 
with some better flushing methods, e.g. flushing buffers with batch.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32870) Merge multiple small buffers into one large buffer when loading data for tiered storage

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32870:
-

 Summary: Merge multiple small buffers into one large buffer when 
loading data for tiered storage
 Key: FLINK-32870
 URL: https://issues.apache.org/jira/browse/FLINK-32870
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan


Currently, when the file reader of tiered storage loads data from the disk 
file, it reads data in buffer granularity. Before compression, each buffer is 
32K by default, after compression the size will become smaller (may less than 
5K), which is pretty small for the network buffer and the file IO. 
We should merge the multiple small buffers into a larger one to decrease the 
buffer competition and the file IO, leading to better performance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink Committer - Hangxiang Yu

[Yuxin Tan]

Congrats, Hangxiang!

Best,
Yuxin


weijie guo  于2023年8月7日周一 17:59写道：

> Congrats, Hangxiang!
>
> Best regards,
>
> Weijie
>
>
> Biao Geng  于2023年8月7日周一 17:04写道：
>
> > Congrats, Hangxiang!
> > Best,
> > Biao Geng
> >
> >
> > 发送自 Outlook for iOS
> > 
> > 发件人: Qingsheng Ren 
> > 发送时间: Monday, August 7, 2023 4:23:11 PM
> > 收件人: dev@flink.apache.org 
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Hangxiang Yu
> >
> > Congratulations and welcome aboard, Hangxiang!
> >
> > Best,
> > Qingsheng
> >
> > On Mon, Aug 7, 2023 at 4:19 PM Matthias Pohl  > .invalid>
> > wrote:
> >
> > > Congratulations, Hangxiang! :)
> > >
> > > On Mon, Aug 7, 2023 at 10:01 AM Junrui Lee 
> wrote:
> > >
> > > > Congratulations, Hangxiang!
> > > >
> > > > Best,
> > > > Junrui
> > > >
> > > > Yun Tang  于2023年8月7日周一 15:19写道：
> > > >
> > > > > Congratulations, Hangxiang!
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > > 
> > > > > From: Danny Cranmer 
> > > > > Sent: Monday, August 7, 2023 15:11
> > > > > To: dev 
> > > > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Hangxiang Yu
> > > > >
> > > > > Congrats Hangxiang! Welcome to the team.
> > > > >
> > > > > Danny.
> > > > >
> > > > > On Mon, 7 Aug 2023, 08:04 Rui Fan, <1996fan...@gmail.com> wrote:
> > > > >
> > > > > > Congratulations Hangxiang!
> > > > > >
> > > > > > Best,
> > > > > > Rui
> > > > > >
> > > > > > On Mon, Aug 7, 2023 at 2:58 PM Yuan Mei 
> > > > wrote:
> > > > > >
> > > > > > > On behalf of the PMC, I'm happy to announce Hangxiang Yu as a
> new
> > > > Flink
> > > > > > > Committer.
> > > > > > >
> > > > > > > Hangxiang has been active in the Flink community for more than
> > 1.5
> > > > > years
> > > > > > > and has played an important role in developing and maintaining
> > > State
> > > > > and
> > > > > > > Checkpoint related features/components, including Generic
> > > Incremental
> > > > > > > Checkpoints (take great efforts to make the feature
> prod-ready).
> > > > > > Hangxiang
> > > > > > > is also the main driver of the FLIP-263: Resolving schema
> > > > > compatibility.
> > > > > > >
> > > > > > > Hangxiang is passionate about the Flink community. Besides the
> > > > > technical
> > > > > > > contribution above, he is also actively promoting Flink: talks
> > > about
> > > > > > > Generic
> > > > > > > Incremental Checkpoints in Flink Forward and Meet-up. Hangxiang
> > > also
> > > > > > spent
> > > > > > > a good amount of time supporting users, participating in
> > > Jira/mailing
> > > > > > list
> > > > > > > discussions, and reviewing code.
> > > > > > >
> > > > > > > Please join me in congratulating Hangxiang for becoming a Flink
> > > > > > Committer!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yuan Mei (on behalf of the Flink PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yanfei Lei

[Yuxin Tan]

Congrats, Yanfei!

Best,
Yuxin


weijie guo  于2023年8月7日周一 17:59写道：

> Congrats, Yanfei!
>
> Best regards,
>
> Weijie
>
>
> Biao Geng  于2023年8月7日周一 17:03写道：
>
> > Congrats, Yanfei!
> > Best,
> > Biao Geng
> >
> > 发送自 Outlook for iOS
> > 
> > 发件人: Qingsheng Ren 
> > 发送时间: Monday, August 7, 2023 4:23:52 PM
> > 收件人: dev@flink.apache.org 
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Yanfei Lei
> >
> > Congratulations and welcome, Yanfei!
> >
> > Best,
> > Qingsheng
> >
> > On Mon, Aug 7, 2023 at 4:19 PM Matthias Pohl  > .invalid>
> > wrote:
> >
> > > Congratulations, Yanfei! :)
> > >
> > > On Mon, Aug 7, 2023 at 10:00 AM Junrui Lee 
> wrote:
> > >
> > > > Congratulations Yanfei!
> > > >
> > > > Best,
> > > > Junrui
> > > >
> > > > Yun Tang  于2023年8月7日周一 15:19写道：
> > > >
> > > > > Congratulations, Yanfei!
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > > 
> > > > > From: Danny Cranmer 
> > > > > Sent: Monday, August 7, 2023 15:10
> > > > > To: dev 
> > > > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Yanfei Lei
> > > > >
> > > > > Congrats Yanfei! Welcome to the team.
> > > > >
> > > > > Danny
> > > > >
> > > > > On Mon, 7 Aug 2023, 08:03 Rui Fan, <1996fan...@gmail.com> wrote:
> > > > >
> > > > > > Congratulations Yanfei!
> > > > > >
> > > > > > Best,
> > > > > > Rui
> > > > > >
> > > > > > On Mon, Aug 7, 2023 at 2:56 PM Yuan Mei 
> > > > wrote:
> > > > > >
> > > > > > > On behalf of the PMC, I'm happy to announce Yanfei Lei as a new
> > > Flink
> > > > > > > Committer.
> > > > > > >
> > > > > > > Yanfei has been active in the Flink community for almost two
> > years
> > > > and
> > > > > > has
> > > > > > > played an important role in developing and maintaining State
> and
> > > > > > Checkpoint
> > > > > > > related features/components, including RocksDB Rescaling
> > > Performance
> > > > > > > Improvement and Generic Incremental Checkpoints.
> > > > > > >
> > > > > > > Yanfei also helps improve community infrastructure in many
> ways,
> > > > > > including
> > > > > > > migrating the Flink Daily performance benchmark to the Apache
> > Flink
> > > > > slack
> > > > > > > channel. She is the maintainer of the benchmark and has
> improved
> > > its
> > > > > > > detection stability significantly. She is also one of the major
> > > > > > maintainers
> > > > > > > of the FrocksDB Repo and released FRocksDB 6.20.3 (part of
> Flink
> > > 1.17
> > > > > > > release). Yanfei is a very active community member, supporting
> > > users
> > > > > and
> > > > > > > participating
> > > > > > > in tons of discussions on the mailing lists.
> > > > > > >
> > > > > > > Please join me in congratulating Yanfei for becoming a Flink
> > > > Committer!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yuan Mei (on behalf of the Flink PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Hong Teoh

[Yuxin Tan]

Congratulations, Hong!

Best,
Yuxin


Sergey Nuyanzin  于2023年8月4日周五 14:24写道：

> Congratulations, Hong!
>
> On Fri, Aug 4, 2023 at 7:25 AM Shammon FY  wrote:
>
> > Congratulations, Hong!
> >
> > Best,
> > Shammon FY
> >
> > On Fri, Aug 4, 2023 at 12:33 PM Jing Ge 
> > wrote:
> >
> > > congrats! Hong!
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Fri, Aug 4, 2023 at 11:48 AM Qingsheng Ren 
> > wrote:
> > >
> > > > Congratulations and welcome aboard, Hong!
> > > >
> > > > Best,
> > > > Qingsheng
> > > >
> > > > On Fri, Aug 4, 2023 at 11:04 AM Matt Wang  wrote:
> > > >
> > > > > Congratulations, Hong!
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best,
> > > > > Matt Wang
> > > > >
> > > > >
> > > > >  Replied Message 
> > > > > | From | Weihua Hu |
> > > > > | Date | 08/4/2023 10:55 |
> > > > > | To |  |
> > > > > | Subject | Re: [ANNOUNCE] New Apache Flink Committer - Hong Teoh |
> > > > > Congratulations, Hong!
> > > > >
> > > > > Best,
> > > > > Weihua
> > > > >
> > > > >
> > > > > On Fri, Aug 4, 2023 at 10:49 AM Samrat Deb 
> > > > wrote:
> > > > >
> > > > > Congratulations, Hong Teoh
> > > > >
> > > > > On Fri, 4 Aug 2023 at 7:52 AM, Benchao Li 
> > > wrote:
> > > > >
> > > > > Congratulations, Hong!
> > > > >
> > > > > yuxia  于2023年8月4日周五 09:23写道：
> > > > >
> > > > > Congratulations, Hong Teoh!
> > > > >
> > > > > Best regards,
> > > > > Yuxia
> > > > >
> > > > > - 原始邮件 -
> > > > > 发件人: "Matthias Pohl" 
> > > > > 收件人: "dev" 
> > > > > 发送时间: 星期四, 2023年 8 月 03日 下午 11:24:39
> > > > > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Hong Teoh
> > > > >
> > > > > Congratulations, Hong! :)
> > > > >
> > > > > On Thu, Aug 3, 2023 at 3:39 PM Leonard Xu 
> wrote:
> > > > >
> > > > > Congratulations, Hong!
> > > > >
> > > > >
> > > > > Best,
> > > > > Leonard
> > > > >
> > > > > On Aug 3, 2023, at 8:45 PM, Jiabao Sun  > > > > .INVALID>
> > > > > wrote:
> > > > >
> > > > > Congratulations, Hong Teoh!
> > > > >
> > > > > Best,
> > > > > Jiabao Sun
> > > > >
> > > > > 2023年8月3日 下午7:32，Danny Cranmer  写道：
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce Hong Teoh as a
> > > > > new
> > > > > Flink
> > > > > Committer.
> > > > >
> > > > > Hong has been active in the Flink community for over 1 year and
> > > > > has
> > > > > played
> > > > > a key role in developing and maintaining AWS integrations, core
> > > > > connector
> > > > > APIs and more recently, improvements to the Flink REST API.
> > > > > Additionally,
> > > > > Hong is a very active community member, supporting users and
> > > > > participating
> > > > > in discussions on the mailing lists, Flink slack channels and
> > > > > speaking
> > > > > at
> > > > > conferences.
> > > > >
> > > > > Please join me in congratulating Hong for becoming a Flink
> > > > > Committer!
> > > > >
> > > > > Thanks,
> > > > > Danny Cranmer (on behalf of the Flink PMC)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best,
> > > > > Benchao Li
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl

[Yuxin Tan]

Congratulations, Matthias!

Best,
Yuxin


Sergey Nuyanzin  于2023年8月4日周五 14:21写道：

> Congratulations, Matthias!
> Well deserved!
>
> On Fri, Aug 4, 2023 at 7:59 AM liu ron  wrote:
>
> > Congrats, Matthias!
> >
> > Best,
> > Ron
> >
> > Shammon FY  于2023年8月4日周五 13:24写道：
> >
> > > Congratulations, Matthias!
> > >
> > > On Fri, Aug 4, 2023 at 1:13 PM Samrat Deb 
> wrote:
> > >
> > > > Congrats, Matthias!
> > > >
> > > >
> > > > On Fri, 4 Aug 2023 at 10:13 AM, Benchao Li 
> > wrote:
> > > >
> > > > > Congratulations, Matthias!
> > > > >
> > > > > Jing Ge  于2023年8月4日周五 12:35写道：
> > > > >
> > > > > > Congrats! Matthias!
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Fri, Aug 4, 2023 at 12:09 PM Yangze Guo 
> > > wrote:
> > > > > >
> > > > > > > Congrats, Matthias!
> > > > > > >
> > > > > > > Best,
> > > > > > > Yangze Guo
> > > > > > >
> > > > > > > On Fri, Aug 4, 2023 at 11:44 AM Qingsheng Ren <
> re...@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Congratulations, Matthias! This is absolutely well deserved.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Qingsheng
> > > > > > > >
> > > > > > > > On Fri, Aug 4, 2023 at 11:31 AM Rui Fan <
> 1996fan...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations Matthias, well deserved!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Rui Fan
> > > > > > > > >
> > > > > > > > > On Fri, Aug 4, 2023 at 11:30 AM Leonard Xu <
> > xbjt...@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congratulations,  Matthias.
> > > > > > > > > >
> > > > > > > > > > Well deserved ^_^
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > On Aug 4, 2023, at 11:18 AM, Xintong Song <
> > > > > tonysong...@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > On behalf of the PMC, I'm very happy to announce that
> > > > Matthias
> > > > > > > Pohl has
> > > > > > > > > > > joined the Flink PMC!
> > > > > > > > > > >
> > > > > > > > > > > Matthias has been consistently contributing to the
> > project
> > > > > since
> > > > > > > Sep
> > > > > > > > > > 2020,
> > > > > > > > > > > and became a committer in Dec 2021. He mainly works in
> > > > Flink's
> > > > > > > > > > distributed
> > > > > > > > > > > coordination and high availability areas. He has worked
> > on
> > > > many
> > > > > > > FLIPs
> > > > > > > > > > > including FLIP195/270/285. He helped a lot with the
> > release
> > > > > > > management,
> > > > > > > > > > > being one of the Flink 1.17 release managers and also
> > very
> > > > > active
> > > > > > > in
> > > > > > > > > > Flink
> > > > > > > > > > > 1.18 / 2.0 efforts. He also contributed a lot to
> > improving
> > > > the
> > > > > > > build
> > > > > > > > > > > stability.
> > > > > > > > > > >
> > > > > > > > > > > Please join me in congratulating Matthias!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > Xintong (on behalf of the Apache Flink PMC)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best,
> > > > > Benchao Li
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu

[Yuxin Tan]

Congratulations Weihua!

Best,
Yuxin


Junrui Lee  于2023年8月4日周五 14:28写道：

> Congrats, Weihua!
> Best,
> Junrui
>
> Geng Biao  于2023年8月4日周五 14:25写道：
>
> > Congrats, Weihua!
> > Best,
> > Biao Geng
> >
> > 发送自 Outlook for iOS
> > 
> > 发件人: 周仁祥 
> > 发送时间: Friday, August 4, 2023 2:23:42 PM
> > 收件人: dev@flink.apache.org 
> > 抄送: Weihua Hu 
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu
> >
> > Congratulations, Weihua~
> >
> > > 2023年8月4日 14:21，Sergey Nuyanzin  写道：
> > >
> > > Congratulations, Weihua!
> > >
> > > On Fri, Aug 4, 2023 at 8:03 AM Chen Zhanghao <
> zhanghao.c...@outlook.com>
> > > wrote:
> > >
> > >> Congratulations, Weihua!
> > >>
> > >> Best,
> > >> Zhanghao Chen
> > >> 
> > >> 发件人: Xintong Song 
> > >> 发送时间: 2023年8月4日 11:18
> > >> 收件人: dev 
> > >> 抄送: Weihua Hu 
> > >> 主题: [ANNOUNCE] New Apache Flink Committer - Weihua Hu
> > >>
> > >> Hi everyone,
> > >>
> > >> On behalf of the PMC, I'm very happy to announce Weihua Hu as a new
> > Flink
> > >> Committer!
> > >>
> > >> Weihua has been consistently contributing to the project since May
> > 2022. He
> > >> mainly works in Flink's distributed coordination areas. He is the main
> > >> contributor of FLIP-298 and many other improvements in large-scale job
> > >> scheduling and improvements. He is also quite active in mailing lists,
> > >> participating discussions and answering user questions.
> > >>
> > >> Please join me in congratulating Weihua!
> > >>
> > >> Best,
> > >>
> > >> Xintong (on behalf of the Apache Flink PMC)
> > >>
> > >
> > >
> > > --
> > > Best regards,
> > > Sergey
> >
> >
>

[jira] [Created] (FLINK-32709) Modify segment size to improve memory utilization for Hybrid Shuffle

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32709:
-

 Summary: Modify segment size to improve memory utilization for 
Hybrid Shuffle
 Key: FLINK-32709
 URL: https://issues.apache.org/jira/browse/FLINK-32709
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan


Currently, each subpartition in Disk/Remote has a segment size of 8M. When 
writing segments to the Disk tier with a parallelism of 1000, only shuffle data 
exceeding 1000 * 8M can be written to the Memory tier again. However, for most 
shuffles, the data volume size falls below this limit, significantly impacting 
Memory tier utilization. 
For better performance, it is necessary to address this issue to improve the 
memory tier utilization.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE][2.0] FLIP-347: Remove IOReadableWritable serialization in Path

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Xintong Song  于2023年7月26日周三 16:09写道：

> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jul 26, 2023 at 2:29 PM Wencong Liu  wrote:
>
> > Hi dev,
> >
> >
> > I'd like to start a vote on FLIP-347.
> >
> >
> > Discussion thread:
> > https://lists.apache.org/thread/3gcxhnqpsvb85golnlxf9tv5p43xkjgj
> > FLIP:
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path
> >
> >
> > Best regards,
> > Wencong Liu
>

Re: [VOTE][2.0] FLIP-344: Remove parameter in RichFunction#open

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Xintong Song  于2023年7月26日周三 16:09写道：

> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jul 26, 2023 at 2:26 PM Wencong Liu  wrote:
>
> > Hi dev,
> >
> >
> > I'd like to start a vote on FLIP-344.
> >
> >
> > Discussion thread:
> > https://lists.apache.org/thread/5lyjrrdtwkngkol2t541r4xwoh7133km
> > FLIP:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231
> >
> >
> > Best regards,
> > Wencong Liu
>

Re: [VOTE][2.0] FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger()

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Xintong Song  于2023年7月26日周三 16:08写道：

> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jul 26, 2023 at 3:35 PM Yuepeng Pan  wrote:
>
> > +1 (non-binding)
> >
> > Thanks.
> >
> > Yuepeng Pan.
> > At 2023-07-26 14:26:04, "Wencong Liu"  wrote:
> > >Hi dev,
> > >
> > >
> > >I'd like to start a vote on FLIP-343.
> > >
> > >
> > >Discussion thread:
> > https://lists.apache.org/thread/zn11f460x70nn7f2ckqph41bvx416wxc
> > >FLIP:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425229
> > >
> > >
> > >Best regards,
> > >Wencong Liu
> >
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Yuxin Tan]

Congratulations, Yong Fang!

Best,
Yuxin


Yanfei Lei  于2023年7月26日周三 10:18写道：

> Congratulations!
>
> Best regards,
> Yanfei
>
> weijie guo  于2023年7月26日周三 10:10写道：
> >
> > Congrats, Yong Fang!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Danny Cranmer  于2023年7月26日周三 03:34写道：
> >
> > > Congrats and welcome!
> > >
> > > Danny.
> > >
> > > On Tue, 25 Jul 2023, 16:48 Matthias Pohl,  .invalid>
> > > wrote:
> > >
> > > > Congratulations :)
> > > >
> > > > On Tue, Jul 25, 2023 at 5:13 PM Jing Ge 
> > > > wrote:
> > > >
> > > > > Congrats, Yong Fang!
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Tue, Jul 25, 2023 at 7:35 PM Yu Li  wrote:
> > > > >
> > > > > > Congrats, Yong!
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin <
> snuyan...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Congratulations, Yong Fang!
> > > > > > >
> > > > > > > On Tue, Jul 25, 2023 at 7:53 AM ConradJam  >
> > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Yong Fang
> > > > > > > >
> > > > > > > > Mang Zhang  于2023年7月25日周二 12:08写道：
> > > > > > > >
> > > > > > > > > Congratulations, Yong Fang!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Mang Zhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 在 2023-07-25 10:30:24，"Jark Wu"  写道：
> > > > > > > > > >Congratulations, Yong Fang!
> > > > > > > > > >
> > > > > > > > > >Best,
> > > > > > > > > >Jark
> > > > > > > > > >
> > > > > > > > > >On Mon, 24 Jul 2023 at 22:11, Wencong Liu <
> > > liuwencle...@163.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Congratulations!
> > > > > > > > > >>
> > > > > > > > > >> Best,
> > > > > > > > > >> Wencong Liu
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> 在 2023-07-24 11:03:30，"Paul Lam"  >
> > > 写道：
> > > > > > > > > >> >Congrats, Shammon!
> > > > > > > > > >> >
> > > > > > > > > >> >Best,
> > > > > > > > > >> >Paul Lam
> > > > > > > > > >> >
> > > > > > > > > >> >> 2023年7月24日 10:56，Jingsong Li  >
> > > 写道：
> > > > > > > > > >> >>
> > > > > > > > > >> >> Shammon
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best
> > > > > > > >
> > > > > > > > ConradJam
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Sergey
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

[jira] [Created] (FLINK-32675) Add doc for the tiered storage of hybrid shuffle

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32675:
-

 Summary: Add doc for the tiered storage of hybrid shuffle
 Key: FLINK-32675
 URL: https://issues.apache.org/jira/browse/FLINK-32675
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


The new Hybrid Shuffle mode supporting remote storage 
(https://issues.apache.org/jira/browse/FLINK-30469) has finished, we should 
also update the Flink doc of Hybrid Shuffle.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS][2.0] FLIP-344: Remove parameter in RichFunction#open

[Yuxin Tan]

+1

Best,
Yuxin


Xintong Song  于2023年7月21日周五 12:04写道：

> +1
>
> Best,
>
> Xintong
>
>
>
> On Fri, Jul 21, 2023 at 10:52 AM Wencong Liu  wrote:
>
> > Hi devs,
> >
> > I would like to start a discussion on FLIP-344: Remove parameter in
> > RichFunction#open [1].
> >
> > The open() method in RichFunction requires a Configuration instance as an
> > argument,
> > which is always passed as a new instance without any configuration
> > parameters in
> > AbstractUdfStreamOperator#open. Thus, it is unnecessary to include this
> > parameter
> > in the open() method.
> > As such I propose to remove the Configuration field from
> > RichFunction#open(Configuration parameters).
> > Looking forward to your feedback.
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231
> > Best regards,
> >
> >
> > Wencong Liu
>

Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path

[Yuxin Tan]

+1

Best,
Yuxin


Xintong Song  于2023年7月21日周五 12:04写道：

> +1
>
> Best,
>
> Xintong
>
>
>
> On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu  wrote:
>
> > Hi devs,
> >
> > I would like to start a discussion on FLIP-347: Remove IOReadableWritable
> > serialization in Path [1].
> >
> >
> > The Path class is currently mutable to support IOReadableWritable
> > serialization. However, many parts
> > of the code assume that the Path is immutable. By making the Path class
> > immutable, we can ensure
> > that paths are stored correctly without the possibility of mutation and
> > eliminate the occurrence of subtle errors.
> > As such I propose to modify the Path class to no longer implement the
> > IOReadableWritable interface.
> > Looking forward to your feedback.
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path
> > Best regards,
> >
> >
> > Wencong Liu
>

Re: [DISCUSS][2.0] FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger()

[Yuxin Tan]

+1

Best,
Yuxin


Jing Ge  于2023年7月21日周五 15:41写道：

> +1
>
> NIT: the release in the FLIP is still empty, it should be 2.0
>
> Best regards,
> Jing
>
> On Fri, Jul 21, 2023 at 6:03 AM Xintong Song 
> wrote:
>
> > +1
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Fri, Jul 21, 2023 at 10:53 AM Wencong Liu 
> wrote:
> >
> > > Hi devs,
> > >
> > > I would like to start a discussion on FLIP-343: Remove parameter in
> > > WindowAssigner#getDefaultTrigger() [1].
> > >
> > >
> > > The method getDefaultTrigger() in WindowAssigner takes a
> > > StreamExecutionEnvironment
> > > parameter, but this parameter is not actually used for any subclasses
> of
> > > WindowAssigner.
> > > Therefore, it is unnecessary to include this parameter.
> > > As such I propose to remove the StreamExecutionEnvironment field from
> > > WindowAssigner#getDefaultTrigger(StreamExecutionEnvironment env).
> > > Looking forward to your feedback.
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425229
> > > Best regards,
> > >
> > >
> > > Wencong Liu
> >
>

[jira] [Created] (FLINK-32595) Kinesis connector doc show wrong deserialization schema version

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32595:
-

 Summary: Kinesis connector doc show wrong deserialization schema 
version
 Key: FLINK-32595
 URL: https://issues.apache.org/jira/browse/FLINK-32595
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kinesis
Affects Versions: aws-connector-4.2.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


[https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kinesis/]

GlueSchemaRegistryJsonDeserializationSchema and 
GlueSchemaRegistryAvroDeserializationSchema show the wrong version(flink 
version), but they have been moved to the repo of aws-connector. 
So we should fix the version number.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32576) ProducerMergedPartitionFileIndex supports spilling to file

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32576:
-

 Summary: ProducerMergedPartitionFileIndex supports spilling to file
 Key: FLINK-32576
 URL: https://issues.apache.org/jira/browse/FLINK-32576
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


When running a very large-scale job, ProducerMergedPartitionFileIndex may 
occupy too much heap memory and cause OOM.

To resolve the issue, ProducerMergedPartitionFileIndex should support spilling 
to file to release the occupied memory when necessary.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32549) Tiered storage memory manager supports ownership transfer for buffers

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32549:
-

 Summary: Tiered storage memory manager supports ownership transfer 
for buffers
 Key: FLINK-32549
 URL: https://issues.apache.org/jira/browse/FLINK-32549
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


Currently, the accumulator is responsible for requesting all buffers, leading 
to an inaccurate number of requested buffers for each tier. 
To address this issue, buffer ownership must be transferred from the 
accumulator to the tiers when writing them, which will enable the memory 
manager to maintain a correct number of requested buffers for different owners.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32541) Fix the buffer leaking in buffer accumulators when a failover occurs

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32541:
-

 Summary: Fix the buffer leaking in buffer accumulators when a 
failover occurs
 Key: FLINK-32541
 URL: https://issues.apache.org/jira/browse/FLINK-32541
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


When a failover occurs, the buffers in the sort/hash accumulators should be 
released correctly to avoid buffers leaking. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32538) CI build failed because node is corrupted when compiling

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32538:
-

 Summary: CI build failed because node is corrupted when compiling
 Key: FLINK-32538
 URL: https://issues.apache.org/jira/browse/FLINK-32538
 Project: Flink
  Issue Type: Bug
  Components: Build System / CI, Tests
Affects Versions: 1.18.0
Reporter: Yuxin Tan


[ERROR] The archive file 
/__w/3/.m2/repository/com/github/eirslett/node/16.13.2/node-16.13.2-linux-x64.tar.gz
 is corrupted and will be deleted. Please try the build again. 
 
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50896=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb=10984]

[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50919=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb=10984]



[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50925=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb=10984]

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Yuxin Tan]

Congratulations!

Best,
Yuxin


Dunn Bangui  于2023年7月4日周二 16:04写道：

> Congratulations!
>
> Best,
> Bangui Dunn
>
> Yangze Guo  于2023年7月4日周二 15:59写道：
>
> > Congrats everyone!
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Jul 4, 2023 at 3:53 PM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > Congratulations!
> > >
> > > Best,
> > > Rui Fan
> > >
> > > On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu  wrote:
> > >
> > > > Congratulations everyone!
> > > >
> > > > Thanks,
> > > > Zhu
> > > >
> > > > Hang Ruan  于2023年7月4日周二 14:06写道：
> > > > >
> > > > > Congratulations!
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Jingsong Li  于2023年7月4日周二 13:47写道：
> > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > > Thank you! All of the Flink community!
> > > > > >
> > > > > > Best,
> > > > > > Jingsong
> > > > > >
> > > > > > On Tue, Jul 4, 2023 at 1:24 PM tison 
> wrote:
> > > > > > >
> > > > > > > Congrats and with honor :D
> > > > > > >
> > > > > > > Best,
> > > > > > > tison.
> > > > > > >
> > > > > > >
> > > > > > > Mang Zhang  于2023年7月4日周二 11:08写道：
> > > > > > >
> > > > > > > > Congratulations!--
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Mang Zhang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 在 2023-07-04 01:53:46，"liu ron"  写道：
> > > > > > > > >Congrats everyone
> > > > > > > > >
> > > > > > > > >Best,
> > > > > > > > >Ron
> > > > > > > > >
> > > > > > > > >Jark Wu  于2023年7月3日周一 22:48写道：
> > > > > > > > >
> > > > > > > > >> Congrats everyone!
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >> Jark
> > > > > > > > >>
> > > > > > > > >> > 2023年7月3日 22:37，Yuval Itzchakov  写道：
> > > > > > > > >> >
> > > > > > > > >> > Congrats team!
> > > > > > > > >> >
> > > > > > > > >> > On Mon, Jul 3, 2023, 17:28 Jing Ge via user <
> > > > > > u...@flink.apache.org
> > > > > > > > >> > wrote:
> > > > > > > > >> >> Congratulations!
> > > > > > > > >> >>
> > > > > > > > >> >> Best regards,
> > > > > > > > >> >> Jing
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia <
> > > > > > luoyu...@alumni.sjtu.edu.cn
> > > > > > > > >> > wrote:
> > > > > > > > >> >>> Congratulations!
> > > > > > > > >> >>>
> > > > > > > > >> >>> Best regards,
> > > > > > > > >> >>> Yuxia
> > > > > > > > >> >>>
> > > > > > > > >> >>> 发件人: "Pushpa Ramakrishnan" <
> > pushpa.ramakrish...@icloud.com
> > > > > >  > > > > > > > >> pushpa.ramakrish...@icloud.com>>
> > > > > > > > >> >>> 收件人: "Xintong Song"  > > > > > > > >> tonysong...@gmail.com>>
> > > > > > > > >> >>> 抄送: "dev"  > > > dev@flink.apache.org>>,
> > > > > > > > >> "User"  u...@flink.apache.org
> > >>
> > > > > > > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > > > > > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023
> SIGMOD
> > > > Systems
> > > > > > > > Award
> > > > > > > > >> >>>
> > > > > > > > >> >>> Congratulations \uD83E\uDD73
> > > > > > > > >> >>>
> > > > > > > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song <
> > > > tonysong...@gmail.com
> > > > > > > > >> > wrote:
> > > > > > > > >> >>>
> > > > > > > > >> >>> 
> > > > > > > > >> >>> Dear Community,
> > > > > > > > >> >>>
> > > > > > > > >> >>> I'm pleased to share this good news with everyone. As
> > some
> > > > of
> > > > > > you
> > > > > > > > may
> > > > > > > > >> have already heard, Apache Flink has won the 2023 SIGMOD
> > Systems
> > > > > > Award
> > > > > > > > [1].
> > > > > > > > >> >>>
> > > > > > > > >> >>> "Apache Flink greatly expanded the use of stream
> > > > > > data-processing."
> > > > > > > > --
> > > > > > > > >> SIGMOD Awards Committee
> > > > > > > > >> >>>
> > > > > > > > >> >>> SIGMOD is one of the most influential data management
> > > > research
> > > > > > > > >> conferences in the world. The Systems Award is awarded to
> an
> > > > > > individual
> > > > > > > > or
> > > > > > > > >> set of individuals to recognize the development of a
> > software or
> > > > > > > > hardware
> > > > > > > > >> system whose technical contributions have had significant
> > > > impact on
> > > > > > the
> > > > > > > > >> theory or practice of large-scale data management systems.
> > > > Winning
> > > > > > of
> > > > > > > > the
> > > > > > > > >> award indicates the high recognition of Flink's
> > technological
> > > > > > > > advancement
> > > > > > > > >> and industry influence from academia.
> > > > > > > > >> >>>
> > > > > > > > >> >>> As an open-source project, Flink wouldn't have come
> > this far
> > > > > > without
> > > > > > > > >> the wide, active and supportive community behind it. Kudos
> > to
> > > > all
> > > > > > of us
> > > > > > > > who
> > > > > > > > >> helped make this happen, including the over 1,400
> > contributors
> > > > and
> > > > > > many
> > > > > > > > >> others who contributed in ways beyond code.
> > > > > > > > >> >>>
> > > > > > > > 

[jira] [Created] (FLINK-32425) Fix Opensearch Connector wrong empty doc link

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32425:
-

 Summary: Fix Opensearch Connector wrong empty doc link
 Key: FLINK-32425
 URL: https://issues.apache.org/jira/browse/FLINK-32425
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Opensearch
Reporter: Yuxin Tan
Assignee: Yuxin Tan


There is an empty link("see here for further information") in 
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/opensearch/.
 And we should fix this.

The link should be like this ("See how to link with it for cluster execution 
here.") in 
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/opensearch/




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

[Yuxin Tan]

+1 (non-binding)

Best,
Yuxin


Yangze Guo  于2023年6月25日周日 12:21写道：

> +1 (binding)
>
> Best,
> Yangze Guo
>
> On Sun, Jun 25, 2023 at 11:41 AM Jark Wu  wrote:
> >
> > +1 (binding)
> >
> > Best,
> > Jark
> >
> > > 2023年6月25日 10:04，Xia Sun  写道：
> > >
> > > +1 (non-binding)
> > >
> > > Best Regards,
> > >
> > > Xia
> > >
> > > yuxia  于2023年6月25日周日 09:23写道：
> > >
> > >> +1 (binding)
> > >> Thanks Lijie driving it.
> > >>
> > >> Best regards,
> > >> Yuxia
> > >>
> > >> - 原始邮件 -
> > >> 发件人: "Yuepeng Pan" 
> > >> 收件人: "dev" 
> > >> 发送时间: 星期六, 2023年 6 月 24日 下午 9:06:53
> > >> 主题: Re:[VOTE] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs
> > >>
> > >> +1 (non-binding)
> > >>
> > >> Thanks,
> > >> Yuepeng Pan
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> At 2023-06-23 23:49:53, "Lijie Wang" 
> wrote:
> > >>> Hi all,
> > >>>
> > >>> Thanks for all the feedback about the FLIP-324: Introduce Runtime
> Filter
> > >>> for Flink Batch Jobs[1]. This FLIP was discussed in [2].
> > >>>
> > >>> I'd like to start a vote for it. The vote will be open for at least
> 72
> > >>> hours (until June 29th 12:00 GMT) unless there is an objection or
> > >>> insufficient votes.
> > >>>
> > >>> [1]
> > >>>
> > >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-324%3A+Introduce+Runtime+Filter+for+Flink+Batch+Jobs
> > >>> [2] https://lists.apache.org/thread/mm0o8fv7x7k13z11htt88zhy7lo8npmg
> > >>>
> > >>> Best,
> > >>> Lijie
> > >>
> >
>

[jira] [Created] (FLINK-32283) Implement SortBufferAccumulator

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32283:
-

 Summary: Implement SortBufferAccumulator
 Key: FLINK-32283
 URL: https://issues.apache.org/jira/browse/FLINK-32283
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: Yuxin Tan
Assignee: Yuxin Tan
 Fix For: 1.18.0


We should implement the sort-based buffer accumulator to decouple the memory 
usage from the parallelism.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-jdbc v3.1.1, release candidate #1

[Yuxin Tan]

+1 (non-binding)

- Checked sign
- Checked the hash
- Checked tag
- Build from source

Best,
Yuxin


weijie guo  于2023年5月29日周一 14:14写道：

> +1 (non-binding)
>
> - checked sign and checksum
> - checked tag in github repository
> - compiled from source
> - checked the web PR
>
> BTW, please remember to update docs/jdbc.yaml for the v3.1 branch after the
> release is completed.
>
> Best regards,
>
> Weijie
>
>
> Jing Ge  于2023年5月29日周一 04:26写道：
>
> > +1 (non-binding)
> >
> > - checked sign
> > - checked hash
> > - checked repos
> > - checked tag
> > - compiled from source
> > - check the web PR
> >
> > Best regards,
> > Jing
> >
> >
> > On Sun, May 28, 2023 at 4:00 PM Benchao Li  wrote:
> >
> > > Thanks Martijn,
> > >
> > > - checked signature/checksum [OK]
> > > - downloaded src, compiled from source [OK]
> > > - diffed src and tag, no binary files [OK]
> > > - gone through nexus staging area, looks good [OK]
> > > - run with flink 1.7.1 [OK]
> > >
> > > One thing I spotted is that the version in `docs/data/jdbc.yml` is
> still
> > > 3.1.0, I'm not sure whether this should be a blocker.
> > >
> > >
> > > Martijn Visser  于2023年5月25日周四 02:55写道：
> > >
> > > > Hi everyone,
> > > > Please review and vote on the release candidate #1 for the version
> > 3.1.1,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release to be deployed to
> dist.apache.org
> > > > [2],
> > > > which are signed with the key with fingerprint
> > > > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > * source code tag v3.1.1-rc1 [5],
> > > > * website pull request listing the new release [6].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Release Manager
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353281
> > > > [2]
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-jdbc-3.1.1-rc1
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1636/
> > > > [5]
> > > https://github.com/apache/flink-connector-jdbc/releases/tag/v3.1.1-rc1
> > > > [6] https://github.com/apache/flink-web/pull/654
> > > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
> >
>

[jira] [Created] (FLINK-32194) Elasticsearch connector should remove the dependency on flink-shaded

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32194:
-

 Summary: Elasticsearch connector should remove the dependency on 
flink-shaded
 Key: FLINK-32194
 URL: https://issues.apache.org/jira/browse/FLINK-32194
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / ElasticSearch
Affects Versions: elasticsearch-4.0.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


The Elasticsearch connector depends on flink-shaded. With the externalization 
of the connector, the connectors shouldn't rely on Flink-Shaded



--
This message was sent by Atlassian Jira
(v8.20.10#820010)