date:20240320

Re: Is there any options to control the file names in file sink

2024-03-20 Thread Yanfei Lei

Hi Lasse,
If the datastream job is used, you can try setting `OutputFileConfig`
for file sink, something like[1]:
```
OutputFileConfig config = OutputFileConfig
 .builder()
 .withPartPrefix("prefix")
 .withPartSuffix(".ext")
 .build();

FileSink> sink = FileSink
 .forRowFormat((new Path(outputPath), new SimpleStringEncoder<>("UTF-8"))
 .withBucketAssigner(new KeyBucketAssigner())
 .withRollingPolicy(OnCheckpointRollingPolicy.build())
 .withOutputFileConfig(config)
 .build();

```

[1] 
https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/filesystem/#bucket-assignment

Lasse Nedergaard  于2024年3月20日周三 20:50写道：
>
> Hi.
>
> Anyone know if it’s possible to control the file names eg change the uuid 
> file names and extensions to something else. I know I can control the path 
> but I would like to be able to set the file name and the extension based on 
> the data in the stream. So I can’t use any general pre or postfix that are 
> applied to all files.
>
> All ideas appreciate
>
> Med venlig hilsen / Best regards
> Lasse Nedergaard
>


-- 
Best,
Yanfei

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Zakelly Lan

Congratulations!


Best,
Zakelly

On Thu, Mar 21, 2024 at 12:05 PM weijie guo 
wrote:

> Congratulations! Well done.
>
>
> Best regards,
>
> Weijie
>
>
> Feng Jin  于2024年3月21日周四 11:40写道：
>
>> Congratulations!
>>
>>
>> Best,
>> Feng
>>
>>
>> On Thu, Mar 21, 2024 at 11:37 AM Ron liu  wrote:
>>
>> > Congratulations!
>> >
>> > Best,
>> > Ron
>> >
>> > Jark Wu  于2024年3月21日周四 10:46写道：
>> >
>> > > Congratulations and welcome!
>> > >
>> > > Best,
>> > > Jark
>> > >
>> > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:
>> > >
>> > > > Congratulations!
>> > > >
>> > > > Best,
>> > > > Rui
>> > > >
>> > > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan 
>> > > wrote:
>> > > >
>> > > > > Congrattulations!
>> > > > >
>> > > > > Best,
>> > > > > Hang
>> > > > >
>> > > > > Lincoln Lee  于2024年3月21日周四 09:54写道：
>> > > > >
>> > > > >>
>> > > > >> Congrats, thanks for the great work!
>> > > > >>
>> > > > >>
>> > > > >> Best,
>> > > > >> Lincoln Lee
>> > > > >>
>> > > > >>
>> > > > >> Peter Huang  于2024年3月20日周三 22:48写道：
>> > > > >>
>> > > > >>> Congratulations
>> > > > >>>
>> > > > >>>
>> > > > >>> Best Regards
>> > > > >>> Peter Huang
>> > > > >>>
>> > > > >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang > >
>> > > > wrote:
>> > > > >>>
>> > > > 
>> > > >  Congratulations
>> > > > 
>> > > > 
>> > > > 
>> > > >  Best,
>> > > >  Huajie Wang
>> > > > 
>> > > > 
>> > > > 
>> > > >  Leonard Xu  于2024年3月20日周三 21:36写道：
>> > > > 
>> > > > > Hi devs and users,
>> > > > >
>> > > > > We are thrilled to announce that the donation of Flink CDC as
>> a
>> > > > > sub-project of Apache Flink has completed. We invite you to
>> > explore
>> > > > the new
>> > > > > resources available:
>> > > > >
>> > > > > - GitHub Repository: https://github.com/apache/flink-cdc
>> > > > > - Flink CDC Documentation:
>> > > > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
>> > > > >
>> > > > > After Flink community accepted this donation[1], we have
>> > completed
>> > > > > software copyright signing, code repo migration, code cleanup,
>> > > > website
>> > > > > migration, CI migration and github issues migration etc.
>> > > > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
>> > > > > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
>> > > > contributors
>> > > > > for their contributions and help during this process！
>> > > > >
>> > > > >
>> > > > > For all previous contributors: The contribution process has
>> > > slightly
>> > > > > changed to align with the main Flink project. To report bugs
>> or
>> > > > suggest new
>> > > > > features, please open tickets
>> > > > > Apache Jira (https://issues.apache.org/jira).  Note that we
>> will
>> > > no
>> > > > > longer accept GitHub issues for these purposes.
>> > > > >
>> > > > >
>> > > > > Welcome to explore the new repository and documentation. Your
>> > > > feedback
>> > > > > and contributions are invaluable as we continue to improve
>> Flink
>> > > CDC.
>> > > > >
>> > > > > Thanks everyone for your support and happy exploring Flink
>> CDC!
>> > > > >
>> > > > > Best，
>> > > > > Leonard
>> > > > > [1]
>> > > https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Zakelly Lan

Congratulations!


Best,
Zakelly

On Thu, Mar 21, 2024 at 12:05 PM weijie guo 
wrote:

> Congratulations! Well done.
>
>
> Best regards,
>
> Weijie
>
>
> Feng Jin  于2024年3月21日周四 11:40写道：
>
>> Congratulations!
>>
>>
>> Best,
>> Feng
>>
>>
>> On Thu, Mar 21, 2024 at 11:37 AM Ron liu  wrote:
>>
>> > Congratulations!
>> >
>> > Best,
>> > Ron
>> >
>> > Jark Wu  于2024年3月21日周四 10:46写道：
>> >
>> > > Congratulations and welcome!
>> > >
>> > > Best,
>> > > Jark
>> > >
>> > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:
>> > >
>> > > > Congratulations!
>> > > >
>> > > > Best,
>> > > > Rui
>> > > >
>> > > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan 
>> > > wrote:
>> > > >
>> > > > > Congrattulations!
>> > > > >
>> > > > > Best,
>> > > > > Hang
>> > > > >
>> > > > > Lincoln Lee  于2024年3月21日周四 09:54写道：
>> > > > >
>> > > > >>
>> > > > >> Congrats, thanks for the great work!
>> > > > >>
>> > > > >>
>> > > > >> Best,
>> > > > >> Lincoln Lee
>> > > > >>
>> > > > >>
>> > > > >> Peter Huang  于2024年3月20日周三 22:48写道：
>> > > > >>
>> > > > >>> Congratulations
>> > > > >>>
>> > > > >>>
>> > > > >>> Best Regards
>> > > > >>> Peter Huang
>> > > > >>>
>> > > > >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang > >
>> > > > wrote:
>> > > > >>>
>> > > > 
>> > > >  Congratulations
>> > > > 
>> > > > 
>> > > > 
>> > > >  Best,
>> > > >  Huajie Wang
>> > > > 
>> > > > 
>> > > > 
>> > > >  Leonard Xu  于2024年3月20日周三 21:36写道：
>> > > > 
>> > > > > Hi devs and users,
>> > > > >
>> > > > > We are thrilled to announce that the donation of Flink CDC as
>> a
>> > > > > sub-project of Apache Flink has completed. We invite you to
>> > explore
>> > > > the new
>> > > > > resources available:
>> > > > >
>> > > > > - GitHub Repository: https://github.com/apache/flink-cdc
>> > > > > - Flink CDC Documentation:
>> > > > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
>> > > > >
>> > > > > After Flink community accepted this donation[1], we have
>> > completed
>> > > > > software copyright signing, code repo migration, code cleanup,
>> > > > website
>> > > > > migration, CI migration and github issues migration etc.
>> > > > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
>> > > > > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
>> > > > contributors
>> > > > > for their contributions and help during this process！
>> > > > >
>> > > > >
>> > > > > For all previous contributors: The contribution process has
>> > > slightly
>> > > > > changed to align with the main Flink project. To report bugs
>> or
>> > > > suggest new
>> > > > > features, please open tickets
>> > > > > Apache Jira (https://issues.apache.org/jira).  Note that we
>> will
>> > > no
>> > > > > longer accept GitHub issues for these purposes.
>> > > > >
>> > > > >
>> > > > > Welcome to explore the new repository and documentation. Your
>> > > > feedback
>> > > > > and contributions are invaluable as we continue to improve
>> Flink
>> > > CDC.
>> > > > >
>> > > > > Thanks everyone for your support and happy exploring Flink
>> CDC!
>> > > > >
>> > > > > Best，
>> > > > > Leonard
>> > > > > [1]
>> > > https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread weijie guo

Congratulations! Well done.


Best regards,

Weijie


Feng Jin  于2024年3月21日周四 11:40写道：

> Congratulations!
>
>
> Best,
> Feng
>
>
> On Thu, Mar 21, 2024 at 11:37 AM Ron liu  wrote:
>
> > Congratulations!
> >
> > Best,
> > Ron
> >
> > Jark Wu  于2024年3月21日周四 10:46写道：
> >
> > > Congratulations and welcome!
> > >
> > > Best,
> > > Jark
> > >
> > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan 
> > > wrote:
> > > >
> > > > > Congrattulations!
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Lincoln Lee  于2024年3月21日周四 09:54写道：
> > > > >
> > > > >>
> > > > >> Congrats, thanks for the great work!
> > > > >>
> > > > >>
> > > > >> Best,
> > > > >> Lincoln Lee
> > > > >>
> > > > >>
> > > > >> Peter Huang  于2024年3月20日周三 22:48写道：
> > > > >>
> > > > >>> Congratulations
> > > > >>>
> > > > >>>
> > > > >>> Best Regards
> > > > >>> Peter Huang
> > > > >>>
> > > > >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang 
> > > > wrote:
> > > > >>>
> > > > 
> > > >  Congratulations
> > > > 
> > > > 
> > > > 
> > > >  Best,
> > > >  Huajie Wang
> > > > 
> > > > 
> > > > 
> > > >  Leonard Xu  于2024年3月20日周三 21:36写道：
> > > > 
> > > > > Hi devs and users,
> > > > >
> > > > > We are thrilled to announce that the donation of Flink CDC as a
> > > > > sub-project of Apache Flink has completed. We invite you to
> > explore
> > > > the new
> > > > > resources available:
> > > > >
> > > > > - GitHub Repository: https://github.com/apache/flink-cdc
> > > > > - Flink CDC Documentation:
> > > > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > > > >
> > > > > After Flink community accepted this donation[1], we have
> > completed
> > > > > software copyright signing, code repo migration, code cleanup,
> > > > website
> > > > > migration, CI migration and github issues migration etc.
> > > > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > > > > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> > > > contributors
> > > > > for their contributions and help during this process！
> > > > >
> > > > >
> > > > > For all previous contributors: The contribution process has
> > > slightly
> > > > > changed to align with the main Flink project. To report bugs or
> > > > suggest new
> > > > > features, please open tickets
> > > > > Apache Jira (https://issues.apache.org/jira).  Note that we
> will
> > > no
> > > > > longer accept GitHub issues for these purposes.
> > > > >
> > > > >
> > > > > Welcome to explore the new repository and documentation. Your
> > > > feedback
> > > > > and contributions are invaluable as we continue to improve
> Flink
> > > CDC.
> > > > >
> > > > > Thanks everyone for your support and happy exploring Flink CDC!
> > > > >
> > > > > Best，
> > > > > Leonard
> > > > > [1]
> > > https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread weijie guo

Congratulations! Well done.


Best regards,

Weijie


Feng Jin  于2024年3月21日周四 11:40写道：

> Congratulations!
>
>
> Best,
> Feng
>
>
> On Thu, Mar 21, 2024 at 11:37 AM Ron liu  wrote:
>
> > Congratulations!
> >
> > Best,
> > Ron
> >
> > Jark Wu  于2024年3月21日周四 10:46写道：
> >
> > > Congratulations and welcome!
> > >
> > > Best,
> > > Jark
> > >
> > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan 
> > > wrote:
> > > >
> > > > > Congrattulations!
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Lincoln Lee  于2024年3月21日周四 09:54写道：
> > > > >
> > > > >>
> > > > >> Congrats, thanks for the great work!
> > > > >>
> > > > >>
> > > > >> Best,
> > > > >> Lincoln Lee
> > > > >>
> > > > >>
> > > > >> Peter Huang  于2024年3月20日周三 22:48写道：
> > > > >>
> > > > >>> Congratulations
> > > > >>>
> > > > >>>
> > > > >>> Best Regards
> > > > >>> Peter Huang
> > > > >>>
> > > > >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang 
> > > > wrote:
> > > > >>>
> > > > 
> > > >  Congratulations
> > > > 
> > > > 
> > > > 
> > > >  Best,
> > > >  Huajie Wang
> > > > 
> > > > 
> > > > 
> > > >  Leonard Xu  于2024年3月20日周三 21:36写道：
> > > > 
> > > > > Hi devs and users,
> > > > >
> > > > > We are thrilled to announce that the donation of Flink CDC as a
> > > > > sub-project of Apache Flink has completed. We invite you to
> > explore
> > > > the new
> > > > > resources available:
> > > > >
> > > > > - GitHub Repository: https://github.com/apache/flink-cdc
> > > > > - Flink CDC Documentation:
> > > > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > > > >
> > > > > After Flink community accepted this donation[1], we have
> > completed
> > > > > software copyright signing, code repo migration, code cleanup,
> > > > website
> > > > > migration, CI migration and github issues migration etc.
> > > > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > > > > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> > > > contributors
> > > > > for their contributions and help during this process！
> > > > >
> > > > >
> > > > > For all previous contributors: The contribution process has
> > > slightly
> > > > > changed to align with the main Flink project. To report bugs or
> > > > suggest new
> > > > > features, please open tickets
> > > > > Apache Jira (https://issues.apache.org/jira).  Note that we
> will
> > > no
> > > > > longer accept GitHub issues for these purposes.
> > > > >
> > > > >
> > > > > Welcome to explore the new repository and documentation. Your
> > > > feedback
> > > > > and contributions are invaluable as we continue to improve
> Flink
> > > CDC.
> > > > >
> > > > > Thanks everyone for your support and happy exploring Flink CDC!
> > > > >
> > > > > Best，
> > > > > Leonard
> > > > > [1]
> > > https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Feng Jin

Congratulations!


Best,
Feng


On Thu, Mar 21, 2024 at 11:37 AM Ron liu  wrote:

> Congratulations!
>
> Best,
> Ron
>
> Jark Wu  于2024年3月21日周四 10:46写道：
>
> > Congratulations and welcome!
> >
> > Best,
> > Jark
> >
> > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:
> >
> > > Congratulations!
> > >
> > > Best,
> > > Rui
> > >
> > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan 
> > wrote:
> > >
> > > > Congrattulations!
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Lincoln Lee  于2024年3月21日周四 09:54写道：
> > > >
> > > >>
> > > >> Congrats, thanks for the great work!
> > > >>
> > > >>
> > > >> Best,
> > > >> Lincoln Lee
> > > >>
> > > >>
> > > >> Peter Huang  于2024年3月20日周三 22:48写道：
> > > >>
> > > >>> Congratulations
> > > >>>
> > > >>>
> > > >>> Best Regards
> > > >>> Peter Huang
> > > >>>
> > > >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang 
> > > wrote:
> > > >>>
> > > 
> > >  Congratulations
> > > 
> > > 
> > > 
> > >  Best,
> > >  Huajie Wang
> > > 
> > > 
> > > 
> > >  Leonard Xu  于2024年3月20日周三 21:36写道：
> > > 
> > > > Hi devs and users,
> > > >
> > > > We are thrilled to announce that the donation of Flink CDC as a
> > > > sub-project of Apache Flink has completed. We invite you to
> explore
> > > the new
> > > > resources available:
> > > >
> > > > - GitHub Repository: https://github.com/apache/flink-cdc
> > > > - Flink CDC Documentation:
> > > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > > >
> > > > After Flink community accepted this donation[1], we have
> completed
> > > > software copyright signing, code repo migration, code cleanup,
> > > website
> > > > migration, CI migration and github issues migration etc.
> > > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > > > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> > > contributors
> > > > for their contributions and help during this process！
> > > >
> > > >
> > > > For all previous contributors: The contribution process has
> > slightly
> > > > changed to align with the main Flink project. To report bugs or
> > > suggest new
> > > > features, please open tickets
> > > > Apache Jira (https://issues.apache.org/jira).  Note that we will
> > no
> > > > longer accept GitHub issues for these purposes.
> > > >
> > > >
> > > > Welcome to explore the new repository and documentation. Your
> > > feedback
> > > > and contributions are invaluable as we continue to improve Flink
> > CDC.
> > > >
> > > > Thanks everyone for your support and happy exploring Flink CDC!
> > > >
> > > > Best，
> > > > Leonard
> > > > [1]
> > https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Feng Jin

Congratulations!


Best,
Feng


On Thu, Mar 21, 2024 at 11:37 AM Ron liu  wrote:

> Congratulations!
>
> Best,
> Ron
>
> Jark Wu  于2024年3月21日周四 10:46写道：
>
> > Congratulations and welcome!
> >
> > Best,
> > Jark
> >
> > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:
> >
> > > Congratulations!
> > >
> > > Best,
> > > Rui
> > >
> > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan 
> > wrote:
> > >
> > > > Congrattulations!
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Lincoln Lee  于2024年3月21日周四 09:54写道：
> > > >
> > > >>
> > > >> Congrats, thanks for the great work!
> > > >>
> > > >>
> > > >> Best,
> > > >> Lincoln Lee
> > > >>
> > > >>
> > > >> Peter Huang  于2024年3月20日周三 22:48写道：
> > > >>
> > > >>> Congratulations
> > > >>>
> > > >>>
> > > >>> Best Regards
> > > >>> Peter Huang
> > > >>>
> > > >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang 
> > > wrote:
> > > >>>
> > > 
> > >  Congratulations
> > > 
> > > 
> > > 
> > >  Best,
> > >  Huajie Wang
> > > 
> > > 
> > > 
> > >  Leonard Xu  于2024年3月20日周三 21:36写道：
> > > 
> > > > Hi devs and users,
> > > >
> > > > We are thrilled to announce that the donation of Flink CDC as a
> > > > sub-project of Apache Flink has completed. We invite you to
> explore
> > > the new
> > > > resources available:
> > > >
> > > > - GitHub Repository: https://github.com/apache/flink-cdc
> > > > - Flink CDC Documentation:
> > > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > > >
> > > > After Flink community accepted this donation[1], we have
> completed
> > > > software copyright signing, code repo migration, code cleanup,
> > > website
> > > > migration, CI migration and github issues migration etc.
> > > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > > > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> > > contributors
> > > > for their contributions and help during this process！
> > > >
> > > >
> > > > For all previous contributors: The contribution process has
> > slightly
> > > > changed to align with the main Flink project. To report bugs or
> > > suggest new
> > > > features, please open tickets
> > > > Apache Jira (https://issues.apache.org/jira).  Note that we will
> > no
> > > > longer accept GitHub issues for these purposes.
> > > >
> > > >
> > > > Welcome to explore the new repository and documentation. Your
> > > feedback
> > > > and contributions are invaluable as we continue to improve Flink
> > CDC.
> > > >
> > > > Thanks everyone for your support and happy exploring Flink CDC!
> > > >
> > > > Best，
> > > > Leonard
> > > > [1]
> > https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Ron liu

Congratulations!

Best,
Ron

Jark Wu  于2024年3月21日周四 10:46写道：

> Congratulations and welcome!
>
> Best,
> Jark
>
> On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:
>
> > Congratulations!
> >
> > Best,
> > Rui
> >
> > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan 
> wrote:
> >
> > > Congrattulations!
> > >
> > > Best,
> > > Hang
> > >
> > > Lincoln Lee  于2024年3月21日周四 09:54写道：
> > >
> > >>
> > >> Congrats, thanks for the great work!
> > >>
> > >>
> > >> Best,
> > >> Lincoln Lee
> > >>
> > >>
> > >> Peter Huang  于2024年3月20日周三 22:48写道：
> > >>
> > >>> Congratulations
> > >>>
> > >>>
> > >>> Best Regards
> > >>> Peter Huang
> > >>>
> > >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang 
> > wrote:
> > >>>
> > 
> >  Congratulations
> > 
> > 
> > 
> >  Best,
> >  Huajie Wang
> > 
> > 
> > 
> >  Leonard Xu  于2024年3月20日周三 21:36写道：
> > 
> > > Hi devs and users,
> > >
> > > We are thrilled to announce that the donation of Flink CDC as a
> > > sub-project of Apache Flink has completed. We invite you to explore
> > the new
> > > resources available:
> > >
> > > - GitHub Repository: https://github.com/apache/flink-cdc
> > > - Flink CDC Documentation:
> > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > >
> > > After Flink community accepted this donation[1], we have completed
> > > software copyright signing, code repo migration, code cleanup,
> > website
> > > migration, CI migration and github issues migration etc.
> > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> > contributors
> > > for their contributions and help during this process！
> > >
> > >
> > > For all previous contributors: The contribution process has
> slightly
> > > changed to align with the main Flink project. To report bugs or
> > suggest new
> > > features, please open tickets
> > > Apache Jira (https://issues.apache.org/jira).  Note that we will
> no
> > > longer accept GitHub issues for these purposes.
> > >
> > >
> > > Welcome to explore the new repository and documentation. Your
> > feedback
> > > and contributions are invaluable as we continue to improve Flink
> CDC.
> > >
> > > Thanks everyone for your support and happy exploring Flink CDC!
> > >
> > > Best，
> > > Leonard
> > > [1]
> https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > >
> > >
> >
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Ron liu

Congratulations!

Best,
Ron

Jark Wu  于2024年3月21日周四 10:46写道：

> Congratulations and welcome!
>
> Best,
> Jark
>
> On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:
>
> > Congratulations!
> >
> > Best,
> > Rui
> >
> > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan 
> wrote:
> >
> > > Congrattulations!
> > >
> > > Best,
> > > Hang
> > >
> > > Lincoln Lee  于2024年3月21日周四 09:54写道：
> > >
> > >>
> > >> Congrats, thanks for the great work!
> > >>
> > >>
> > >> Best,
> > >> Lincoln Lee
> > >>
> > >>
> > >> Peter Huang  于2024年3月20日周三 22:48写道：
> > >>
> > >>> Congratulations
> > >>>
> > >>>
> > >>> Best Regards
> > >>> Peter Huang
> > >>>
> > >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang 
> > wrote:
> > >>>
> > 
> >  Congratulations
> > 
> > 
> > 
> >  Best,
> >  Huajie Wang
> > 
> > 
> > 
> >  Leonard Xu  于2024年3月20日周三 21:36写道：
> > 
> > > Hi devs and users,
> > >
> > > We are thrilled to announce that the donation of Flink CDC as a
> > > sub-project of Apache Flink has completed. We invite you to explore
> > the new
> > > resources available:
> > >
> > > - GitHub Repository: https://github.com/apache/flink-cdc
> > > - Flink CDC Documentation:
> > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > >
> > > After Flink community accepted this donation[1], we have completed
> > > software copyright signing, code repo migration, code cleanup,
> > website
> > > migration, CI migration and github issues migration etc.
> > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> > contributors
> > > for their contributions and help during this process！
> > >
> > >
> > > For all previous contributors: The contribution process has
> slightly
> > > changed to align with the main Flink project. To report bugs or
> > suggest new
> > > features, please open tickets
> > > Apache Jira (https://issues.apache.org/jira).  Note that we will
> no
> > > longer accept GitHub issues for these purposes.
> > >
> > >
> > > Welcome to explore the new repository and documentation. Your
> > feedback
> > > and contributions are invaluable as we continue to improve Flink
> CDC.
> > >
> > > Thanks everyone for your support and happy exploring Flink CDC!
> > >
> > > Best，
> > > Leonard
> > > [1]
> https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > >
> > >
> >
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread shuai xu

Congratulations!


Best!
Xushuai

> 2024年3月21日 10:54，Yanquan Lv  写道：
> 
> Congratulations and  Looking forward to future versions！
> 
> Jark Wu  于2024年3月21日周四 10:47写道：
> 
>> Congratulations and welcome!
>> 
>> Best,
>> Jark
>> 
>> On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:
>> 
>>> Congratulations!
>>> 
>>> Best,
>>> Rui
>>> 
>>> On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan 
>> wrote:
>>> 
 Congrattulations!
 
 Best,
 Hang
 
 Lincoln Lee  于2024年3月21日周四 09:54写道：
 
> 
> Congrats, thanks for the great work!
> 
> 
> Best,
> Lincoln Lee
> 
> 
> Peter Huang  于2024年3月20日周三 22:48写道：
> 
>> Congratulations
>> 
>> 
>> Best Regards
>> Peter Huang
>> 
>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang 
>>> wrote:
>> 
>>> 
>>> Congratulations
>>> 
>>> 
>>> 
>>> Best,
>>> Huajie Wang
>>> 
>>> 
>>> 
>>> Leonard Xu  于2024年3月20日周三 21:36写道：
>>> 
 Hi devs and users,
 
 We are thrilled to announce that the donation of Flink CDC as a
 sub-project of Apache Flink has completed. We invite you to explore
>>> the new
 resources available:
 
 - GitHub Repository: https://github.com/apache/flink-cdc
 - Flink CDC Documentation:
 https://nightlies.apache.org/flink/flink-cdc-docs-stable
 
 After Flink community accepted this donation[1], we have completed
 software copyright signing, code repo migration, code cleanup,
>>> website
 migration, CI migration and github issues migration etc.
 Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
 Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
>>> contributors
 for their contributions and help during this process！
 
 
 For all previous contributors: The contribution process has
>> slightly
 changed to align with the main Flink project. To report bugs or
>>> suggest new
 features, please open tickets
 Apache Jira (https://issues.apache.org/jira).  Note that we will
>> no
 longer accept GitHub issues for these purposes.
 
 
 Welcome to explore the new repository and documentation. Your
>>> feedback
 and contributions are invaluable as we continue to improve Flink
>> CDC.
 
 Thanks everyone for your support and happy exploring Flink CDC!
 
 Best，
 Leonard
 [1]
>> https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
 
 
>>> 
>>

Indefinite hanging when querying batch transactional data from kafka

2024-03-20 Thread Dongwoo Kim

Hello community,
I reported a potential bug with the flink kafka connector on Jira a few
weeks ago.
But I haven't received feedback yet.
Could anyone kindly review this issue? Any feedback would be much
appreciated.

Jira issue: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34470

Thanks in advance

Best,
dongwoo

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Yanquan Lv

Congratulations and  Looking forward to future versions！

Jark Wu  于2024年3月21日周四 10:47写道：

> Congratulations and welcome!
>
> Best,
> Jark
>
> On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:
>
> > Congratulations!
> >
> > Best,
> > Rui
> >
> > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan 
> wrote:
> >
> > > Congrattulations!
> > >
> > > Best,
> > > Hang
> > >
> > > Lincoln Lee  于2024年3月21日周四 09:54写道：
> > >
> > >>
> > >> Congrats, thanks for the great work!
> > >>
> > >>
> > >> Best,
> > >> Lincoln Lee
> > >>
> > >>
> > >> Peter Huang  于2024年3月20日周三 22:48写道：
> > >>
> > >>> Congratulations
> > >>>
> > >>>
> > >>> Best Regards
> > >>> Peter Huang
> > >>>
> > >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang 
> > wrote:
> > >>>
> > 
> >  Congratulations
> > 
> > 
> > 
> >  Best,
> >  Huajie Wang
> > 
> > 
> > 
> >  Leonard Xu  于2024年3月20日周三 21:36写道：
> > 
> > > Hi devs and users,
> > >
> > > We are thrilled to announce that the donation of Flink CDC as a
> > > sub-project of Apache Flink has completed. We invite you to explore
> > the new
> > > resources available:
> > >
> > > - GitHub Repository: https://github.com/apache/flink-cdc
> > > - Flink CDC Documentation:
> > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > >
> > > After Flink community accepted this donation[1], we have completed
> > > software copyright signing, code repo migration, code cleanup,
> > website
> > > migration, CI migration and github issues migration etc.
> > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> > contributors
> > > for their contributions and help during this process！
> > >
> > >
> > > For all previous contributors: The contribution process has
> slightly
> > > changed to align with the main Flink project. To report bugs or
> > suggest new
> > > features, please open tickets
> > > Apache Jira (https://issues.apache.org/jira).  Note that we will
> no
> > > longer accept GitHub issues for these purposes.
> > >
> > >
> > > Welcome to explore the new repository and documentation. Your
> > feedback
> > > and contributions are invaluable as we continue to improve Flink
> CDC.
> > >
> > > Thanks everyone for your support and happy exploring Flink CDC!
> > >
> > > Best，
> > > Leonard
> > > [1]
> https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > >
> > >
> >
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Jark Wu

Congratulations and welcome!

Best,
Jark

On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:

> Congratulations!
>
> Best,
> Rui
>
> On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan  wrote:
>
> > Congrattulations!
> >
> > Best,
> > Hang
> >
> > Lincoln Lee  于2024年3月21日周四 09:54写道：
> >
> >>
> >> Congrats, thanks for the great work!
> >>
> >>
> >> Best,
> >> Lincoln Lee
> >>
> >>
> >> Peter Huang  于2024年3月20日周三 22:48写道：
> >>
> >>> Congratulations
> >>>
> >>>
> >>> Best Regards
> >>> Peter Huang
> >>>
> >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang 
> wrote:
> >>>
> 
>  Congratulations
> 
> 
> 
>  Best,
>  Huajie Wang
> 
> 
> 
>  Leonard Xu  于2024年3月20日周三 21:36写道：
> 
> > Hi devs and users,
> >
> > We are thrilled to announce that the donation of Flink CDC as a
> > sub-project of Apache Flink has completed. We invite you to explore
> the new
> > resources available:
> >
> > - GitHub Repository: https://github.com/apache/flink-cdc
> > - Flink CDC Documentation:
> > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> >
> > After Flink community accepted this donation[1], we have completed
> > software copyright signing, code repo migration, code cleanup,
> website
> > migration, CI migration and github issues migration etc.
> > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> contributors
> > for their contributions and help during this process！
> >
> >
> > For all previous contributors: The contribution process has slightly
> > changed to align with the main Flink project. To report bugs or
> suggest new
> > features, please open tickets
> > Apache Jira (https://issues.apache.org/jira).  Note that we will no
> > longer accept GitHub issues for these purposes.
> >
> >
> > Welcome to explore the new repository and documentation. Your
> feedback
> > and contributions are invaluable as we continue to improve Flink CDC.
> >
> > Thanks everyone for your support and happy exploring Flink CDC!
> >
> > Best，
> > Leonard
> > [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> >
> >
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Jark Wu

Congratulations and welcome!

Best,
Jark

On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:

> Congratulations!
>
> Best,
> Rui
>
> On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan  wrote:
>
> > Congrattulations!
> >
> > Best,
> > Hang
> >
> > Lincoln Lee  于2024年3月21日周四 09:54写道：
> >
> >>
> >> Congrats, thanks for the great work!
> >>
> >>
> >> Best,
> >> Lincoln Lee
> >>
> >>
> >> Peter Huang  于2024年3月20日周三 22:48写道：
> >>
> >>> Congratulations
> >>>
> >>>
> >>> Best Regards
> >>> Peter Huang
> >>>
> >>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang 
> wrote:
> >>>
> 
>  Congratulations
> 
> 
> 
>  Best,
>  Huajie Wang
> 
> 
> 
>  Leonard Xu  于2024年3月20日周三 21:36写道：
> 
> > Hi devs and users,
> >
> > We are thrilled to announce that the donation of Flink CDC as a
> > sub-project of Apache Flink has completed. We invite you to explore
> the new
> > resources available:
> >
> > - GitHub Repository: https://github.com/apache/flink-cdc
> > - Flink CDC Documentation:
> > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> >
> > After Flink community accepted this donation[1], we have completed
> > software copyright signing, code repo migration, code cleanup,
> website
> > migration, CI migration and github issues migration etc.
> > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> > Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> contributors
> > for their contributions and help during this process！
> >
> >
> > For all previous contributors: The contribution process has slightly
> > changed to align with the main Flink project. To report bugs or
> suggest new
> > features, please open tickets
> > Apache Jira (https://issues.apache.org/jira).  Note that we will no
> > longer accept GitHub issues for these purposes.
> >
> >
> > Welcome to explore the new repository and documentation. Your
> feedback
> > and contributions are invaluable as we continue to improve Flink CDC.
> >
> > Thanks everyone for your support and happy exploring Flink CDC!
> >
> > Best，
> > Leonard
> > [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> >
> >
>

Re:Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Xuyang

Cheers!




--

Best！
Xuyang

在 2024-03-21 10:28:45，"Rui Fan" <1996fan...@gmail.com> 写道：
>Congratulations!
>
>Best,
>Rui
>
>On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan  wrote:
>
>> Congrattulations!
>>
>> Best,
>> Hang
>>
>> Lincoln Lee  于2024年3月21日周四 09:54写道：
>>
>>>
>>> Congrats, thanks for the great work!
>>>
>>>
>>> Best,
>>> Lincoln Lee
>>>
>>>
>>> Peter Huang  于2024年3月20日周三 22:48写道：
>>>
 Congratulations


 Best Regards
 Peter Huang

 On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang  wrote:

>
> Congratulations
>
>
>
> Best,
> Huajie Wang
>
>
>
> Leonard Xu  于2024年3月20日周三 21:36写道：
>
>> Hi devs and users,
>>
>> We are thrilled to announce that the donation of Flink CDC as a
>> sub-project of Apache Flink has completed. We invite you to explore the 
>> new
>> resources available:
>>
>> - GitHub Repository: https://github.com/apache/flink-cdc
>> - Flink CDC Documentation:
>> https://nightlies.apache.org/flink/flink-cdc-docs-stable
>>
>> After Flink community accepted this donation[1], we have completed
>> software copyright signing, code repo migration, code cleanup, website
>> migration, CI migration and github issues migration etc.
>> Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
>> Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other 
>> contributors
>> for their contributions and help during this process！
>>
>>
>> For all previous contributors: The contribution process has slightly
>> changed to align with the main Flink project. To report bugs or suggest 
>> new
>> features, please open tickets
>> Apache Jira (https://issues.apache.org/jira).  Note that we will no
>> longer accept GitHub issues for these purposes.
>>
>>
>> Welcome to explore the new repository and documentation. Your feedback
>> and contributions are invaluable as we continue to improve Flink CDC.
>>
>> Thanks everyone for your support and happy exploring Flink CDC!
>>
>> Best，
>> Leonard
>> [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>>
>>

Re:Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Xuyang

Cheers!




--

Best！
Xuyang

在 2024-03-21 10:28:45，"Rui Fan" <1996fan...@gmail.com> 写道：
>Congratulations!
>
>Best,
>Rui
>
>On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan  wrote:
>
>> Congrattulations!
>>
>> Best,
>> Hang
>>
>> Lincoln Lee  于2024年3月21日周四 09:54写道：
>>
>>>
>>> Congrats, thanks for the great work!
>>>
>>>
>>> Best,
>>> Lincoln Lee
>>>
>>>
>>> Peter Huang  于2024年3月20日周三 22:48写道：
>>>
 Congratulations


 Best Regards
 Peter Huang

 On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang  wrote:

>
> Congratulations
>
>
>
> Best,
> Huajie Wang
>
>
>
> Leonard Xu  于2024年3月20日周三 21:36写道：
>
>> Hi devs and users,
>>
>> We are thrilled to announce that the donation of Flink CDC as a
>> sub-project of Apache Flink has completed. We invite you to explore the 
>> new
>> resources available:
>>
>> - GitHub Repository: https://github.com/apache/flink-cdc
>> - Flink CDC Documentation:
>> https://nightlies.apache.org/flink/flink-cdc-docs-stable
>>
>> After Flink community accepted this donation[1], we have completed
>> software copyright signing, code repo migration, code cleanup, website
>> migration, CI migration and github issues migration etc.
>> Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
>> Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other 
>> contributors
>> for their contributions and help during this process！
>>
>>
>> For all previous contributors: The contribution process has slightly
>> changed to align with the main Flink project. To report bugs or suggest 
>> new
>> features, please open tickets
>> Apache Jira (https://issues.apache.org/jira).  Note that we will no
>> longer accept GitHub issues for these purposes.
>>
>>
>> Welcome to explore the new repository and documentation. Your feedback
>> and contributions are invaluable as we continue to improve Flink CDC.
>>
>> Thanks everyone for your support and happy exploring Flink CDC!
>>
>> Best，
>> Leonard
>> [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>>
>>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Rui Fan

Congratulations!

Best,
Rui

On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan  wrote:

> Congrattulations!
>
> Best,
> Hang
>
> Lincoln Lee  于2024年3月21日周四 09:54写道：
>
>>
>> Congrats, thanks for the great work!
>>
>>
>> Best,
>> Lincoln Lee
>>
>>
>> Peter Huang  于2024年3月20日周三 22:48写道：
>>
>>> Congratulations
>>>
>>>
>>> Best Regards
>>> Peter Huang
>>>
>>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang  wrote:
>>>

 Congratulations



 Best,
 Huajie Wang



 Leonard Xu  于2024年3月20日周三 21:36写道：

> Hi devs and users,
>
> We are thrilled to announce that the donation of Flink CDC as a
> sub-project of Apache Flink has completed. We invite you to explore the 
> new
> resources available:
>
> - GitHub Repository: https://github.com/apache/flink-cdc
> - Flink CDC Documentation:
> https://nightlies.apache.org/flink/flink-cdc-docs-stable
>
> After Flink community accepted this donation[1], we have completed
> software copyright signing, code repo migration, code cleanup, website
> migration, CI migration and github issues migration etc.
> Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors
> for their contributions and help during this process！
>
>
> For all previous contributors: The contribution process has slightly
> changed to align with the main Flink project. To report bugs or suggest 
> new
> features, please open tickets
> Apache Jira (https://issues.apache.org/jira).  Note that we will no
> longer accept GitHub issues for these purposes.
>
>
> Welcome to explore the new repository and documentation. Your feedback
> and contributions are invaluable as we continue to improve Flink CDC.
>
> Thanks everyone for your support and happy exploring Flink CDC!
>
> Best，
> Leonard
> [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Rui Fan

Congratulations!

Best,
Rui

On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan  wrote:

> Congrattulations!
>
> Best,
> Hang
>
> Lincoln Lee  于2024年3月21日周四 09:54写道：
>
>>
>> Congrats, thanks for the great work!
>>
>>
>> Best,
>> Lincoln Lee
>>
>>
>> Peter Huang  于2024年3月20日周三 22:48写道：
>>
>>> Congratulations
>>>
>>>
>>> Best Regards
>>> Peter Huang
>>>
>>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang  wrote:
>>>

 Congratulations



 Best,
 Huajie Wang



 Leonard Xu  于2024年3月20日周三 21:36写道：

> Hi devs and users,
>
> We are thrilled to announce that the donation of Flink CDC as a
> sub-project of Apache Flink has completed. We invite you to explore the 
> new
> resources available:
>
> - GitHub Repository: https://github.com/apache/flink-cdc
> - Flink CDC Documentation:
> https://nightlies.apache.org/flink/flink-cdc-docs-stable
>
> After Flink community accepted this donation[1], we have completed
> software copyright signing, code repo migration, code cleanup, website
> migration, CI migration and github issues migration etc.
> Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors
> for their contributions and help during this process！
>
>
> For all previous contributors: The contribution process has slightly
> changed to align with the main Flink project. To report bugs or suggest 
> new
> features, please open tickets
> Apache Jira (https://issues.apache.org/jira).  Note that we will no
> longer accept GitHub issues for these purposes.
>
>
> Welcome to explore the new repository and documentation. Your feedback
> and contributions are invaluable as we continue to improve Flink CDC.
>
> Thanks everyone for your support and happy exploring Flink CDC!
>
> Best，
> Leonard
> [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>
>

Re:Is there any options to control the file names in file sink

2024-03-20 Thread Xuyang

Hi, Nedergaard.
IIUC, currently, filesystem sink is not supported to custom the name of the 
output files, and only custom naming of directories should be supported.


[1] 
https://github.com/apache/flink/blob/4142c4386a92f1ec5016583f4832f8869782765e/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSink.java#L232



--

Best！
Xuyang





At 2024-03-20 20:41:11, "Lasse Nedergaard"  
wrote:
>Hi. 
>
>Anyone know if it’s possible to control the file names eg change the uuid file 
>names and extensions to something else. I know I can control the path but I 
>would like to be able to set the file name and the extension based on the data 
>in the stream. So I can’t use any general pre or postfix that are applied to 
>all files. 
>
>All ideas appreciate 
>
>Med venlig hilsen / Best regards
>Lasse Nedergaard

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Hang Ruan

Congrattulations!

Best,
Hang

Lincoln Lee  于2024年3月21日周四 09:54写道：

>
> Congrats, thanks for the great work!
>
>
> Best,
> Lincoln Lee
>
>
> Peter Huang  于2024年3月20日周三 22:48写道：
>
>> Congratulations
>>
>>
>> Best Regards
>> Peter Huang
>>
>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang  wrote:
>>
>>>
>>> Congratulations
>>>
>>>
>>>
>>> Best,
>>> Huajie Wang
>>>
>>>
>>>
>>> Leonard Xu  于2024年3月20日周三 21:36写道：
>>>
 Hi devs and users,

 We are thrilled to announce that the donation of Flink CDC as a
 sub-project of Apache Flink has completed. We invite you to explore the new
 resources available:

 - GitHub Repository: https://github.com/apache/flink-cdc
 - Flink CDC Documentation:
 https://nightlies.apache.org/flink/flink-cdc-docs-stable

 After Flink community accepted this donation[1], we have completed
 software copyright signing, code repo migration, code cleanup, website
 migration, CI migration and github issues migration etc.
 Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
 Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors
 for their contributions and help during this process！

 For all previous contributors: The contribution process has slightly
 changed to align with the main Flink project. To report bugs or suggest new
 features, please open tickets
 Apache Jira (https://issues.apache.org/jira).  Note that we will no
 longer accept GitHub issues for these purposes.

 Welcome to explore the new repository and documentation. Your feedback
 and contributions are invaluable as we continue to improve Flink CDC.

 Thanks everyone for your support and happy exploring Flink CDC!

 Best，
 Leonard
 [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Hang Ruan

Congrattulations!

Best,
Hang

Lincoln Lee  于2024年3月21日周四 09:54写道：

>
> Congrats, thanks for the great work!
>
>
> Best,
> Lincoln Lee
>
>
> Peter Huang  于2024年3月20日周三 22:48写道：
>
>> Congratulations
>>
>>
>> Best Regards
>> Peter Huang
>>
>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang  wrote:
>>
>>>
>>> Congratulations
>>>
>>>
>>>
>>> Best,
>>> Huajie Wang
>>>
>>>
>>>
>>> Leonard Xu  于2024年3月20日周三 21:36写道：
>>>
 Hi devs and users,

 We are thrilled to announce that the donation of Flink CDC as a
 sub-project of Apache Flink has completed. We invite you to explore the new
 resources available:

 - GitHub Repository: https://github.com/apache/flink-cdc
 - Flink CDC Documentation:
 https://nightlies.apache.org/flink/flink-cdc-docs-stable

 After Flink community accepted this donation[1], we have completed
 software copyright signing, code repo migration, code cleanup, website
 migration, CI migration and github issues migration etc.
 Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
 Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors
 for their contributions and help during this process！

 For all previous contributors: The contribution process has slightly
 changed to align with the main Flink project. To report bugs or suggest new
 features, please open tickets
 Apache Jira (https://issues.apache.org/jira).  Note that we will no
 longer accept GitHub issues for these purposes.

 Welcome to explore the new repository and documentation. Your feedback
 and contributions are invaluable as we continue to improve Flink CDC.

 Thanks everyone for your support and happy exploring Flink CDC!

 Best，
 Leonard
 [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Lincoln Lee

Congrats, thanks for the great work!


Best,
Lincoln Lee


Peter Huang  于2024年3月20日周三 22:48写道：

> Congratulations
>
>
> Best Regards
> Peter Huang
>
> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang  wrote:
>
>>
>> Congratulations
>>
>>
>>
>> Best,
>> Huajie Wang
>>
>>
>>
>> Leonard Xu  于2024年3月20日周三 21:36写道：
>>
>>> Hi devs and users,
>>>
>>> We are thrilled to announce that the donation of Flink CDC as a
>>> sub-project of Apache Flink has completed. We invite you to explore the new
>>> resources available:
>>>
>>> - GitHub Repository: https://github.com/apache/flink-cdc
>>> - Flink CDC Documentation:
>>> https://nightlies.apache.org/flink/flink-cdc-docs-stable
>>>
>>> After Flink community accepted this donation[1], we have completed
>>> software copyright signing, code repo migration, code cleanup, website
>>> migration, CI migration and github issues migration etc.
>>> Here I am particularly grateful to Hang Ruan, Zhongqaing Gong, Qingsheng
>>> Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors for their
>>> contributions and help during this process！
>>>
>>>
>>> For all previous contributors: The contribution process has slightly
>>> changed to align with the main Flink project. To report bugs or suggest new
>>> features, please open tickets
>>> Apache Jira (https://issues.apache.org/jira).  Note that we will no
>>> longer accept GitHub issues for these purposes.
>>>
>>>
>>> Welcome to explore the new repository and documentation. Your feedback
>>> and contributions are invaluable as we continue to improve Flink CDC.
>>>
>>> Thanks everyone for your support and happy exploring Flink CDC!
>>>
>>> Best，
>>> Leonard
>>> [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>>>
>>>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Lincoln Lee

Congrats, thanks for the great work!


Best,
Lincoln Lee


Peter Huang  于2024年3月20日周三 22:48写道：

> Congratulations
>
>
> Best Regards
> Peter Huang
>
> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang  wrote:
>
>>
>> Congratulations
>>
>>
>>
>> Best,
>> Huajie Wang
>>
>>
>>
>> Leonard Xu  于2024年3月20日周三 21:36写道：
>>
>>> Hi devs and users,
>>>
>>> We are thrilled to announce that the donation of Flink CDC as a
>>> sub-project of Apache Flink has completed. We invite you to explore the new
>>> resources available:
>>>
>>> - GitHub Repository: https://github.com/apache/flink-cdc
>>> - Flink CDC Documentation:
>>> https://nightlies.apache.org/flink/flink-cdc-docs-stable
>>>
>>> After Flink community accepted this donation[1], we have completed
>>> software copyright signing, code repo migration, code cleanup, website
>>> migration, CI migration and github issues migration etc.
>>> Here I am particularly grateful to Hang Ruan, Zhongqaing Gong, Qingsheng
>>> Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors for their
>>> contributions and help during this process！
>>>
>>>
>>> For all previous contributors: The contribution process has slightly
>>> changed to align with the main Flink project. To report bugs or suggest new
>>> features, please open tickets
>>> Apache Jira (https://issues.apache.org/jira).  Note that we will no
>>> longer accept GitHub issues for these purposes.
>>>
>>>
>>> Welcome to explore the new repository and documentation. Your feedback
>>> and contributions are invaluable as we continue to improve Flink CDC.
>>>
>>> Thanks everyone for your support and happy exploring Flink CDC!
>>>
>>> Best，
>>> Leonard
>>> [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>>>
>>>

Re: End-to-end lag spikes when closing a large number of panes

2024-03-20 Thread Caio Camatta via user

Hey Asimansu,

The inputDataStream is a KeyedStream, I forgot to mention that.

Caio

On Wed, Mar 20, 2024 at 6:56 PM Asimansu Bera 
wrote:

> Hello Caio,
>
> Based on the pseudocode, there is no keyed function present. Hence, the
> window will not be processed parallely . Please check again and respond
> back.
>
> val windowDataStream =
>
>   inputDataStream
>
> .window(TumblingEventTimeWindows of 1 hour)
>
> .trigger(custom trigger)
>
> .aggregate(
>
>preAggregator = custom AggregateFunction,
>
>windowFunction = custom ProcessWindowFunction
>
> )
>
>
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/operators/windows/#keyed-vs-non-keyed-windows
>
> -A
>
>
> On Wed, Mar 20, 2024 at 8:55 AM Caio Camatta via user <
> user@flink.apache.org> wrote:
>
>> We run a large-scale Flink 1.16 cluster that uses windowed aggregations
>> and we’re seeing lag spikes on window closure. I’m curious if others have
>> encountered similar issues before and if anyone has suggestions for how to
>> tackle this problem (other than simply increasing parallelism).
>> ContextLag definition
>>
>> We define end-to-end lag as the delta between the time when the event
>> was persisted in Kafka and the time when Flink finishes processing the
>> event.
>> Window operator definition
>>
>> The important parts (in pseudocode):
>>
>> val windowDataStream =
>>
>>   inputDataStream
>>
>> .window(TumblingEventTimeWindows of 1 hour)
>>
>> .trigger(custom trigger)
>>
>> .aggregate(
>>
>>preAggregator = custom AggregateFunction,
>>
>>windowFunction = custom ProcessWindowFunction
>>
>> )
>>
>> The custom trigger emits a TriggerResult.CONTINUE in onEventTime, i.e.
>> we don’t run any user-defined logic at the end of the window. (This trigger
>> only fires while the window is active via custom logic in onElement.)
>> Numbers
>>
>> Our Flink app processes ~3K events per second and I’ve calculated that
>> there are around 200-300K panes to close per Task at the end of the 1-hour
>> window. Our lag is fairly stable at a few hundred milliseconds during
>> the window but spikes to 5-10 seconds when the window expires, which is a
>> problem for us.
>> The issue
>>
>> The magnitude of the lag spikes on window closure correlate with
>>
>>-
>>
>>the size of the window (a 1-hour window has bigger spikes than a
>>5-minute window.)
>>-
>>
>>the cardinality of the keys in the event stream.
>>-
>>
>>the number of events being processed per second.
>>
>> In other words, the more panes to close, the bigger the lag spike. I'm
>> fairly sure that the lag is coming entirely from the WindowOperator’s
>> clearAllState and I’ve validated that CPU profiles show clearAllState
>> using a significant amount of CPU.
>>
>> Does this theory sound plausible? What could we do to minimize the
>> effects of window clean-up? It would be nice to do it incrementally or
>> asynchronously but I'm not sure if Flink provides this functionality today.
>> Thanks,
>> Caio
>>
>

Re: End-to-end lag spikes when closing a large number of panes

2024-03-20 Thread Asimansu Bera

Hello Caio,

Based on the pseudocode, there is no keyed function present. Hence, the
window will not be processed parallely . Please check again and respond
back.

val windowDataStream =

  inputDataStream

.window(TumblingEventTimeWindows of 1 hour)

.trigger(custom trigger)

.aggregate(

   preAggregator = custom AggregateFunction,

   windowFunction = custom ProcessWindowFunction

)

https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/operators/windows/#keyed-vs-non-keyed-windows

-A


On Wed, Mar 20, 2024 at 8:55 AM Caio Camatta via user 
wrote:

> We run a large-scale Flink 1.16 cluster that uses windowed aggregations
> and we’re seeing lag spikes on window closure. I’m curious if others have
> encountered similar issues before and if anyone has suggestions for how to
> tackle this problem (other than simply increasing parallelism).
> ContextLag definition
>
> We define end-to-end lag as the delta between the time when the event was
> persisted in Kafka and the time when Flink finishes processing the event.
> Window operator definition
>
> The important parts (in pseudocode):
>
> val windowDataStream =
>
>   inputDataStream
>
> .window(TumblingEventTimeWindows of 1 hour)
>
> .trigger(custom trigger)
>
> .aggregate(
>
>preAggregator = custom AggregateFunction,
>
>windowFunction = custom ProcessWindowFunction
>
> )
>
> The custom trigger emits a TriggerResult.CONTINUE in onEventTime, i.e. we
> don’t run any user-defined logic at the end of the window. (This trigger
> only fires while the window is active via custom logic in onElement.)
> Numbers
>
> Our Flink app processes ~3K events per second and I’ve calculated that
> there are around 200-300K panes to close per Task at the end of the 1-hour
> window. Our lag is fairly stable at a few hundred milliseconds during the
> window but spikes to 5-10 seconds when the window expires, which is a
> problem for us.
> The issue
>
> The magnitude of the lag spikes on window closure correlate with
>
>-
>
>the size of the window (a 1-hour window has bigger spikes than a
>5-minute window.)
>-
>
>the cardinality of the keys in the event stream.
>-
>
>the number of events being processed per second.
>
> In other words, the more panes to close, the bigger the lag spike. I'm
> fairly sure that the lag is coming entirely from the WindowOperator’s
> clearAllState and I’ve validated that CPU profiles show clearAllState
> using a significant amount of CPU.
>
> Does this theory sound plausible? What could we do to minimize the effects
> of window clean-up? It would be nice to do it incrementally or
> asynchronously but I'm not sure if Flink provides this functionality today.
> Thanks,
> Caio
>

Re: Need help in understanding PojoSerializer

2024-03-20 Thread Alexis Sarda-Espinosa

Hi Sachin,

Check the last few comments I wrote in this thread:

https://lists.apache.org/thread/l71d1cqo9xv8rsw0gfjo19kb1pct2xj1

Regards,
Alexis.

On Wed, 20 Mar 2024, 18:51 Sachin Mittal,  wrote:

> Hi,
> I saw the post but I did not understand how I would configure these fields
> to use those serializers. (I can change the set type to a list type for
> now).
> As per the docs I see that we can annotate fields with @TypeInfo
>
> But what I did not get is how using this annotation I can use ListSerializer
> and *MapSerializer.*
>
> Thanks
> Sachin
>
>
> On Wed, Mar 20, 2024 at 10:47 PM Ken Krugler 
> wrote:
>
>> Flink doesn’t have built-in support for serializing Sets.
>>
>> See this (stale) issue about the same:
>> https://issues.apache.org/jira/browse/FLINK-16729
>>
>> You could create a custom serializer for sets, see
>> https://stackoverflow.com/questions/59800851/flink-serialization-of-java-util-list-and-java-util-map
>> and
>> https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/ListSerializer.html
>> for details on how this was done for a list, but it’s not trivial.
>>
>> Or as a hack, use a Map and the existing support for map
>> serialization via
>> https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/MapSerializer.html
>>
>> — Ken
>>
>>
>> On Mar 20, 2024, at 10:04 AM, Sachin Mittal  wrote:
>>
>> Hi,
>> I have a Pojo class like this
>>
>> public class A {
>>
>> public String str;
>>
>> public Set aSet;
>>
>> public Map dMap;
>>
>> }
>>
>> However when I start the flink program I get this message:
>>
>> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
>> dMap will be processed as GenericType. Please read the Flink
>> documentation on "Data Types & Serialization" for details of the effect on
>> performance and schema evolution.
>>
>> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
>> aSet will be processed as GenericType. Please read the Flink
>> documentation on "Data Types & Serialization" for details of the effect on
>> performance and schema evolution.
>>
>> Also in my code I have added
>>
>> env.getConfig().disableGenericTypes();
>>
>> So I don't understand when I use Maps and Sets of primitive types why is 
>> Flink not
>>
>> able to use PojoSerializer for these fields and even when I have disabled 
>> generics types.
>>
>> why I am getting message that it will be processed as GenericType?
>>
>>
>> Any help in understanding what I need to do to ensure all the fields of my 
>> object are handled using PojoSerializer.
>>
>>
>> Thanks
>>
>> Sachin
>>
>>
>>
>> --
>> Ken Krugler
>> http://www.scaleunlimited.com
>> Custom big data solutions
>> Flink & Pinot
>>
>>
>>
>>

Re: Need help in understanding PojoSerializer

2024-03-20 Thread Sachin Mittal

Hi,
I saw the post but I did not understand how I would configure these fields
to use those serializers. (I can change the set type to a list type for
now).
As per the docs I see that we can annotate fields with @TypeInfo

But what I did not get is how using this annotation I can use ListSerializer
and *MapSerializer.*

Thanks
Sachin


On Wed, Mar 20, 2024 at 10:47 PM Ken Krugler 
wrote:

> Flink doesn’t have built-in support for serializing Sets.
>
> See this (stale) issue about the same:
> https://issues.apache.org/jira/browse/FLINK-16729
>
> You could create a custom serializer for sets, see
> https://stackoverflow.com/questions/59800851/flink-serialization-of-java-util-list-and-java-util-map
> and
> https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/ListSerializer.html
> for details on how this was done for a list, but it’s not trivial.
>
> Or as a hack, use a Map and the existing support for map
> serialization via
> https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/MapSerializer.html
>
> — Ken
>
>
> On Mar 20, 2024, at 10:04 AM, Sachin Mittal  wrote:
>
> Hi,
> I have a Pojo class like this
>
> public class A {
>
> public String str;
>
> public Set aSet;
>
> public Map dMap;
>
> }
>
> However when I start the flink program I get this message:
>
> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
> dMap will be processed as GenericType. Please read the Flink
> documentation on "Data Types & Serialization" for details of the effect on
> performance and schema evolution.
>
> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
> aSet will be processed as GenericType. Please read the Flink
> documentation on "Data Types & Serialization" for details of the effect on
> performance and schema evolution.
>
> Also in my code I have added
>
> env.getConfig().disableGenericTypes();
>
> So I don't understand when I use Maps and Sets of primitive types why is 
> Flink not
>
> able to use PojoSerializer for these fields and even when I have disabled 
> generics types.
>
> why I am getting message that it will be processed as GenericType?
>
>
> Any help in understanding what I need to do to ensure all the fields of my 
> object are handled using PojoSerializer.
>
>
> Thanks
>
> Sachin
>
>
>
> --
> Ken Krugler
> http://www.scaleunlimited.com
> Custom big data solutions
> Flink & Pinot
>
>
>
>

Re: Need help in understanding PojoSerializer

2024-03-20 Thread Ken Krugler

Flink doesn’t have built-in support for serializing Sets.

See this (stale) issue about the same: 
https://issues.apache.org/jira/browse/FLINK-16729

You could create a custom serializer for sets, see 
https://stackoverflow.com/questions/59800851/flink-serialization-of-java-util-list-and-java-util-map
 and 
https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/ListSerializer.html
 for details on how this was done for a list, but it’s not trivial.

Or as a hack, use a Map and the existing support for map 
serialization via 
https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/api/common/typeutils/base/MapSerializer.html

— Ken


> On Mar 20, 2024, at 10:04 AM, Sachin Mittal  wrote:
> 
> Hi,
> I have a Pojo class like this
> 
> public class A {
> public String str;
> public Set aSet;
> public Map dMap;
> }
> 
> However when I start the flink program I get this message:
> 
> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field 
> A#dMap will be processed as GenericType. Please read the Flink documentation 
> on "Data Types & Serialization" for details of the effect on performance and 
> schema evolution.
> 
> org.apache.flink.api.java.typeutils.TypeExtractor[] - Field 
> A#aSet will be processed as GenericType. Please read the Flink documentation 
> on "Data Types & Serialization" for details of the effect on performance and 
> schema evolution.
> 
> Also in my code I have added 
> env.getConfig().disableGenericTypes();
> So I don't understand when I use Maps and Sets of primitive types why is 
> Flink not
> able to use PojoSerializer for these fields and even when I have disabled 
> generics types.
> why I am getting message that it will be processed as GenericType?
> 
> Any help in understanding what I need to do to ensure all the fields of my 
> object are handled using PojoSerializer.
> 
> Thanks
> Sachin
>  
> 

--
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink & Pinot

Need help in understanding PojoSerializer

2024-03-20 Thread Sachin Mittal

Hi,
I have a Pojo class like this

public class A {

public String str;

public Set aSet;

public Map dMap;

}

However when I start the flink program I get this message:

org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
dMap will be processed as GenericType. Please read the Flink documentation
on "Data Types & Serialization" for details of the effect on performance
and schema evolution.

org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A#
aSet will be processed as GenericType. Please read the Flink documentation
on "Data Types & Serialization" for details of the effect on performance
and schema evolution.

Also in my code I have added

env.getConfig().disableGenericTypes();

So I don't understand when I use Maps and Sets of primitive types why
is Flink not

able to use PojoSerializer for these fields and even when I have
disabled generics types.

why I am getting message that it will be processed as GenericType?


Any help in understanding what I need to do to ensure all the fields
of my object are handled using PojoSerializer.


Thanks

Sachin

End-to-end lag spikes when closing a large number of panes

2024-03-20 Thread Caio Camatta via user

We run a large-scale Flink 1.16 cluster that uses windowed aggregations and
we’re seeing lag spikes on window closure. I’m curious if others have
encountered similar issues before and if anyone has suggestions for how to
tackle this problem (other than simply increasing parallelism).
ContextLag definition

We define end-to-end lag as the delta between the time when the event was
persisted in Kafka and the time when Flink finishes processing the event.
Window operator definition

The important parts (in pseudocode):

val windowDataStream =

  inputDataStream

.window(TumblingEventTimeWindows of 1 hour)

.trigger(custom trigger)

.aggregate(

   preAggregator = custom AggregateFunction,

   windowFunction = custom ProcessWindowFunction

)

The custom trigger emits a TriggerResult.CONTINUE in onEventTime, i.e. we
don’t run any user-defined logic at the end of the window. (This trigger
only fires while the window is active via custom logic in onElement.)
Numbers

Our Flink app processes ~3K events per second and I’ve calculated that
there are around 200-300K panes to close per Task at the end of the 1-hour
window. Our lag is fairly stable at a few hundred milliseconds during the
window but spikes to 5-10 seconds when the window expires, which is a
problem for us.
The issue

The magnitude of the lag spikes on window closure correlate with

   -

   the size of the window (a 1-hour window has bigger spikes than a
   5-minute window.)
   -

   the cardinality of the keys in the event stream.
   -

   the number of events being processed per second.

In other words, the more panes to close, the bigger the lag spike. I'm
fairly sure that the lag is coming entirely from the WindowOperator’s
clearAllState and I’ve validated that CPU profiles show clearAllState using
a significant amount of CPU.

Does this theory sound plausible? What could we do to minimize the effects
of window clean-up? It would be nice to do it incrementally or
asynchronously but I'm not sure if Flink provides this functionality today.
Thanks,
Caio

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Peter Huang

Congratulations


Best Regards
Peter Huang

On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang  wrote:

>
> Congratulations
>
>
>
> Best,
> Huajie Wang
>
>
>
> Leonard Xu  于2024年3月20日周三 21:36写道：
>
>> Hi devs and users,
>>
>> We are thrilled to announce that the donation of Flink CDC as a
>> sub-project of Apache Flink has completed. We invite you to explore the new
>> resources available:
>>
>> - GitHub Repository: https://github.com/apache/flink-cdc
>> - Flink CDC Documentation:
>> https://nightlies.apache.org/flink/flink-cdc-docs-stable
>>
>> After Flink community accepted this donation[1], we have completed
>> software copyright signing, code repo migration, code cleanup, website
>> migration, CI migration and github issues migration etc.
>> Here I am particularly grateful to Hang Ruan, Zhongqaing Gong, Qingsheng
>> Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors for their
>> contributions and help during this process！
>>
>>
>> For all previous contributors: The contribution process has slightly
>> changed to align with the main Flink project. To report bugs or suggest new
>> features, please open tickets
>> Apache Jira (https://issues.apache.org/jira).  Note that we will no
>> longer accept GitHub issues for these purposes.
>>
>>
>> Welcome to explore the new repository and documentation. Your feedback
>> and contributions are invaluable as we continue to improve Flink CDC.
>>
>> Thanks everyone for your support and happy exploring Flink CDC!
>>
>> Best，
>> Leonard
>> [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>>
>>

Announcing the Community Over Code 2024 Streaming Track

2024-03-20 Thread James Hughes

Hi all,

Community Over Code , the ASF conference,
will be held in Denver, Colorado,

October 7-10, 2024. The call for presentations

is open now through April 15, 2024.  (This is two months earlier than last
year!)

I am one of the co-chairs for the stream processing track, and we would
love to see you there and hope that you will consider submitting a talk.

About the Streaming track:

There are many top-level ASF projects which focus on and push the envelope
for stream and event processing.  ActiveMQ, Beam, Bookkeeper, Camel, Flink,
Kafka, Pulsar, RocketMQ, and Spark are all house-hold names in the stream
processing and analytics world at this point.  These projects show that
stream processing has unique characteristics requiring deep expertise.  On
the other hand, users need easy to apply solutions.  The streaming track
will host talks focused on the use cases and advances of these projects as
well as other developments in the streaming world.

Thanks and see you in October!

Jim

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Huajie Wang

Congratulations



Best,
Huajie Wang



Leonard Xu  于2024年3月20日周三 21:36写道：

> Hi devs and users,
>
> We are thrilled to announce that the donation of Flink CDC as a
> sub-project of Apache Flink has completed. We invite you to explore the new
> resources available:
>
> - GitHub Repository: https://github.com/apache/flink-cdc
> - Flink CDC Documentation:
> https://nightlies.apache.org/flink/flink-cdc-docs-stable
>
> After Flink community accepted this donation[1], we have completed
> software copyright signing, code repo migration, code cleanup, website
> migration, CI migration and github issues migration etc.
> Here I am particularly grateful to Hang Ruan, Zhongqaing Gong, Qingsheng
> Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors for their
> contributions and help during this process！
>
>
> For all previous contributors: The contribution process has slightly
> changed to align with the main Flink project. To report bugs or suggest new
> features, please open tickets
> Apache Jira (https://issues.apache.org/jira).  Note that we will no
> longer accept GitHub issues for these purposes.
>
>
> Welcome to explore the new repository and documentation. Your feedback and
> contributions are invaluable as we continue to improve Flink CDC.
>
> Thanks everyone for your support and happy exploring Flink CDC!
>
> Best，
> Leonard
> [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>
>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Huajie Wang

Congratulations



Best,
Huajie Wang



Leonard Xu  于2024年3月20日周三 21:36写道：

> Hi devs and users,
>
> We are thrilled to announce that the donation of Flink CDC as a
> sub-project of Apache Flink has completed. We invite you to explore the new
> resources available:
>
> - GitHub Repository: https://github.com/apache/flink-cdc
> - Flink CDC Documentation:
> https://nightlies.apache.org/flink/flink-cdc-docs-stable
>
> After Flink community accepted this donation[1], we have completed
> software copyright signing, code repo migration, code cleanup, website
> migration, CI migration and github issues migration etc.
> Here I am particularly grateful to Hang Ruan, Zhongqaing Gong, Qingsheng
> Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors for their
> contributions and help during this process！
>
>
> For all previous contributors: The contribution process has slightly
> changed to align with the main Flink project. To report bugs or suggest new
> features, please open tickets
> Apache Jira (https://issues.apache.org/jira).  Note that we will no
> longer accept GitHub issues for these purposes.
>
>
> Welcome to explore the new repository and documentation. Your feedback and
> contributions are invaluable as we continue to improve Flink CDC.
>
> Thanks everyone for your support and happy exploring Flink CDC!
>
> Best，
> Leonard
> [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>
>

[ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Leonard Xu

Hi devs and users,

We are thrilled to announce that the donation of Flink CDC as a sub-project of 
Apache Flink has completed. We invite you to explore the new resources 
available:

- GitHub Repository: https://github.com/apache/flink-cdc
- Flink CDC Documentation: 
https://nightlies.apache.org/flink/flink-cdc-docs-stable

After Flink community accepted this donation[1], we have completed software 
copyright signing, code repo migration, code cleanup, website migration, CI 
migration and github issues migration etc. 
Here I am particularly grateful to Hang Ruan, Zhongqaing Gong, Qingsheng Ren, 
Jiabao Sun, LvYanquan, loserwang1024 and other contributors for their 
contributions and help during this process！


For all previous contributors: The contribution process has slightly changed to 
align with the main Flink project. To report bugs or suggest new features, 
please open tickets 
Apache Jira (https://issues.apache.org/jira).  Note that we will no longer 
accept GitHub issues for these purposes.


Welcome to explore the new repository and documentation. Your feedback and 
contributions are invaluable as we continue to improve Flink CDC.

Thanks everyone for your support and happy exploring Flink CDC!

Best，
Leonard
[1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob

[ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Leonard Xu

Hi devs and users,

We are thrilled to announce that the donation of Flink CDC as a sub-project of 
Apache Flink has completed. We invite you to explore the new resources 
available:

- GitHub Repository: https://github.com/apache/flink-cdc
- Flink CDC Documentation: 
https://nightlies.apache.org/flink/flink-cdc-docs-stable

After Flink community accepted this donation[1], we have completed software 
copyright signing, code repo migration, code cleanup, website migration, CI 
migration and github issues migration etc. 
Here I am particularly grateful to Hang Ruan, Zhongqaing Gong, Qingsheng Ren, 
Jiabao Sun, LvYanquan, loserwang1024 and other contributors for their 
contributions and help during this process！


For all previous contributors: The contribution process has slightly changed to 
align with the main Flink project. To report bugs or suggest new features, 
please open tickets 
Apache Jira (https://issues.apache.org/jira).  Note that we will no longer 
accept GitHub issues for these purposes.


Welcome to explore the new repository and documentation. Your feedback and 
contributions are invaluable as we continue to improve Flink CDC.

Thanks everyone for your support and happy exploring Flink CDC!

Best，
Leonard
[1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob

Is there any options to control the file names in file sink

2024-03-20 Thread Lasse Nedergaard

Hi. 

Anyone know if it’s possible to control the file names eg change the uuid file 
names and extensions to something else. I know I can control the path but I 
would like to be able to set the file name and the extension based on the data 
in the stream. So I can’t use any general pre or postfix that are applied to 
all files. 

All ideas appreciate 

Med venlig hilsen / Best regards
Lasse Nedergaard

Re: flink operator 高可用任务偶发性报错unable to update ConfigMapLock

2024-03-20 Thread Yang Wang

这种一般是因为APIServer那边有问题导致单次的ConfigMap renew lease annotation的操作失败，Flink默认会重试的

如果你发现因为这个SocketTimeoutException原因导致了任务Failover，可以把下面两个参数调大
high-availability.kubernetes.leader-election.lease-duration: 60s
high-availability.kubernetes.leader-election.renew-deadline: 60s


Best,
Yang

On Tue, Mar 12, 2024 at 11:38 AM kellygeorg...@163.com <
kellygeorg...@163.com> wrote:

> 有没有高手指点一二？？？在线等
>
>
>
>  回复的原邮件 
> | 发件人 | kellygeorg...@163.com |
> | 日期 | 2024年03月11日 20:29 |
> | 收件人 | user-zh |
> | 抄送至 | |
> | 主题 | flink operator 高可用任务偶发性报错unable to update ConfigMapLock |
> jobmanager的报错如下所示，请问是什么原因？
> Exception occurred while renewing lock:Unable to update ConfigMapLock
>
> Caused by:io.fabric8.kubernetes.client.Kubernetes Client
> Exception:Operation:[replace] for kind:[ConfigMap] with name:[flink task
> xx- configmap] in namespace:[default]
>
>
> Caused by: Java.net.SocketTimeoutException:timeout
>
>
>
>
>
>
>

Re: can we use Scan Newly Added Tables without restarting the existing job ?

2024-03-20 Thread Hang Ruan

Hi, 3pang zhu.

This `Scan Newly added tables` feature requires restarting the job from the
savepoint. We cannot add new tables to the running job without restarting
by now.

Best,
Hang

3pang zhu  于2024年3月20日周三 15:22写道：

> this link has describe the usage for [Scan Newly Added Tables]
> https://nightlies.apache.org/flink/flink-cdc-docs-release-3.0/docs/connectors/legacy-flink-cdc-sources/mysql-cdc/#scan-newly-added-tables
> .
> if we can use if without restarting job. i have try this patch, use a
> schedule task in MysqlSnapshotSplitAssigner#open(), when added table more
> than twice, it occur this issue
> https://github.com/apache/flink-cdc/issues/2282
>
>
>  
> .../flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner.java
>  | 26 +++---
>  
> .../flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/reader/MySqlSourceReader.java
>  |  5 +++--
>  2 files changed, 26 insertions(+), 5 deletions(-)
>
> diff --cc
> flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner.jav
> index 0536a262,0536a262..d52acc26
> ---
> a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner.java
> +++
> b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner.java
> @@@ -56,7 -56,7 +56,9 @@@ import java.util.Set
>   import java.util.concurrent.CopyOnWriteArrayList;
>   import java.util.concurrent.ExecutorService;
>   import java.util.concurrent.Executors;
> ++import java.util.concurrent.ScheduledExecutorService;
>   import java.util.concurrent.ThreadFactory;
> ++import java.util.concurrent.TimeUnit;
>   import java.util.stream.Collectors;
>
>   import static
> com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.discoverCapturedTables;
> @@@ -94,6 -94,6 +96,7 @@@ public class MySqlSnapshotSplitAssigne
>   private MySqlChunkSplitter chunkSplitter;
>   private boolean isTableIdCaseSensitive;
>   private ExecutorService executor;
> ++private ScheduledExecutorService scheduledExecutor;
>
>   @Nullable private Long checkpointIdToFinish;
>
> @@@ -179,12 -179,12 +182,24 @@@
>   @Override
>   public void open() {
>   chunkSplitter.open();
> --discoveryCaptureTables();
> --captureNewlyAddedTables();
> --startAsynchronouslySplit();
> ++if (scheduledExecutor == null) {
> ++ThreadFactory threadFactory =
> ++new
> ThreadFactoryBuilder().setNameFormat("snapshot-splitting").build();
> ++this.scheduledExecutor =
> Executors.newSingleThreadScheduledExecutor(threadFactory);
> ++}
> ++scheduledExecutor.scheduleAtFixedRate(
> ++() -> {
> ++discoveryCaptureTables();
> ++captureNewlyAddedTables();
> ++startAsynchronouslySplit();
> ++},
> ++0,
> ++1,
> ++TimeUnit.MINUTES);
>   }
>
>   private void discoveryCaptureTables() {
> ++LOG.info("start discovery capture tables");
>   // discovery the tables lazily
>   if (needToDiscoveryTables()) {
>   long start = System.currentTimeMillis();
> @@@ -216,6 -216,6 +231,7 @@@
>   }
>
>   private void captureNewlyAddedTables() {
> ++LOG.info("start to capture newly added tables");
>   if (sourceConfig.isScanNewlyAddedTableEnabled()) {
>   // check whether we got newly added tables
>   try (JdbcConnection jdbc = openJdbcConnection(sourceConfig))
> {
> @@@ -282,6 -282,6 +298,7 @@@
>   }
>
>   private void startAsynchronouslySplit() {
> ++LOG.info("start asynchronously split");
>   if (chunkSplitter.hasNextChunk() || !remainingTables.isEmpty()) {
>   if (executor == null) {
>   ThreadFactory threadFactory =
> @@@ -497,6 -497,6 +514,9 @@@
>   if (executor != null) {
>   executor.shutdown();
>   }
> ++if (scheduledExecutor != null) {
> ++scheduledExecutor.shutdown();
> ++}
>   }
>

39 matches

Mail list logo