[jira] [Created] (FLINK-29800) Continuous failover will leak the inprogress output file
Aitozi created FLINK-29800: -- Summary: Continuous failover will leak the inprogress output file Key: FLINK-29800 URL: https://issues.apache.org/jira/browse/FLINK-29800 Project: Flink Issue Type: Bug Components: Connectors / FileSystem Reporter: Aitozi When running job which sink to the file system, the inprogress files will keep growing when job keeps failover, it will do harm to the filesystem. I think the clean up to the file which is currently written to should be performed when job failing -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Would like to fix "FLINK-27246 Code of method "processElement...grows beyond 64 KB"
Hi, Chielewski The author of JavaCodeSplitter Caizhi has kindly replied under the jira ticket, we can remove the discussion to jira now。 Best, Leonard > 2022年10月29日 上午3:55，Krzysztof Chmielewski 写道： > > Hi community, > I would like to work on fixing FLINK-27246 [1]. > I verified that it still happens on current master branch. > > I also did an initial investigation and I believe I've found what seems to > be a cause of this problem. I have added comment to the ticket [2] where > I've asked couple of question. > > I would appreciate for comment if my analysis is correct. > > Thanks, > Krzysztof Chielewski > > [1] https://issues.apache.org/jira/browse/FLINK-27246 > [2] > https://issues.apache.org/jira/browse/FLINK-27246?focusedCommentId=17625871=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17625871
Re: [ANNOUNCE] Apache Flink 1.16.0 released
Congrats! Finally! Best, Paul Lam > 2022年10月31日 11:58，Yuxin Tan 写道： > > Congrats! Glad to hear that. > > Best, > Yuxin > > > weijie guo 于2022年10月31日周一 11:51写道： > >> Congratulations, this is a version with many new features and thanks to >> everyone involved! >> >> Best regards, >> >> Weijie >> >> >> Yun Tang 于2022年10月31日周一 11:32写道： >> >>> Congratulations, and thanks to everyone involved! >>> >>> >>> Best >>> Yun Tang >>> >>> From: Zakelly Lan >>> Sent: Monday, October 31, 2022 10:44 >>> To: dev@flink.apache.org >>> Subject: Re: [ANNOUNCE] Apache Flink 1.16.0 released >>> >>> Good to know & Congrats! >>> >>> Thanks everyone involved! >>> >>> Best, >>> Zakelly >>> >>> On Mon, Oct 31, 2022 at 10:40 AM Becket Qin >> wrote: Hooray!! Congratulations to the team! Cheers, Jiangjie (Becket) Qin On Mon, Oct 31, 2022 at 9:57 AM Hang Ruan >>> wrote: > Congratulations! > > Best, > Hang > > Shengkai Fang 于2022年10月31日周一 09:40写道： > >> Congratulations! >> >> Best, >> Shengkai >> >> Hangxiang Yu 于2022年10月31日周一 09:38写道： >> >>> Congratulations! >>> Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the >> release. >>> >>> On Fri, Oct 28, 2022 at 7:35 PM Jing Ge >>> wrote: >>> Congrats! On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 >> wrote: > Congratulations and a big thanks to Chesnay, Martijn, Godfrey >>> and >> Xingbo > for the awesome work for 1.16! > > Best regards, > Qingsheng Ren > >> On Oct 28, 2022, at 14:46, Xingbo Huang >>> wrote: >> >> The Apache Flink community is very happy to announce the >>> release > of > Apache >> Flink 1.16.0, which is the first release for the Apache >> Flink >>> 1.16 > series. >> >> Apache Flink® is an open-source stream processing framework >>> for >> distributed, high-performing, always-available, and accurate >>> data > streaming >> applications. >> >> The release is available for download at: >> https://flink.apache.org/downloads.html >> >> Please check out the release blog post for an overview of >> the >> improvements for this release: >> >>> https://flink.apache.org/news/2022/10/28/1.16-announcement.html >> >> The full release notes are available in Jira: >> > >>> >> > >>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275 >> >> We would like to thank all contributors of the Apache Flink >> community >> who made this release possible! >> >> Regards, >> Chesnay, Martijn, Godfrey & Xingbo > >>> >>> -- >>> Best, >>> Hangxiang. >>> >> > >>> >>
Re: [ANNOUNCE] Apache Flink 1.16.0 released
Congrats! Glad to hear that. Best, Yuxin weijie guo 于2022年10月31日周一 11:51写道： > Congratulations, this is a version with many new features and thanks to > everyone involved! > > Best regards, > > Weijie > > > Yun Tang 于2022年10月31日周一 11:32写道： > > > Congratulations, and thanks to everyone involved! > > > > > > Best > > Yun Tang > > > > From: Zakelly Lan > > Sent: Monday, October 31, 2022 10:44 > > To: dev@flink.apache.org > > Subject: Re: [ANNOUNCE] Apache Flink 1.16.0 released > > > > Good to know & Congrats! > > > > Thanks everyone involved! > > > > Best, > > Zakelly > > > > On Mon, Oct 31, 2022 at 10:40 AM Becket Qin > wrote: > > > > > > Hooray!! Congratulations to the team! > > > > > > Cheers, > > > > > > Jiangjie (Becket) Qin > > > > > > On Mon, Oct 31, 2022 at 9:57 AM Hang Ruan > > wrote: > > > > > > > Congratulations! > > > > > > > > Best, > > > > Hang > > > > > > > > Shengkai Fang 于2022年10月31日周一 09:40写道： > > > > > > > > > Congratulations! > > > > > > > > > > Best, > > > > > Shengkai > > > > > > > > > > Hangxiang Yu 于2022年10月31日周一 09:38写道： > > > > > > > > > > > Congratulations! > > > > > > Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the > release. > > > > > > > > > > > > On Fri, Oct 28, 2022 at 7:35 PM Jing Ge > > wrote: > > > > > > > > > > > > > Congrats! > > > > > > > > > > > > > > On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 > wrote: > > > > > > > > > > > > > >> Congratulations and a big thanks to Chesnay, Martijn, Godfrey > > and > > > > > Xingbo > > > > > > >> for the awesome work for 1.16! > > > > > > >> > > > > > > >> Best regards, > > > > > > >> Qingsheng Ren > > > > > > >> > > > > > > >> > On Oct 28, 2022, at 14:46, Xingbo Huang > > wrote: > > > > > > >> > > > > > > > >> > The Apache Flink community is very happy to announce the > > release > > > > of > > > > > > >> Apache > > > > > > >> > Flink 1.16.0, which is the first release for the Apache > Flink > > 1.16 > > > > > > >> series. > > > > > > >> > > > > > > > >> > Apache Flink® is an open-source stream processing framework > > for > > > > > > >> > distributed, high-performing, always-available, and accurate > > data > > > > > > >> streaming > > > > > > >> > applications. > > > > > > >> > > > > > > > >> > The release is available for download at: > > > > > > >> > https://flink.apache.org/downloads.html > > > > > > >> > > > > > > > >> > Please check out the release blog post for an overview of > the > > > > > > >> > improvements for this release: > > > > > > >> > > > https://flink.apache.org/news/2022/10/28/1.16-announcement.html > > > > > > >> > > > > > > > >> > The full release notes are available in Jira: > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275 > > > > > > >> > > > > > > > >> > We would like to thank all contributors of the Apache Flink > > > > > community > > > > > > >> > who made this release possible! > > > > > > >> > > > > > > > >> > Regards, > > > > > > >> > Chesnay, Martijn, Godfrey & Xingbo > > > > > > >> > > > > > > > > > > > > > > > > > > > -- > > > > > > Best, > > > > > > Hangxiang. > > > > > > > > > > > > > > > > > >
Re: [ANNOUNCE] Apache Flink 1.16.0 released
Congratulations, this is a version with many new features and thanks to everyone involved! Best regards, Weijie Yun Tang 于2022年10月31日周一 11:32写道： > Congratulations, and thanks to everyone involved! > > > Best > Yun Tang > > From: Zakelly Lan > Sent: Monday, October 31, 2022 10:44 > To: dev@flink.apache.org > Subject: Re: [ANNOUNCE] Apache Flink 1.16.0 released > > Good to know & Congrats! > > Thanks everyone involved! > > Best, > Zakelly > > On Mon, Oct 31, 2022 at 10:40 AM Becket Qin wrote: > > > > Hooray!! Congratulations to the team! > > > > Cheers, > > > > Jiangjie (Becket) Qin > > > > On Mon, Oct 31, 2022 at 9:57 AM Hang Ruan > wrote: > > > > > Congratulations! > > > > > > Best, > > > Hang > > > > > > Shengkai Fang 于2022年10月31日周一 09:40写道： > > > > > > > Congratulations! > > > > > > > > Best, > > > > Shengkai > > > > > > > > Hangxiang Yu 于2022年10月31日周一 09:38写道： > > > > > > > > > Congratulations! > > > > > Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the release. > > > > > > > > > > On Fri, Oct 28, 2022 at 7:35 PM Jing Ge > wrote: > > > > > > > > > > > Congrats! > > > > > > > > > > > > On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 wrote: > > > > > > > > > > > >> Congratulations and a big thanks to Chesnay, Martijn, Godfrey > and > > > > Xingbo > > > > > >> for the awesome work for 1.16! > > > > > >> > > > > > >> Best regards, > > > > > >> Qingsheng Ren > > > > > >> > > > > > >> > On Oct 28, 2022, at 14:46, Xingbo Huang > wrote: > > > > > >> > > > > > > >> > The Apache Flink community is very happy to announce the > release > > > of > > > > > >> Apache > > > > > >> > Flink 1.16.0, which is the first release for the Apache Flink > 1.16 > > > > > >> series. > > > > > >> > > > > > > >> > Apache Flink® is an open-source stream processing framework > for > > > > > >> > distributed, high-performing, always-available, and accurate > data > > > > > >> streaming > > > > > >> > applications. > > > > > >> > > > > > > >> > The release is available for download at: > > > > > >> > https://flink.apache.org/downloads.html > > > > > >> > > > > > > >> > Please check out the release blog post for an overview of the > > > > > >> > improvements for this release: > > > > > >> > > https://flink.apache.org/news/2022/10/28/1.16-announcement.html > > > > > >> > > > > > > >> > The full release notes are available in Jira: > > > > > >> > > > > > > >> > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275 > > > > > >> > > > > > > >> > We would like to thank all contributors of the Apache Flink > > > > community > > > > > >> > who made this release possible! > > > > > >> > > > > > > >> > Regards, > > > > > >> > Chesnay, Martijn, Godfrey & Xingbo > > > > > >> > > > > > > > > > > > > > > > > -- > > > > > Best, > > > > > Hangxiang. > > > > > > > > > > > > >
Re: [ANNOUNCE] Apache Flink 1.16.0 released
Congratulations, and thanks to everyone involved! Best Yun Tang From: Zakelly Lan Sent: Monday, October 31, 2022 10:44 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] Apache Flink 1.16.0 released Good to know & Congrats! Thanks everyone involved! Best, Zakelly On Mon, Oct 31, 2022 at 10:40 AM Becket Qin wrote: > > Hooray!! Congratulations to the team! > > Cheers, > > Jiangjie (Becket) Qin > > On Mon, Oct 31, 2022 at 9:57 AM Hang Ruan wrote: > > > Congratulations! > > > > Best, > > Hang > > > > Shengkai Fang 于2022年10月31日周一 09:40写道： > > > > > Congratulations! > > > > > > Best, > > > Shengkai > > > > > > Hangxiang Yu 于2022年10月31日周一 09:38写道： > > > > > > > Congratulations! > > > > Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the release. > > > > > > > > On Fri, Oct 28, 2022 at 7:35 PM Jing Ge wrote: > > > > > > > > > Congrats! > > > > > > > > > > On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 wrote: > > > > > > > > > >> Congratulations and a big thanks to Chesnay, Martijn, Godfrey and > > > Xingbo > > > > >> for the awesome work for 1.16! > > > > >> > > > > >> Best regards, > > > > >> Qingsheng Ren > > > > >> > > > > >> > On Oct 28, 2022, at 14:46, Xingbo Huang wrote: > > > > >> > > > > > >> > The Apache Flink community is very happy to announce the release > > of > > > > >> Apache > > > > >> > Flink 1.16.0, which is the first release for the Apache Flink 1.16 > > > > >> series. > > > > >> > > > > > >> > Apache Flink® is an open-source stream processing framework for > > > > >> > distributed, high-performing, always-available, and accurate data > > > > >> streaming > > > > >> > applications. > > > > >> > > > > > >> > The release is available for download at: > > > > >> > https://flink.apache.org/downloads.html > > > > >> > > > > > >> > Please check out the release blog post for an overview of the > > > > >> > improvements for this release: > > > > >> > https://flink.apache.org/news/2022/10/28/1.16-announcement.html > > > > >> > > > > > >> > The full release notes are available in Jira: > > > > >> > > > > > >> > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275 > > > > >> > > > > > >> > We would like to thank all contributors of the Apache Flink > > > community > > > > >> > who made this release possible! > > > > >> > > > > > >> > Regards, > > > > >> > Chesnay, Martijn, Godfrey & Xingbo > > > > >> > > > > > > > > > > > > > -- > > > > Best, > > > > Hangxiang. > > > > > > > > >
Re: [ANNOUNCE] Apache Flink 1.16.0 released
Good to know & Congrats! Thanks everyone involved! Best, Zakelly On Mon, Oct 31, 2022 at 10:40 AM Becket Qin wrote: > > Hooray!! Congratulations to the team! > > Cheers, > > Jiangjie (Becket) Qin > > On Mon, Oct 31, 2022 at 9:57 AM Hang Ruan wrote: > > > Congratulations! > > > > Best, > > Hang > > > > Shengkai Fang 于2022年10月31日周一 09:40写道： > > > > > Congratulations! > > > > > > Best, > > > Shengkai > > > > > > Hangxiang Yu 于2022年10月31日周一 09:38写道： > > > > > > > Congratulations! > > > > Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the release. > > > > > > > > On Fri, Oct 28, 2022 at 7:35 PM Jing Ge wrote: > > > > > > > > > Congrats! > > > > > > > > > > On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 wrote: > > > > > > > > > >> Congratulations and a big thanks to Chesnay, Martijn, Godfrey and > > > Xingbo > > > > >> for the awesome work for 1.16! > > > > >> > > > > >> Best regards, > > > > >> Qingsheng Ren > > > > >> > > > > >> > On Oct 28, 2022, at 14:46, Xingbo Huang wrote: > > > > >> > > > > > >> > The Apache Flink community is very happy to announce the release > > of > > > > >> Apache > > > > >> > Flink 1.16.0, which is the first release for the Apache Flink 1.16 > > > > >> series. > > > > >> > > > > > >> > Apache Flink® is an open-source stream processing framework for > > > > >> > distributed, high-performing, always-available, and accurate data > > > > >> streaming > > > > >> > applications. > > > > >> > > > > > >> > The release is available for download at: > > > > >> > https://flink.apache.org/downloads.html > > > > >> > > > > > >> > Please check out the release blog post for an overview of the > > > > >> > improvements for this release: > > > > >> > https://flink.apache.org/news/2022/10/28/1.16-announcement.html > > > > >> > > > > > >> > The full release notes are available in Jira: > > > > >> > > > > > >> > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275 > > > > >> > > > > > >> > We would like to thank all contributors of the Apache Flink > > > community > > > > >> > who made this release possible! > > > > >> > > > > > >> > Regards, > > > > >> > Chesnay, Martijn, Godfrey & Xingbo > > > > >> > > > > > > > > > > > > > -- > > > > Best, > > > > Hangxiang. > > > > > > > > >
Re: [ANNOUNCE] Apache Flink 1.16.0 released
Hooray!! Congratulations to the team! Cheers, Jiangjie (Becket) Qin On Mon, Oct 31, 2022 at 9:57 AM Hang Ruan wrote: > Congratulations! > > Best, > Hang > > Shengkai Fang 于2022年10月31日周一 09:40写道： > > > Congratulations! > > > > Best, > > Shengkai > > > > Hangxiang Yu 于2022年10月31日周一 09:38写道： > > > > > Congratulations! > > > Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the release. > > > > > > On Fri, Oct 28, 2022 at 7:35 PM Jing Ge wrote: > > > > > > > Congrats! > > > > > > > > On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 wrote: > > > > > > > >> Congratulations and a big thanks to Chesnay, Martijn, Godfrey and > > Xingbo > > > >> for the awesome work for 1.16! > > > >> > > > >> Best regards, > > > >> Qingsheng Ren > > > >> > > > >> > On Oct 28, 2022, at 14:46, Xingbo Huang wrote: > > > >> > > > > >> > The Apache Flink community is very happy to announce the release > of > > > >> Apache > > > >> > Flink 1.16.0, which is the first release for the Apache Flink 1.16 > > > >> series. > > > >> > > > > >> > Apache Flink® is an open-source stream processing framework for > > > >> > distributed, high-performing, always-available, and accurate data > > > >> streaming > > > >> > applications. > > > >> > > > > >> > The release is available for download at: > > > >> > https://flink.apache.org/downloads.html > > > >> > > > > >> > Please check out the release blog post for an overview of the > > > >> > improvements for this release: > > > >> > https://flink.apache.org/news/2022/10/28/1.16-announcement.html > > > >> > > > > >> > The full release notes are available in Jira: > > > >> > > > > >> > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275 > > > >> > > > > >> > We would like to thank all contributors of the Apache Flink > > community > > > >> > who made this release possible! > > > >> > > > > >> > Regards, > > > >> > Chesnay, Martijn, Godfrey & Xingbo > > > >> > > > > > > > > > > -- > > > Best, > > > Hangxiang. > > > > > >
Re: [VOTE] Dedicated AWS externalized connector repo
+1 (binding) Thanks, Jiangjie (Becket) Qin On Mon, Oct 31, 2022 at 10:17 AM Jark Wu wrote: > +1 (binding) > > Best, > Jark > > > 2022年10月29日 03:11，Jing Ge 写道： > > > > +1 (non-binding) > > > > Thanks! > > > > Best Regards, > > Jing > > > > On Fri, Oct 28, 2022 at 5:29 PM Samrat Deb > wrote: > > > >> +1 (non binding) > >> > >> Thanks for driving Danny > >> > >> Bests > >> Samrat > >> > >> On Fri, 28 Oct 2022 at 8:36 PM, Ahmed Hamdy > wrote: > >> > >>> +1 (non-binding) > >>> Regards, > >>> Ahmed > >>> > >>> On Thu, 27 Oct 2022 at 08:38, Teoh, Hong > > >>> wrote: > >>> > +1 (non-binding) > > Thanks for driving this, Danny! > > Hong > > On 26/10/2022, 08:14, "Martijn Visser" > >>> wrote: > > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you can confirm the sender > >> and > know the content is safe. > > > > +1 binding > > Thanks Danny! > > On Wed, Oct 26, 2022 at 8:48 AM Danny Cranmer < > >>> dannycran...@apache.org > > > wrote: > > > Hello all, > > > > As discussed in the discussion thread [1], I propose to create a > dedicated > > repository for AWS connectors called flink-connector-aws. This > >> will > house > > 3x connectors: Amazon Kinesis Data Streams, Amazon Kinesis Data > Firehose > > and Amazon DynamoDB and any future AWS connectors. We will also > externalize > > the AWS base module from the main Flink repository [2] and > >> create a > parent > > pom for version management. > > > > All modules within this repository will share the same version, > >> and > be > > released/evolved together. We will adhere to the common Flink > >> rules > [3] for > > connector development. > > > > Motivation: grouping AWS connectors together will reduce the > >> number > of > > connector releases, simplify development, dependency management > >> and > > versioning for users. > > > > Voting schema: > > Consensus, committers have binding votes, open for at least 72 > >>> hours. > > > > [1] > >>> https://lists.apache.org/thread/swp4bs8407gtsgn2gh0k3wx1m4o3kqqp > > [2] > > > > > > >>> > >> > https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base > > [3] > > > > > > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development > > > > > >>> > >> > >
Re: [DISCUSS] Delete useless branches from Flink repository
Thank you for driving this effort, Leonard! Best, Jark > 2022年10月28日 11:25，Yuan Mei 写道： > > Hey Leonard, > > Thanks for your efforts to clean up our repo! > > Best > Yuan > > > > On Thu, Oct 27, 2022 at 11:55 PM Leonard Xu wrote: > >> Thanks Matthias and Chesnay for the quick ACK. >> >> I’ve deleted following branches. >>> FLINK-29638-1.15 >>> FLINK-29638-1.16 >>> 28733 >>> revert-16606-materialization_on_runtime >>> release0 Updated 2 years ago >>> docs_experimental__docsUpdated 2 years ago >>> docs_experimental__docs_compile Updated 3 years ago >> >> BTW, I fork a Flink repo [1] as a backup to avoid we delete required >> branches. >> >> >> Best, >> Leonard >> [1] https://github.com/flink-tpc-ds/flink-221027
Re: [VOTE] Dedicated AWS externalized connector repo
+1 (binding) Best, Jark > 2022年10月29日 03:11，Jing Ge 写道： > > +1 (non-binding) > > Thanks! > > Best Regards, > Jing > > On Fri, Oct 28, 2022 at 5:29 PM Samrat Deb wrote: > >> +1 (non binding) >> >> Thanks for driving Danny >> >> Bests >> Samrat >> >> On Fri, 28 Oct 2022 at 8:36 PM, Ahmed Hamdy wrote: >> >>> +1 (non-binding) >>> Regards, >>> Ahmed >>> >>> On Thu, 27 Oct 2022 at 08:38, Teoh, Hong >>> wrote: >>> +1 (non-binding) Thanks for driving this, Danny! Hong On 26/10/2022, 08:14, "Martijn Visser" >>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender >> and know the content is safe. +1 binding Thanks Danny! On Wed, Oct 26, 2022 at 8:48 AM Danny Cranmer < >>> dannycran...@apache.org > wrote: > Hello all, > > As discussed in the discussion thread [1], I propose to create a dedicated > repository for AWS connectors called flink-connector-aws. This >> will house > 3x connectors: Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose > and Amazon DynamoDB and any future AWS connectors. We will also externalize > the AWS base module from the main Flink repository [2] and >> create a parent > pom for version management. > > All modules within this repository will share the same version, >> and be > released/evolved together. We will adhere to the common Flink >> rules [3] for > connector development. > > Motivation: grouping AWS connectors together will reduce the >> number of > connector releases, simplify development, dependency management >> and > versioning for users. > > Voting schema: > Consensus, committers have binding votes, open for at least 72 >>> hours. > > [1] >>> https://lists.apache.org/thread/swp4bs8407gtsgn2gh0k3wx1m4o3kqqp > [2] > > >>> >> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base > [3] > > >>> >> https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development > >>> >>
Re: [ANNOUNCE] Apache Flink 1.16.0 released
Congratulations! Best, Hang Shengkai Fang 于2022年10月31日周一 09:40写道： > Congratulations! > > Best, > Shengkai > > Hangxiang Yu 于2022年10月31日周一 09:38写道： > > > Congratulations! > > Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the release. > > > > On Fri, Oct 28, 2022 at 7:35 PM Jing Ge wrote: > > > > > Congrats! > > > > > > On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 wrote: > > > > > >> Congratulations and a big thanks to Chesnay, Martijn, Godfrey and > Xingbo > > >> for the awesome work for 1.16! > > >> > > >> Best regards, > > >> Qingsheng Ren > > >> > > >> > On Oct 28, 2022, at 14:46, Xingbo Huang wrote: > > >> > > > >> > The Apache Flink community is very happy to announce the release of > > >> Apache > > >> > Flink 1.16.0, which is the first release for the Apache Flink 1.16 > > >> series. > > >> > > > >> > Apache Flink® is an open-source stream processing framework for > > >> > distributed, high-performing, always-available, and accurate data > > >> streaming > > >> > applications. > > >> > > > >> > The release is available for download at: > > >> > https://flink.apache.org/downloads.html > > >> > > > >> > Please check out the release blog post for an overview of the > > >> > improvements for this release: > > >> > https://flink.apache.org/news/2022/10/28/1.16-announcement.html > > >> > > > >> > The full release notes are available in Jira: > > >> > > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275 > > >> > > > >> > We would like to thank all contributors of the Apache Flink > community > > >> > who made this release possible! > > >> > > > >> > Regards, > > >> > Chesnay, Martijn, Godfrey & Xingbo > > >> > > > > > > > -- > > Best, > > Hangxiang. > > >
Re: [ANNOUNCE] Apache Flink 1.16.0 released
Congratulations! Best, Shengkai Hangxiang Yu 于2022年10月31日周一 09:38写道： > Congratulations! > Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the release. > > On Fri, Oct 28, 2022 at 7:35 PM Jing Ge wrote: > > > Congrats! > > > > On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 wrote: > > > >> Congratulations and a big thanks to Chesnay, Martijn, Godfrey and Xingbo > >> for the awesome work for 1.16! > >> > >> Best regards, > >> Qingsheng Ren > >> > >> > On Oct 28, 2022, at 14:46, Xingbo Huang wrote: > >> > > >> > The Apache Flink community is very happy to announce the release of > >> Apache > >> > Flink 1.16.0, which is the first release for the Apache Flink 1.16 > >> series. > >> > > >> > Apache Flink® is an open-source stream processing framework for > >> > distributed, high-performing, always-available, and accurate data > >> streaming > >> > applications. > >> > > >> > The release is available for download at: > >> > https://flink.apache.org/downloads.html > >> > > >> > Please check out the release blog post for an overview of the > >> > improvements for this release: > >> > https://flink.apache.org/news/2022/10/28/1.16-announcement.html > >> > > >> > The full release notes are available in Jira: > >> > > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275 > >> > > >> > We would like to thank all contributors of the Apache Flink community > >> > who made this release possible! > >> > > >> > Regards, > >> > Chesnay, Martijn, Godfrey & Xingbo > >> > > > > -- > Best, > Hangxiang. >
Re: [ANNOUNCE] Apache Flink 1.16.0 released
Congratulations! Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the release. On Fri, Oct 28, 2022 at 7:35 PM Jing Ge wrote: > Congrats! > > On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 wrote: > >> Congratulations and a big thanks to Chesnay, Martijn, Godfrey and Xingbo >> for the awesome work for 1.16! >> >> Best regards, >> Qingsheng Ren >> >> > On Oct 28, 2022, at 14:46, Xingbo Huang wrote: >> > >> > The Apache Flink community is very happy to announce the release of >> Apache >> > Flink 1.16.0, which is the first release for the Apache Flink 1.16 >> series. >> > >> > Apache Flink® is an open-source stream processing framework for >> > distributed, high-performing, always-available, and accurate data >> streaming >> > applications. >> > >> > The release is available for download at: >> > https://flink.apache.org/downloads.html >> > >> > Please check out the release blog post for an overview of the >> > improvements for this release: >> > https://flink.apache.org/news/2022/10/28/1.16-announcement.html >> > >> > The full release notes are available in Jira: >> > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275 >> > >> > We would like to thank all contributors of the Apache Flink community >> > who made this release possible! >> > >> > Regards, >> > Chesnay, Martijn, Godfrey & Xingbo >> > -- Best, Hangxiang.
Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
+1 (non-binding) On Sat, Oct 29, 2022 at 2:21 AM Tzu-Li (Gordon) Tai wrote: > +1 > > On Fri, Oct 28, 2022 at 10:21 AM Konstantin Knauf > wrote: > > > +1 (binding) > > > > Am Fr., 28. Okt. 2022 um 16:58 Uhr schrieb Piotr Nowojski < > > pnowoj...@apache.org>: > > > > > Hi, > > > > > > As discussed on the dev mailing list [0] I would like to start a vote > to > > > drop support of older savepoint formats (for Flink versions older than > > > 1.8). You can find the original explanation from the aforementioned dev > > > mailing list thread at the bottom of this message. > > > > > > Draft PR containing the proposed change you can find here: > > > https://github.com/apache/flink/pull/21056 > > > > > > Vote will be open at least until Wednesday, November 2nd 18:00 CET. > > > > > > Best, > > > Piotrek > > > > > > [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w > > > > > > I would like to open a discussion to remove the long deprecated > > > (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the > related > > > code. > > > > > > The motivation behind this move is two fold. One reason is that it > > > complicates our code base unnecessarily and creates confusion on how to > > > actually implement custom serializers. The immediate reason is that I > > > wanted to clean up Flink's configuration stack a bit and refactor the > > > ExecutionConfig class [2]. This refactor would keep the API > compatibility > > > of the ExecutionConfig, but it would break savepoint compatibility with > > > snapshots written with some of the old serializers, which had > > > ExecutionConfig as a field and were serialized in the snapshot. This > > issue > > > has been resolved by the introduction of TypeSerializerSnapshot in > Flink > > > 1.7 [3], where serializers are no longer part of the snapshot. > > > > > > TypeSerializerConfigSnapshot has been deprecated and no longer used by > > > built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged > > to > > > migrate to TypeSerializerSnapshot since then with their own custom > > > serializers. That has been plenty of time for the migration. > > > > > > This proposal would have the following impact for the users: > > > 1. we would drop support for recovery from savepoints taken with Flink > < > > > 1.7.0 for all built in types serializers > > > 2. we would drop support for recovery from savepoints taken with Flink > < > > > 1.8.0 for built in kryo serializers > > > 3. we would drop support for recovery from savepoints taken with Flink > < > > > 1.17 for custom serializers using deprecated > TypeSerializerConfigSnapshot > > > > > > 1. and 2. would have a simple migration path. Users migrating from > those > > > old savepoints would have to first start his job using a Flink version > > from > > > the [1.8, 1.16] range, and take a new savepoint that would be > compatible > > > with Flink 1.17. > > > 3. This is a bit more problematic, because users would have to first > > > migrate their own custom serializers to use TypeSerializerSnapshot > > (using a > > > Flink version from the [1.8, 1.16]), take a savepoint, and only then > > > migrate to Flink 1.17. However users had already 4 years to migrate, > > which > > > in my opinion has been plenty of time to do so. > > > > > > As a side effect, we could also drop support for some of the legacy > > > metadata serializers from LegacyStateMetaInfoReaders and potentially > > other > > > places that we are keeping for the sake of compatibility with old > > > savepoints. > > > > > > [1] > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html > > > [2] https://issues.apache.org/jira/browse/FLINK-29379 > > > [3] https://issues.apache.org/jira/browse/FLINK-9377 > > > [4] https://issues.apache.org/jira/browse/FLINK-9376 > > > [5] https://issues.apache.org/jira/browse/FLINK-11323 > > > > > > > > > -- > > https://twitter.com/snntrable > > https://github.com/knaufk > > > -- Best, Hangxiang.
Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility
Sure, Thanks a lot for the reminder !
On Fri, Oct 28, 2022 at 3:59 PM Dawid Wysakowicz
wrote:
> I see the majority of the community prefers the not immediate breaking
> of the API. Even though I have different preference I am more than fine
> to commit to the others choice. Just please make sure it is prominently
> documented that one or the other methods MUST be implemented in any new
> serializer.
>
> Best,
>
> Dawid
>
> On 27/10/2022 08:33, Yun Gao wrote:
> > Hi,
> > I have discussed offline with Hangxiang and Yuan, and the current
> > proposal also looks good to me.
> > Thanks Hangxiang for driving this.
> > Best,
> > Yun Gao
> > --
> > From:Yuan Mei
> > Send Time:2022 Oct. 26 (Wed.) 12:20
> > To:dev
> > Subject:Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility
> > Hey Huangxiang,
> > The section of `Rejected Alternatives` may also need an update.
> > Current plan sounds like a reasonable one. I am fine with it.
> > Thanks for driving this.
> > Best
> > Yuan
> > On Tue, Oct 25, 2022 at 5:11 PM Hangxiang Yu
> wrote:
> >> （Resend the mail to fix the format issue）
> >> Hi, everyone.
> >>
> >> Thanks for your suggestions!
> >>
> >> Let me summarize the remaining questions in the thread and share my
> ideas
> >> based on your suggestions:
> >>
> >>
> >> 1. Should we put the new opposite interface in TypeSerializer or
> >> TypeSerializerSnapshot ?
> >>
> >> Just as I replied to Dawid, I'd like to put it in
> >> TypeSerializerSnapshot so that we could still follow the contract
> between
> >> two classes and make later code migration easier based on current tools.
> >>
> >> Thanks Dawid for the initial suggestion and Godfrey for the additional
> >> supplement.
> >>
> >>
> >>
> >> 2. Do we just break changes and make user codes incompatible or make
> sure
> >> compatibility using a more suitable migration plan ?
> >>
> >> I agree with Yuan that we should make sure that user jobs still work
> >> without modifying any codes before removing the deprecated method.
> >>
> >> Thanks Yuan for the migration plan. Let me try to add something to the
> >> suggestion of Yuan:
> >>
> >>
> >> a. In Step 1, I prefer to make the new interface like:
> >>
> >>
> >> default TypeSerializerSchemaCompatibility
> >> resolveSchemaCompatibility(
> >> // Use 'oldSnapshot' not 'oldSerializer'
> >> TypeSerializerSnapshot oldSnapshot) {
> >> return INCOMPATIBLE;
> >> }
> >> }
> >>
> >> I think using 'oldSnapshot' as the parameter will make the
> >> logic clear --- TypeSerializerSnapshot will take all responsibility for
> >> compatibility checks.
> >> BTW, It's also easy to migrate original check logic to this
> >> interface.
> >>
> >> b. In Step 1, In addition to introducing default implementations
> >> for both interfaces, we also need to implement the new interface in all
> >> inner TypeSerializerSnapshots.
> >> Users may implement their own serializers based on inner
> >> serializers, we should make sure that the new interface of inner
> >> TypeSerializerSnapshots is usable.
> >>
> >> Then I think it could work for both old custom serializers or new
> >> custom serializers.
> >> No matter which interface the user implements, it could always work.
> >> Of course, we will deprecate the old interface and encourage users to
> >> use the new one.
> >>
> >>
> >> 3. Do we need to squash this with
> >> https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w <
> https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w > ?
> >> We will not break the compatibility based on 2, so it's not necessary
> >> to squash them together.
> >>
> >>
> >> Do you have any other suggestions ? Look forward to your reply!
> >>
> >> On Tue, Oct 25, 2022 at 1:31 PM Hangxiang Yu
> wrote:
> >>
> >>> Hi, everyone.
> >>>
> >>> Thanks for your suggestions!
> >>>
> >>> Let me summarize the remaining questions in the thread and share my
> ideas
> >>> based on your suggestions:
> >>>
> >>> 1. Should we put the new opposite interface in TypeSerializer or
> >>> TypeSerializerSnapshot ?
> >>>
> >>> Just as I replied to Dawid, I'd like to put it in
> TypeSerializerSnapshot
> >>> so that we could still follow the contract between two classes and make
> >>> later code migration easier based on current tools.
> >>>
> >>> Thanks Dawid for the initial suggestion and Godfrey for the additional
> >>> supplement.
> >>>
> >>>
> >>>
> >>> 1. Do we just break changes and make user codes incompatible or make
> >>> sure compatibility using a more suitable migration plan ?
> >>>
> >>> I agree with Yuan that we should make sure that user jobs still work
> >>> without modifying any codes before removing the deprecated method.
> >>>
> >>> Thanks Yuan for the migration plan. Let me try to add something to the
> >>> suggestion of Yuan:
> >>>
> >>> 1. In Step 1, I prefer to make the new interface like:
> >>>
> >>> default TypeSerializerSchemaCompatibility
> resolveSchemaCompatibility(
> >>>
> >>>
[jira] [Created] (FLINK-29799) How to override ConfigMap values while deploying the Operator via OLM?
Cansu Kavili created FLINK-29799: Summary: How to override ConfigMap values while deploying the Operator via OLM? Key: FLINK-29799 URL: https://issues.apache.org/jira/browse/FLINK-29799 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Cansu Kavili Hi, Flink Kubernetes Operator is great - thank you! I deployed it via OLM on OpenShift 4.10 and would like to override some config from `flink-operator-config` CM. When I override a value manually, it persists (which is strange - it's a resource managed via Operator) so when I change a value in CM and restart Operator, it works. How can I give the parameters I want while installing the operator? Some operators support setting environment variables in `Subscription` object but I couldn't find such a thing in the documentation. I can do it easily with Helm installation but I want to use OLM. It'd be appreciated if you can guide me. Many many thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29798) Rename K8s operator client code module
Márton Balassi created FLINK-29798: -- Summary: Rename K8s operator client code module Key: FLINK-29798 URL: https://issues.apache.org/jira/browse/FLINK-29798 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Affects Versions: kubernetes-operator-1.2.0 Reporter: Márton Balassi Assignee: Márton Balassi Fix For: kubernetes-operator-1.3.0 The example code module in the k8s operator is named simply kubernetes-client-examples, and thus is published like so: [https://repo1.maven.org/maven2/org/apache/flink/kubernetes-client-examples/1.2.0/] We should make this more specific. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-263: Improve resolving schema compatibility
+1 (binding) Thanks Hangxiang for driving this and thanks all for the thorough discussion. Best Regards, Yu On Fri, 28 Oct 2022 at 16:01, Dawid Wysakowicz wrote: > +1, > > Best, > > Dawid > > On 28/10/2022 08:08, godfrey he wrote: > > +1 (binding) > > > > Thanks for driving this! > > > > Best, > > Godfrey > > > > Yun Gao 于2022年10月28日周五 13:50写道： > >> +1 (binding) > >> > >> Thanks Hangxiang for driving the FLIP. > >> > >> Best, > >> Yun Gao > >> > >> > >> > >> > >> --Original Mail -- > >> Sender:Zakelly Lan > >> Send Date:Fri Oct 28 12:27:01 2022 > >> Recipients:Flink Dev > >> Subject:Re: [VOTE] FLIP-263: Improve resolving schema compatibility > >> Hi Hangxiang, > >> > >> The current plan looks good to me, +1 (non-binding). Thanks for driving > this. > >> > >> Best, > >> Zakelly > >> > >> On Fri, Oct 28, 2022 at 11:18 AM Yuan Mei > wrote: > >>> +1 (binding) > >>> > >>> Thanks for driving this. > >>> > >>> Best > >>> Yuan > >>> > >>> On Fri, Oct 28, 2022 at 11:17 AM yanfei lei > wrote: > >>> > +1(non-binding) and thanks for Hangxiang's driving. > > > > Hangxiang Yu 于2022年10月28日周五 09:24写道： > > > Hi everyone, > > > > I'd like to start the vote for FLIP-263 [1]. > > > > Thanks for your feedback and the discussion in [2][3]. > > > > The vote will be open for at least 72 hours. > > > > Best regards, > > Hangxiang. > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility > > [2] https://lists.apache.org/thread/4w36oof8dh28b9f593sgtk21o8qh8qx4 > > > > [3] https://lists.apache.org/thread/t0bdkx1161rlbnsf06x0kswb05mch164 > > > > -- > Best, > Yanfei > >
