Re: Flink RocketMQ Connector
hi casel Flink RocketMQ connector 是由 RockeMQ 社区维护的， 对应的项目地址是: https://github.com/apache/rocketmq-flink 这个版本默认的消息是格式 DELIMIT 格式(默认消息是 String，按分隔符进行分割)， 只能指定消息的列分隔符. best, feng On Fri, May 26, 2023 at 7:44 PM casel.chen wrote: > 有没有Flink RocketMQ官方连接器? 需要自己开发吗？Flink生态组件网址（用户上传自己开发的连接器格式什么的）是什么？
Re: [ANNOUNCE] Apache Flink 1.16.2 released
Hi Weijie, Thanks again for your effort. I was wondering if there were any obstacles you had to overcome while releasing 1.16.2 and 1.17.1 that could lead us to any improvement wrt the release process and management? Best regards, Jing On Fri, May 26, 2023 at 4:41 PM Martijn Visser wrote: > Thank you Weijie and those who helped with testing! > > On Fri, May 26, 2023 at 1:06 PM weijie guo > wrote: > > > The Apache Flink community is very happy to announce the release of > > Apache Flink 1.16.2, which is the second bugfix release for the Apache > > Flink 1.16 series. > > > > > > > > Apache Flink® is an open-source stream processing framework for > > distributed, high-performing, always-available, and accurate data > > streaming applications. > > > > > > > > The release is available for download at: > > > > https://flink.apache.org/downloads.html > > > > > > > > Please check out the release blog post for an overview of the > > improvements for this bugfix release: > > > > https://flink.apache.org/news/2023/05/25/release-1.16.2.html > > > > > > > > The full release notes are available in Jira: > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352765 > > > > > > > > We would like to thank all contributors of the Apache Flink community > > who made this release possible! > > > > > > > > Feel free to reach out to the release managers (or respond to this > > thread) with feedback on the release process. Our goal is to > > constantly improve the release process. Feedback on what could be > > improved or things that didn't go so well are appreciated. > > > > > > > > Regards, > > > > Release Manager > > >
Re: [ANNOUNCE] Apache Flink 1.17.1 released
Hi Weijie, That is earlier than I expected! Thank you so much for your effort! Best regards, Jing On Fri, May 26, 2023 at 4:44 PM Martijn Visser wrote: > Same here as with Flink 1.16.2, thank you Weijie and those who helped with > testing! > > On Fri, May 26, 2023 at 1:08 PM weijie guo > wrote: > >> >> The Apache Flink community is very happy to announce the release of Apache >> Flink 1.17.1, which is the first bugfix release for the Apache Flink 1.17 >> series. >> >> >> >> >> Apache Flink® is an open-source stream processing framework for distributed, >> high-performing, always-available, and accurate data streaming applications. >> >> >> >> The release is available for download at: >> >> https://flink.apache.org/downloads.html >> >> >> >> >> Please check out the release blog post for an overview of the improvements >> for this bugfix release: >> >> https://flink.apache.org/news/2023/05/25/release-1.17.1.html >> >> >> >> The full release notes are available in Jira: >> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352886 >> >> >> >> >> We would like to thank all contributors of the Apache Flink community who >> made this release possible! >> >> >> >> >> Feel free to reach out to the release managers (or respond to this thread) >> with feedback on the release process. Our goal is to constantly improve the >> release process. Feedback on what could be improved or things that didn't go >> so well are appreciated. >> >> >> >> Regards, >> >> Release Manager >> >
Re: [ANNOUNCE] Apache Flink 1.17.1 released
Same here as with Flink 1.16.2, thank you Weijie and those who helped with testing! On Fri, May 26, 2023 at 1:08 PM weijie guo wrote: > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.17.1, which is the first bugfix release for the Apache Flink 1.17 > series. > > > > > Apache Flink® is an open-source stream processing framework for distributed, > high-performing, always-available, and accurate data streaming applications. > > > > The release is available for download at: > > https://flink.apache.org/downloads.html > > > > > Please check out the release blog post for an overview of the improvements > for this bugfix release: > > https://flink.apache.org/news/2023/05/25/release-1.17.1.html > > > > The full release notes are available in Jira: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352886 > > > > > We would like to thank all contributors of the Apache Flink community who > made this release possible! > > > > > Feel free to reach out to the release managers (or respond to this thread) > with feedback on the release process. Our goal is to constantly improve the > release process. Feedback on what could be improved or things that didn't go > so well are appreciated. > > > > Regards, > > Release Manager >
Re: [ANNOUNCE] Apache Flink 1.16.2 released
Thank you Weijie and those who helped with testing! On Fri, May 26, 2023 at 1:06 PM weijie guo wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink 1.16.2, which is the second bugfix release for the Apache > Flink 1.16 series. > > > > Apache Flink® is an open-source stream processing framework for > distributed, high-performing, always-available, and accurate data > streaming applications. > > > > The release is available for download at: > > https://flink.apache.org/downloads.html > > > > Please check out the release blog post for an overview of the > improvements for this bugfix release: > > https://flink.apache.org/news/2023/05/25/release-1.16.2.html > > > > The full release notes are available in Jira: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352765 > > > > We would like to thank all contributors of the Apache Flink community > who made this release possible! > > > > Feel free to reach out to the release managers (or respond to this > thread) with feedback on the release process. Our goal is to > constantly improve the release process. Feedback on what could be > improved or things that didn't go so well are appreciated. > > > > Regards, > > Release Manager >
Flink RocketMQ Connector
有没有Flink RocketMQ官方连接器? 需要自己开发吗？Flink生态组件网址（用户上传自己开发的连接器格式什么的）是什么？
[ANNOUNCE] Apache Flink 1.17.1 released
The Apache Flink community is very happy to announce the release of Apache Flink 1.17.1, which is the first bugfix release for the Apache Flink 1.17 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: https://flink.apache.org/news/2023/05/25/release-1.17.1.html The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352886 We would like to thank all contributors of the Apache Flink community who made this release possible! Feel free to reach out to the release managers (or respond to this thread) with feedback on the release process. Our goal is to constantly improve the release process. Feedback on what could be improved or things that didn't go so well are appreciated. Regards, Release Manager
Re: 咨询flink1.17+jdk11环境下使用G1 GC发生大量full gc的问题
hi, yidan 除 jvm 参数外，flink 其他配置完全一样吗？比如 state backend 是否有变化？ 另外， jdk11 是不是用的最新版本，不是的话，觉得也可以尝试一下最新版本。 如果 jdk11 用的最新版本，可以尝试下使用其他 GC 算法是否也有同样问题。比如 -XX:+UseParallelGC -XX:NewRatio=3 -XX:ParallelGCThreads=4 -XX:CICompilerCount=4 -XX:-CompactStrings Best, Yuxin yidan zhao 于2023年5月26日周五 17:39写道： > 最近升级flink版本和jdk版本，flink从1.15.2升级到1.17.0，jdk从8升级到11。然后出现大量full gc。 > 分析后，发现主要是 System.gc() 导致。 进一步定位到是 redisson 库中 netty 部分用到了 DirectMemory > 导致。 直接内存不足，导致频繁调用 System.gc 触发 full gc。 > 我现在问题是，通过测试对比实验发现，jdk8+flink1.17没问题，jdk11+flink1.17就会有该问题。 > 有人知道原因嘛？ > > 其他信息： > > jdk8和jdk11情况下都是G1GC，且vm参数一致，直接内存max限制也一致。但是通过jinfo等查看，确实jdk8场景下的directMemory使用较少。 >
Re: 用flink sql如何实现累计当天交易数量每来一笔交易都实时更新状态并输出到下游?
Hi 可以将天级时间和其他需要聚合的字段组成key，使用聚合算子，默认会每条数据完成计算后实时输出结果 Best, Shammon FY On Fri, May 26, 2023 at 3:44 PM casel.chen wrote: > 用flink sql如何实现累计当天交易数量每来一笔交易都实时更新状态并输出到下游?
咨询flink1.17+jdk11环境下使用G1 GC发生大量full gc的问题
最近升级flink版本和jdk版本，flink从1.15.2升级到1.17.0，jdk从8升级到11。然后出现大量full gc。 分析后，发现主要是 System.gc() 导致。 进一步定位到是 redisson 库中 netty 部分用到了 DirectMemory 导致。 直接内存不足，导致频繁调用 System.gc 触发 full gc。 我现在问题是，通过测试对比实验发现，jdk8+flink1.17没问题，jdk11+flink1.17就会有该问题。 有人知道原因嘛？ 其他信息： jdk8和jdk11情况下都是G1GC，且vm参数一致，直接内存max限制也一致。但是通过jinfo等查看，确实jdk8场景下的directMemory使用较少。
用flink sql如何实现累计当天交易数量每来一笔交易都实时更新状态并输出到下游?
用flink sql如何实现累计当天交易数量每来一笔交易都实时更新状态并输出到下游?
FlinkSQL大窗口小步长的滑动窗口解决方案
Hi, all. 我在使用FlinkSQL的window tvf滑动窗口时遇到一些问题。 滑动步长为5分钟，窗口为24小时，group by user_id的滑动窗口，当任务挂掉了或者从kafka的earliest-offset消费，checkpoint很难成功。 因为从earliest开始消费，数据很快就会堆满缓冲区产生背压，这时这一批数据可能会触发N次窗口计算往下游发，每次触发的操作成本是（用户基数 * 24 * 60 / 5），checkpoint barrier可能会一直卡住。 这时候有什么办法可以破局吗？ best, tanjialiang.
