Re: Flink RocketMQ Connector

2023-05-26 文章 Feng Jin
hi casel Flink RocketMQ connector 是由 RockeMQ 社区维护的， 对应的项目地址是: https://github.com/apache/rocketmq-flink 这个版本默认的消息是格式 DELIMIT 格式(默认消息是 String，按分隔符进行分割)， 只能指定消息的列分隔符. best, feng On Fri, May 26, 2023 at 7:44 PM casel.chen wrote: > 有没有Flink RocketMQ官方连接器?

Re: [ANNOUNCE] Apache Flink 1.16.2 released

2023-05-26 文章 Jing Ge
Hi Weijie, Thanks again for your effort. I was wondering if there were any obstacles you had to overcome while releasing 1.16.2 and 1.17.1 that could lead us to any improvement wrt the release process and management? Best regards, Jing On Fri, May 26, 2023 at 4:41 PM Martijn Visser wrote: >

Re: [ANNOUNCE] Apache Flink 1.17.1 released

2023-05-26 文章 Jing Ge
Hi Weijie, That is earlier than I expected! Thank you so much for your effort! Best regards, Jing On Fri, May 26, 2023 at 4:44 PM Martijn Visser wrote: > Same here as with Flink 1.16.2, thank you Weijie and those who helped with > testing! > > On Fri, May 26, 2023 at 1:08 PM weijie guo >

Re: [ANNOUNCE] Apache Flink 1.17.1 released

2023-05-26 文章 Martijn Visser
Same here as with Flink 1.16.2, thank you Weijie and those who helped with testing! On Fri, May 26, 2023 at 1:08 PM weijie guo wrote: > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.17.1, which is the first bugfix release for the Apache Flink 1.17 >

Re: [ANNOUNCE] Apache Flink 1.16.2 released

2023-05-26 文章 Martijn Visser
Thank you Weijie and those who helped with testing! On Fri, May 26, 2023 at 1:06 PM weijie guo wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink 1.16.2, which is the second bugfix release for the Apache > Flink 1.16 series. > > > > Apache Flink® is an

Flink RocketMQ Connector

2023-05-26 文章 casel.chen
有没有Flink RocketMQ官方连接器? 需要自己开发吗？Flink生态组件网址（用户上传自己开发的连接器格式什么的）是什么？

[ANNOUNCE] Apache Flink 1.17.1 released

2023-05-26 文章 weijie guo
The Apache Flink community is very happy to announce the release of Apache Flink 1.17.1, which is the first bugfix release for the Apache Flink 1.17 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data

Re: 咨询flink1.17+jdk11环境下使用G1 GC发生大量full gc的问题

2023-05-26 文章 Yuxin Tan
hi, yidan 除 jvm 参数外，flink 其他配置完全一样吗？比如 state backend 是否有变化？ 另外， jdk11 是不是用的最新版本，不是的话，觉得也可以尝试一下最新版本。 如果 jdk11 用的最新版本，可以尝试下使用其他 GC 算法是否也有同样问题。比如 -XX:+UseParallelGC -XX:NewRatio=3 -XX:ParallelGCThreads=4 -XX:CICompilerCount=4 -XX:-CompactStrings Best, Yuxin yidan zhao 于2023年5月26日周五 17:39写道： >

Re: 用flink sql如何实现累计当天交易数量每来一笔交易都实时更新状态并输出到下游?

2023-05-26 文章 Shammon FY
Hi 可以将天级时间和其他需要聚合的字段组成key，使用聚合算子，默认会每条数据完成计算后实时输出结果 Best, Shammon FY On Fri, May 26, 2023 at 3:44 PM casel.chen wrote: > 用flink sql如何实现累计当天交易数量每来一笔交易都实时更新状态并输出到下游?

咨询flink1.17+jdk11环境下使用G1 GC发生大量full gc的问题

2023-05-26 文章 yidan zhao
最近升级flink版本和jdk版本，flink从1.15.2升级到1.17.0，jdk从8升级到11。然后出现大量full gc。 分析后，发现主要是 System.gc() 导致。 进一步定位到是 redisson 库中 netty 部分用到了 DirectMemory 导致。 直接内存不足，导致频繁调用 System.gc 触发 full gc。 我现在问题是，通过测试对比实验发现，jdk8+flink1.17没问题，jdk11+flink1.17就会有该问题。 有人知道原因嘛？ 其他信息：

用flink sql如何实现累计当天交易数量每来一笔交易都实时更新状态并输出到下游?

2023-05-26 文章 casel.chen
用flink sql如何实现累计当天交易数量每来一笔交易都实时更新状态并输出到下游?

FlinkSQL大窗口小步长的滑动窗口解决方案

2023-05-26 文章 tanjialiang
Hi, all. 我在使用FlinkSQL的window tvf滑动窗口时遇到一些问题。 滑动步长为5分钟，窗口为24小时，group by user_id的滑动窗口，当任务挂掉了或者从kafka的earliest-offset消费，checkpoint很难成功。 因为从earliest开始消费，数据很快就会堆满缓冲区产生背压，这时这一批数据可能会触发N次窗口计算往下游发，每次触发的操作成本是（用户基数 * 24 * 60 / 5），checkpoint barrier可能会一直卡住。 这时候有什么办法可以破局吗？ best, tanjialiang.