Re: flinksql 经过优化后,group by字段少了

2024-05-20 Thread Lincoln Lee
%9a%84%e7%a1%ae%e5%ae%9a%e6%80%a7 Best, Lincoln Lee ℡小新的蜡笔不见嘞、 <1515827...@qq.com.invalid> 于2024年5月20日周一 22:07写道: > 当前用的是 flink 1.16 版本,这个issue虽然合并到了 calcite-1.22.0 中,但是在之后一段时间内,又被新的 pr ( > https://github.com/apache/calcite/pull/1735/files)合并了。 > 所以,当前

Re: flink version stable

2024-03-30 Thread Lincoln Lee
of specific connectors on the mailing list. [1] https://lists.apache.org/thread/7qr8jc053y8xpygcwbhlqq4r7c7fj1p3 [2] https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development [3] https://lists.apache.org/thread/sz7f4o1orh96zgjjztcp5gh85l3ks26x Best, Lincoln Lee Fokou Toukam

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Lincoln Lee
Congrats, thanks for the great work! Best, Lincoln Lee Peter Huang 于2024年3月20日周三 22:48写道: > Congratulations > > > Best Regards > Peter Huang > > On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang wrote: > >> >> Congratulations >> >> >> >>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Lincoln Lee
Congrats, thanks for the great work! Best, Lincoln Lee Peter Huang 于2024年3月20日周三 22:48写道: > Congratulations > > > Best Regards > Peter Huang > > On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang wrote: > >> >> Congratulations >> >> >> >>

[ANNOUNCE] Apache Flink 1.19.0 released

2024-03-18 Thread Lincoln Lee
The Apache Flink community is very happy to announce the release of Apache Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

[ANNOUNCE] Apache Flink 1.19.0 released

2024-03-18 Thread Lincoln Lee
The Apache Flink community is very happy to announce the release of Apache Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

Re: [DISCUSS][FLINK-32993] Datagen connector handles length-constrained fields according to the schema definition by default

2023-11-21 Thread Lincoln Lee
) is just max length, we can extending datagen to generate variable length values(maybe a new option to enable it, e.g., 'fields.f0.var-len'='true'). Of course, this is a new feature that is not part of this problem. Best, Lincoln Lee Jane Chan 于2023年11月21日周二 21:07写道: > Hi Yubin, > &g

[SUMMARY] Flink 1.19 Release Sync 11/14/2023

2023-11-15 Thread Lincoln Lee
Hi devs and users, Yesterday was the first release sync of Flink 1.19, I’d like to share the summary: - Sync meeting We switched back to google meet because there's some account limitation for zoom on some region and the google meet is available when creator is not online. The meeting will

[SUMMARY] Flink 1.19 Release Sync 11/14/2023

2023-11-15 Thread Lincoln Lee
Hi devs and users, Yesterday was the first release sync of Flink 1.19, I’d like to share the summary: - Sync meeting We switched back to google meet because there's some account limitation for zoom on some region and the google meet is available when creator is not online. The meeting will

Re: [ANNOUNCE] Apache Flink 1.18.0 released

2023-10-26 Thread Lincoln Lee
Thanks for the great work! Congrats all! Best, Lincoln Lee Jing Ge 于2023年10月27日周五 00:16写道: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.18.0, which is the first release for the Apache Flink 1.18 series. > > Apache Flink® is an open-so

Re: [ANNOUNCE] Apache Flink 1.18.0 released

2023-10-26 Thread Lincoln Lee
Thanks for the great work! Congrats all! Best, Lincoln Lee Jing Ge 于2023年10月27日周五 00:16写道: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.18.0, which is the first release for the Apache Flink 1.18 series. > > Apache Flink® is an open-so

Re: [DISCUSS][FLINK-31788][FLINK-33015] Add back Support emitUpdateWithRetract for TableAggregateFunction

2023-09-07 Thread Lincoln Lee
recommend holding off on removing `TableAggregateFunction` until we have a replacement for it, e.g., user-defined-operator as Jark mentioned or something else. Best, Lincoln Lee Best, Lincoln Lee Jark Wu 于2023年9月7日周四 21:30写道: > +1 to fix it first. > > I also agree to deprecate it if ther

Re: AsyncDataStream: Retries keep executing after timeout

2022-12-21 Thread Lincoln Lee
& testProcessingTimeWithTimeoutFunctionOrderedWithRetry. You can wait for the fix after merge or try the patch if urgently needed. Thanks again for reporting this! Best, Lincoln Lee Yoni Gibbs 于2022年12月15日周四 18:10写道: > Hi Lincoln, > > Thanks very much for the reply! The issue seems

Re: AsyncDataStream: Retries keep executing after timeout

2022-12-14 Thread Lincoln Lee
the problem is. Best, Lincoln Lee Yoni Gibbs 于2022年12月13日周二 23:46写道: > Hi, > > I've got a Kinesis consumer which reacts to each record by doing some > async work using an implementation of RichAsyncFunction. I'm adding a > retry strategy. After x failed attempts I want this to time

Re: flink 1.16 lookup join重试策略问题

2022-12-07 Thread Lincoln Lee
如果结束时还未关联上,就视为当前记录不存在,按 inner join 过滤或 left join 补 null 值 https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/dev/table/sql/queries/hints/#%e5%bc%80%e5%90%af%e7%bc%93%e5%ad%98%e5%af%b9%e9%87%8d%e8%af%95%e7%9a%84%e5%bd%b1%e5%93%8d Best, Lincoln Lee casel.chen 于2022年12月7日周三 23:40

Re: 如何扩展flink sql以实现延迟调用?

2022-12-06 Thread Lincoln Lee
/dev/table/sql/queries/hints/#3-enable-delayed-retry-strategy-for-lookup Best, Lincoln Lee casel.chen 于2022年12月7日周三 11:52写道: > 有人能够解答一下吗? > > > > > > > > > > > > > > > > > > 在 2022-11-26 11:20:34,"casel.chen" 写道: > >双流关联场

Re: flinksql 维表join

2022-10-27 Thread Lincoln Lee
Hi, Flink 的 lookup join 目前不支持对维表进行预处理, 并且需要有对维表原始字段的等值连接条件(因为需要通过确定的字段值去查找) 示例中 t4 字段不做计算应该是 work 的, 比如 udf(t1.telephone_no) = t4.de_mobile Best, Lincoln Lee Fei Han 于2022年10月27日周四 12:12写道: > 大家好!请教几个问题 > 1. FlinkSQL维表join的时候,能不能把维表先用flinksql 当做一个临时表,然后在用临时表做维表join &g

Re: Flink 1.15 Deduplicate之后Interval Join出错

2022-10-17 Thread Lincoln Lee
Hi, 基于事件时间的 first row 去重是可能会产生更新消息的, 所以在 interval join 还不支持更新输入时会报错, 当前一个可行的方式时考虑基于 proctime 进行 first row 去重 (这将不会输出更新消息) Best, Lincoln Lee 余列冰 于2022年10月15日周六 09:46写道: > Hi! > > 我在使用Deduplicate之后进行Interval Join出现问题。我使用的Flink版本是1.15 > > 我希望使用Flink的Interval Join进行双流关联,并且我的第

Re: 来自潘明文的邮件

2022-06-24 Thread Lincoln Lee
Hi, 邮件中直接贴图片无法正常看到,可以发下文本 Best, Lincoln Lee 潘明文 于2022年6月24日周五 16:36写道: > 你好,下面2个SINK 能够并发同时处理吗?还是要窜行,等第一个SINK 好了,才能第二个SINK. > >

Re: lookup join对应task无法正常恢复?

2022-06-24 Thread Lincoln Lee
Hi, 请问使用的 mysql 维表是 flink sql 内置的 jdbc connector 吗? 如果是的话,对应内部的 cache 只是读缓存, 并不会持久化, 任务重启或者到达设定的缓存淘汰条件就失效了 如果是自己开发的维表,建议增加相应的数据加载日志, 以便确认 failover 时的处理是否有异常 Best, Lincoln Lee Xuchao 于2022年6月24日周五 17:15写道: > 您好! > 我在使用flink时遇到一些问题。 > flink-1.14.4 > sqlserver-cdc-2.2.1 &g

Re: Re: 使用join+聚合时,checkpoint异常

2022-06-21 Thread Lincoln Lee
Hi, 确认了下, cdc source 目前全量结束后 task 还是保持的,不会 finish, 这里的 finished task 应该是你提到的 " 使用了lookup join + 外部mysql维表,任务开始时,全量加载了一次维表数据,对应task状态就变成了finished。" Best, Lincoln Lee amber_...@qq.com.INVALID 于2022年6月21日周二 14:35写道: > 非常感谢!你的建议很有用。 > > 我在代码中添加execution.checkpointing.chec

Re: Re: 使用join+聚合时,checkpoint异常

2022-06-20 Thread Lincoln Lee
ing with finished tasks is not enabled Best, Lincoln Lee amber_...@qq.com.INVALID 于2022年6月21日周二 10:27写道: > 感谢! > 未发生背压,但我在日志中发现了一些异常信息,如下: > Failed to trigger checkpoint for job 297c5a840f8fd3a1cbcb63825200e8d4 > because Some tasks of the job have already finished and checkpointing with &

Re: 关于flinksql聚合函数实现的学习疑问

2022-06-01 Thread Lincoln Lee
flink sql 的实现可以参考下 flink-table planner 部分的代码 从 datastream 层面来说, 可以基于 KeyedProcessFunction, 比如 datastream.keyby(...).process(keyedProcessFunction)... 来实现自己定义的逻辑 Best, Lincoln Lee hdxg1101300...@163.com 于2022年6月1日周三 15:49写道: > 您好: >最近再使用flinksql的过程中突然有这样一个疑问:(目前flink1.12.4) >比如

Re: 1.13.5版本sql大小64k限制bug

2022-05-26 Thread Lincoln Lee
是 compile 遇到 beyond 64k 异常了? 可以考虑使用 1.14 及以上版本, 引入了自动的 code splitting: https://issues.apache.org/jira/browse/FLINK-23007 Best, Lincoln Lee godfrey he 于2022年5月26日周四 23:29写道: > 确认一下是sql文本超过64k?具体的异常是什么? > > Yun Tang 于2022年5月26日周四 10:06写道: > > > > Hi > > > > 请使

Re: 回撤流的窗口统计

2021-06-10 Thread Lincoln Lee
你好, 目前的 tumble window 还不支持输入是更新流, 可以用 group by ts 配合 state ttl 来实现。 'table.exec.state.ttl' 是作业级别设置, 当前 sql 还不支持更细粒度的 state ttl 设置。如果有具体的想法,可以在 flink dev 邮件列表发起讨论或提交 jira issue,欢迎贡献社区! lincoln lee casel.chen 于2021年6月5日周六 下午2:24写道: > 上游是binlog cdc消费获取的回撤流,现要使用flink sql统计分析该回撤流上每5分钟的sum

Re: flinksql ttl不生效

2021-06-09 Thread Lincoln Lee
你好, sql 作业可以尝试设置作业参数 "table.exec.state.ttl" 观察下效果 另外开启 "state.backend.incremental" 也可以减少 checkpoint 的大小 参数说明: https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html#checkpoints-and-state-backends lincoln lee chenchencc <1353637...@qq.com