Re: Apache Flink - A question about Tables API and SQL interfaces

Hi Mans, There are currently no public APIs for doing so, though if you're willing to deal with some breaking changes there are some experimental config options for late events in the Table API and SQL, seen in the WIndowEmitStrategy class[1]. Best, Austin [1]:

RabbitMQ source does not stop unless message arrives in queue

Hi, I am using Flink 1.12 to read from and write to a RabbitMQ cluster. Flink's RabbitMQ source has some surprising behavior when a stop-with-savepoint request is made. *Expected Behavior:* The stop-with-savepoint request stops the job with a FINISHED state. *Actual Behavior:* The

Long checkpoint duration for Kafka source operators

Hello, I have an application that reads from two Kafka sources, joins them, and produces to a Kafka sink. The application is experiencing long end to end checkpoint durations for the Kafka source operators. I'm hoping I could get some direction in how to debug this further. Here is a UI

Incorrect flink documentation example

Hi, I was trying out examples given here to convert Datastream to Table and one of the example gives out error. My code is as follows : /StreamExecutionEnvironment env =

Re: RabbitMQ source does not stop unless message arrives in queue

Hey Jose, Thanks for bringing this up – it indeed sounds like a bug. There is ongoing work to update the RMQ source to the new interface, which might address some of these issues (or should, if it is not already), tracked in FLINK-20628[1]. Would you be able to create a JIRA issue for this, or

Re: RabbitMQ source does not stop unless message arrives in queue

Hi Jose, hey Austin!! I know we were just recently looking at trying to consume a fixed number of messages from an RMQ source, process them and output them to an RMQ sink. As a naive first attempt at stopping the job when the target number of messaged had been processed, we put a counter state

How to deal with the abnormal extraction time of the calculated column？

Hi My Friends： How to deal with the abnormal extraction time of the calculated column ？ My DDL ： CREATE TABLE mytable ( xx string, t as get_event_time_from_xx(xx), WATERMARK FOR t AS t - INTERVAL '1' SECOND ) WITH (xxx) If an exception occurs inside my

Re: Apache Flink - A question about Tables API and SQL interfaces

Hi Mans, +1 for Austin's reply. I would like to add something about "allow lateness". After introduce Windowing table-valued function in Flink 1.13, User could use two SQL solution to do window aggregate. And 'allow lateness' behavior is different in these two solutions. 1. If

Re: Root Exception can not be shown on Web UI in Flink 1.13.0

Hi, Gary I think it might be a bug. So would you like to open a jira for this. And could you share the exception ,which the TaskManagerLocation is null? It might be very helpful to verify the cause. Best, Guowei On Thu, May 13, 2021 at 10:36 AM Yangze Guo wrote: > Hi, it seems to be related

Flink 1.13.0 reactive mode: Job stop and cannot restore from checkpoint

Hi, We run our application in Flink 1.13.0, Kubernetes standalone application cluster with reactive mode enabled. The application has stopped and cannot restore today, so we try to restore the application from checkpoint. However, the application cannot restart from checkpoint due to the

Re: Flink sql task failure recovery does not work.

Hi I think you could configure some restart strategy[1] likes restart-strategy: fixed-delay [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/task_failure_recovery/#fixed-delay-restart-strategy Best, Guowei On Thu, May 13, 2021 at 12:02 PM 1095193...@qq.com

Re: Need Clarity about Checkpoint for Flink-1.12.2

Hi Sudhansu, I think you do not need to set the config in flink-conf. Best, Guowei On Thu, May 13, 2021 at 1:06 PM sudhansu jena wrote: > Hi Team, > > We have recently enabled Check Pointing in our flink job using > FSStateBackend pointing to S3 bucket. > > Below is the sample code for

Re: Regarding Stateful Functions

Hi Austin, Thanks for your insights. We are currently following a microservice architecture for accomplishing our data processing requirements. We are planning to use Flink as our unified platform for all data processing tasks. Although most of our use cases are a suitable fit for Flink, there

Re: Nested match_recognize query not supported in SQL ?

I was able to get something like this working, but only by introducing a view: CREATE TEMPORARY VIEW mmm AS SELECT id FROM events MATCH_RECOGNIZE (...); SELECT * FROM event WHERE id IN (SELECT id FROM mmm); Regards, David On Tue, May 11, 2021 at 9:22 PM Tejas wrote: > Hi, > I am using flink

Re: Nested match_recognize query not supported in SQL ?

By the way, views that use MATCH_RECOGNIZE don't work in Flink 1.11. [1] [1] https://issues.apache.org/jira/browse/FLINK-20077 On Thu, May 13, 2021 at 11:06 AM David Anderson wrote: > I was able to get something like this working, but only by introducing a > view: > > CREATE TEMPORARY VIEW mmm

Re: How does JobManager terminate dangling task manager

Hi, In fact, not only JobManager(ResoruceManager) will kill TimeOut's TaskManager, but if TaskManager finds that it cannot connect to JobManager(ResourceManager), it will also exit by itself. You can look at the time period during which the HB timeout occurred and what happened in the log. Under

Re: How does JobManager terminate dangling task manager

Hi narasimha, For each TaskManager, there are two kinds of connections to the JobManager process. - One single connection to the ResourceManager, which allows RM to monitor the slots' availability and assign them to Flink jobs. - Connections to each JobMaster that the slots of this TM are

FlinkCEP 尽可能多的匹配的问题

我有一个flinkCEP程序，采用eventTime，监控形如如下的数据 [13/May/2021:20:45:36 +0800] [13/May/2021:20:45:36 +0800] [13/May/2021:20:45:37 +0800] [13/May/2021:20:45:37 +0800] [13/May/2021:20:45:50 +0800] 程序中关键设置如下： 设置了水印延迟2s 跳过测略AfterMatchSkipStrategy.skipPastLastEvent() .times(3) .within(Time.seconds(3));

Mysql cdc 事件时间

你好，想问下mysql cdc做维度表left join的时候，能使用处理时间吗，我测试貌似只能使用事件时间 发自我的iPhone

flink使用hive作为维表，kafka作为数据源，join时候报错

select FROM jdqTableSources AS a JOIN tmmmp FOR SYSTEM_TIME AS OF a.proctime AS b Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Temporal Table Join requires primary key in versioned table, but no primary key can be found. The

请教flink cep如何对无序数据处理

兄弟们，想问下Flink的CEP能够对无序数据流进行处理匹配嘛？ 我这里指的无序事件是：例如有两个事件，事件A和事件B，在一个时间窗口内，只要匹配到了A和B，不论A和B的到来顺序，我都认为是符合我的条件

flink ????ceph

Hi ALL?? ??flink ceph ceph Best, Luke Yan

Re: 回复：flink使用hive作为维表，kafka作为数据源，join时候报错

写on了，刚发现描述的问题没有写全 -- Sent from: http://apache-flink.147419.n8.nabble.com/

Re: 回复：flink使用hive作为维表，kafka作为数据源，join时候报错

sorry , 描述没写全，是有on的 -- Sent from: http://apache-flink.147419.n8.nabble.com/

回复：flink使用hive作为维表，kafka作为数据源，join时候报错

你看异常信息，提示时态表join的时候需要主键，但是你没有定义。而且你join的时候不需要on吗？ | | allanqinjy | | allanqi...@163.com | 签名由网易邮箱大师定制 在2021年05月14日 09:32，hehuiyuan 写道： select FROM jdqTableSources AS a JOIN tmmmp FOR SYSTEM_TIME AS OF a.proctime AS b Caused by:

UpsertStreamTableSink requires that Table has a full primary keys if it is updated.

Hi: 我自定义了一个UpsertStreamTableSink，在接入cdc数据向我的这个Sink写数据的时候，抛出了下面的异常： Exception in thread "main" org.apache.flink.table.api.TableException: UpsertStreamTableSink requires that Table has a full primary keys if it is updated. at

Flink etl 的应用场景

大家好： 方向:ETL 除了延迟上的区别，离线能实现的，flink 实时实现不了的应用场景有哪些或者有缺陷的点？

?????? flink on k8s native ????????

Hi Yang?? flink session on k8s ??examples application ?? flink run-application \ --target kubernetes-application \ -Dkubernetes.namespace=flink-session-cluster \

Re: Flink 1.11版本LeaseRenewer线程不释放

并没有定位到具体原因，只能靠重启作业缓解。。。 zhisheng 于2021年5月13日周四 下午4:20写道： > 你好，这个问题后来定位到问题了吗？ > > 我们生产也有一个作业有这样的问题，Flink 版本是 1.10.0，这个作业是 JM 的线程数很多（快 6k），作业是 flink 读取 > Kafka，会关联 HBase ，开启了 Checkpoint，就这个作业有问题，很奇怪 > > https://tva1.sinaimg.cn/large/008i3skNgy1gqgvhdu674j31je0u0795.jpg > > zilong xiao

Re:回复：flink sql写hbase问题

不是，原因找到了，是函数多次嵌套导致，flink原始类型是not null，不能转换为string，这个报错信息真的是蛋疼，让人迷惑 在 2021-05-13 10:09:49，"allanqinjy" 写道： >光看异常，应该是你插入了空值吧，你插入hbase的时候做个filter过滤吧，比如你的rowkey空了，你往hbase插入应该是不行的。你可以试试。 > > >| | >allanqinjy >| >| >allanqi...@163.com >| >签名由网易邮箱大师定制 > > >在2021年05月12日 19:23，酷酷的浑蛋 写道：

Re: Flink 1.11版本LeaseRenewer线程不释放

你好，这个问题后来定位到问题了吗？ 我们生产也有一个作业有这样的问题，Flink 版本是 1.10.0，这个作业是 JM 的线程数很多（快 6k），作业是 flink 读取 Kafka，会关联 HBase ，开启了 Checkpoint，就这个作业有问题，很奇怪 https://tva1.sinaimg.cn/large/008i3skNgy1gqgvhdu674j31je0u0795.jpg zilong xiao 于2020年12月8日周二 下午6:21写道： > 作业数据流是 kafka -> flink -> >

?????? Flink 1.11????LeaseRenewer??????????

Flink java ?? ---- ??: "zhisheng"https://tva1.sinaimg.cn/large/008i3skNgy1gqgvhdu674j31je0u0795.jpg zilong xiao

flink sql怎样将change log stream转换成append log stream?

flink sql怎样将change log stream转换成append log stream? 通过cdc接入了change log stream，后面就不能使用窗口聚合了，像Tumble和Hop, Session窗口。只能通过state ttl + group by timestamp这种方式聚合。 问一下有没有办法将change log stream转换成append log stream，从而可以使用上述窗口聚合了呢？谢谢！

类型转换问题 String 类型如何转 decimal类型

source 端接收到的数据类型为 String， sink 端 MySQL 数据库字段类型定义为 decimal(12, 2) , 在编写 insert 语句时该如何把 String 类型转换为 decimal, 尝试了 flink build-in 的 cast 并不行，请问各位有什么好的方法？ -- Sent from: http://apache-flink.147419.n8.nabble.com/