Re: Question about time-based operators with RocksDB backend

2024-03-05 Thread Jinzhong Li
Hi Gabriele, The keyed state APIs (ValueState、ListState、etc) are supported by all types of state backend (hashmap、rocksdb、etc.). And the built-in window operators are implemented with these state APIs internally. So you can use these built-in operators/functions with the RocksDB state backend

Re: I used the yarn per job mode to submit tasks, which will end in 4 minutes

2024-03-05 Thread Junrui Lee
Hello, The issue you're encountering is related to a new heartbeat mechanism between the client and job in Flink-1.17. If the job does not receive any heartbeats from the client within a specific timeout, it will cancel itself to avoid hanging indefinitely. To address this, you have two options:

I used the yarn per job mode to submit tasks, which will end in 4 minutes

2024-03-05 Thread 程意
In versions 1.17.1 and 1.18.1, I used the yarn per job mode to submit tasks, which will end in 4 minutes. But I tried it on Flink 1.13.1, 1.15.2, and 1.16.3, all of which were normal. command line at 1.17.1 version: ``` ./bin/flink run -t yarn-per-job -ys 1 -yjm 1G -ytm 3G -yqu default -p 1

Re: Handling late events with Table API / SQL

2024-03-05 Thread Feng Jin
You can use the CURRENT_WATERMARK(rowtime) function for some filtering, please refer to [1] for details. https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/systemfunctions/ Best, Feng On Wed, Mar 6, 2024 at 1:56 AM Sunny S wrote: > Hi, > > I am using Flink SQL to

Re:Handling late events with Table API / SQL

2024-03-05 Thread Xuyang
Hi, for out of order events, watermark can handle them. However, for late events, Flink Table & SQL are not supported to output them to a side channel like DataStream API. There have been some JIRAs related this.[1][2] If you really need this feature, you may consider initiating related

Handling late events with Table API / SQL

2024-03-05 Thread Sunny S
Hi, I am using Flink SQL to create a table something like this : CREATE TABLE some-table (   ...,   ...,   ...,   ...,   event_timestamp as TO_TIMESTAMP_LTZ(event_time*1000, 3),   WATERMARK FOR event_timestamp AS event_timestamp - INTERVAL '30' SECOND ) WITH (   

Re: Temporal join on rolling aggregate

2024-03-05 Thread Gyula Fóra
Hi Everyone! I have discussed this with Sébastien Chevalley, he is going to prepare and drive the FLIP while I will assist him along the way. Thanks Gyula On Tue, Mar 5, 2024 at 9:57 AM wrote: > I do agree with Ron Liu. > This would definitely need a FLIP as it would impact SQL and extend it

Re: Temporal join on rolling aggregate

2024-03-05 Thread lorenzo.affetti.ververica.com via user
I do agree with Ron Liu. This would definitely need a FLIP as it would impact SQL and extend it with the equivalent of TimestampAssigners in the Java API. Is there any existing JIRA here, or is anybody willing to drive a FLIP? On Feb 26, 2024 at 02:36 +0100, Ron liu , wrote: > +1, > But I think