Re: Flink1.13 standalone基于zk进行HA，经常出现重启后无限leader选举的情况

这个问题今天再另外一个场景遇到了，经过仔细对比日志，观察ZK节点信息，目前定位到问题。 因为这个问题历史遇到太多次，不好说是否都是这个原因，但起码有一大部分是这个原因。 这个主要是因为部分场景可能人工重启了JM进程，不是基于 bin/start-cluster.sh 脚本启，而是基于 ./bin/jobmanager.sh start 这样启动的。 假设有5个机器，JM1、JM2、JM3，TM1-TM5，前3跑JM，5台都跑TM。 JM123之间选举，但是JM123上报到zk的信息存在localhost，导致集群无法正常启动。

Flink SQL 1.12 How to implement query Hbase table on secondary index

Hi, now my data store hbase and I want use flink to implement kafka table temproal join hbase table , but condtion is not rowkey , I realize hbase secondary index, how can I implement this function what can use flink sql first query secondary index(such as es) and then use rowkey query

Potential Bug with Date Serialization for Table Stream

Per the docs , I'm hoping to confirm whether or not an error we are seeing is a bug with Flink. We have a job that uses a Kafka source to read Avro records. The kafka source is converted into a StreamTableSource. We are using the new Blink

Stateful function with GCP Pub/Sub ingress/egress

The statefun docs have some nice examples of how to use Kafka and Kinesis for ingress/egress in conjunction with a function. Is there some documentation or example code I could reference to do the same with a GCP Pub/Sub topic? Thanks. Dave

RE: Watermarks event time vs processing time

Hi Hanspeter, Event time mode should work just the same … for your example below you your need only one single arbitrary event per kafka partition that has a timestamp > 1646992800560 + sessionWindowGap + outOfOrderness in order for the session window to be triggered. I’m not sure why

Re: Watermarks event time vs processing time

Hi Matthias and others Thanks for the answer. I will remove the Idleness. However I am not doing max/min etc. Unfortunately most examples are about aggregations. The inputs are like this {"handling_time":1646992800260,"transaction_id":"017f6af1548e-119dfb",}

RE: Watermarks event time vs processing time

Hi Hanspeter, Let me relate some hints that might help you getting concepts clearer. From your description I make following assumptions where your are not specific enough (please confirm or correct in your answer): 1. You store incoming events in state per transaction_id to be

Re: Flink1.13 standalone基于zk进行HA，经常出现重启后无限leader选举的情况

Hi, yidan 建议排查下 Leader 频繁选举时 JobManager 是否存在频繁 GC 的情况。 如果不是还可以通过 jobmanager.log 查看是否存在其他异常。 > 2022年3月11日 下午4:34，yidan zhao 写道： > > 我指的选举主要是flink的选举，就是访问web ui后，显示正处于选举过程中。然后我一直等，一直处于选举过程中。 > > Biao Geng 于2022年3月11日周五 13:00写道： > >> Hi yidian， >> >>

Adding a custom Kafka deserializer to Statefun Job

Hi, I am writing a StateFun application using remote functions, where I want to include a custom Kafka deserializer that adds the timestamp of the Kafka ingress messages to the messages sent between my remote functions. I can't seem to find a solution to this using remote functions, as the

Re: Rescaling REST API not working

HI, Aarsh Shah By consulting the information, I found that this API has been temporarily deprecated, and there is no API to deal with rescaling. Maybe you can try to use the reactive scheduler. > 2022年3月16日 下午2:15，Aarsh Shah 写道： > > Hello, > I tried to call the rescaling api with patch to

Re: RocksDB metrics for effective memory consumption

Hi Donatien, The managed memory in Flink actually locates off-heap and stays as native memory, e.g memory consumed by RocksDB, python. In other words, JVM cannot know how much the memory that third-party software used. Thus, Flink will just set the upper limit for the managed memory and let

Watermarks event time vs processing time

Hi, I read from a Kafka topic events that are in JSON format These event contain a handling time (aka event time) in epoch milliseconds, a transaction_id and a large nested JSON structure. I need to group the events by transaction_id, order them by handling time and calculate the differences in

Re: [ANNOUNCE] Apache Flink 1.1.4.4 released

Hi Konstantin, I have installed the wheel packages of 1.13.6 and 1.14.4 respectively, and tested them through some examples. Thanks a lot for your work. Best, Xingbo Konstantin Knauf 于2022年3月16日周三 15:29写道： > Hi Xingbo, > > you are totally right. Thank you for noticing. This also affected

Re: [ANNOUNCE] Apache Flink 1.1.4.4 released

Hi Konstantin, I have installed the wheel packages of 1.13.6 and 1.14.4 respectively, and tested them through some examples. Thanks a lot for your work. Best, Xingbo Konstantin Knauf 于2022年3月16日周三 15:29写道： > Hi Xingbo, > > you are totally right. Thank you for noticing. This also affected

Re: Setting S3 as State Backend in SQL Client

Hi dz902, I actually can't find that sentence on the website you've linked to. It does state "The following sections list all available options that can be used to adjust Flink Table & SQL API programs.". So that list are the available options that you can use. The options that you're trying are

Re: Setting S3 as State Backend in SQL Client

Hi, Per SQL Lite doc ( https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/sqlclient/) I see this: > SQL Client Configuration > You can configure the SQL client by setting the options below, or any valid Flink configuration entry: So any valid Flink configuration should

Re: [ANNOUNCE] Apache Flink 1.1.4.4 released

Hi Xingbo, you are totally right. Thank you for noticing. This also affected Flink 1.13.6, the other release I was recently managing. I simply skipped a step in the release guide. It should be fixed now. Could you double-check? Cheers, Konstantin On Wed, Mar 16, 2022 at 4:07 AM Xingbo Huang

Re: [ANNOUNCE] Apache Flink 1.1.4.4 released

Hi Xingbo, you are totally right. Thank you for noticing. This also affected Flink 1.13.6, the other release I was recently managing. I simply skipped a step in the release guide. It should be fixed now. Could you double-check? Cheers, Konstantin On Wed, Mar 16, 2022 at 4:07 AM Xingbo Huang

Re: Setting S3 as State Backend in SQL Client

Hi, If I remember correctly, set operations supports only a limited set of configurations. Most of them are table options that are listed on table configuration [1] plus some pipeline options. State backend options are not likely one of them. [1]

how to convert Table to DataStream / DataSet by TableEnvironment on Batch mode?

Hi, community! When dealing with retractable stream, i meet a problem about converting Table to DataSet / DataStream on batch mode in Flink-1.13.5. scenario and process: - 1. Database CDC to Kafka - 2. Sync data into Hive with HoodieTableFormat(Apache Hudi) - 3. Incremental processing

Rescaling REST API not working

Hello, I tried to call the rescaling api with patch to automatically rescale, but it shows 503, is it deprecated? because it is present in the docs too, and if it is deprecated, is there any API through which I can rescale directly? Because the mode which we are using is not the reactive mode, we