Re: Will Flink loss some old Keyed State when changing the parallelism

Hi Yang, Flink keeps the max key groups the same no matter how parallelism changes, and use this to avoid state data lost [1] [1] https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html Best Yun Tang On 2021/11/26 10:07:29 Nicolaus Weidner wrote: > Hi, > > to rescale, you

Re: Windows and data loss.

Or as an example we have a 5 minutes window and lateness of 5 minutes. We have the following events in the logs 10:00:01 PM > Already pushed to Kafka 10:00:30 PM > Already pushed to Kafka 10:01:00 PM > Already pushed to Kafka 10:03:45 PM > Already pushed to Kafka 10:04:00 PM >

Re: Windows and data loss.

Ok, So processing time we get 100% accuracy because we don't care when the event comes, we just count and move along. As for event time processing, what I meant to say is if for example if the log shipper is late at pushing events into Kafka, Flink will not notice this, the watermarks will keep

Re: Views support in PostgresCatalog

Hi Flavio, that error message refers to Flink tables and Flink views, not Postgres views: within the catalog, no Flink table or Flink view of that name exists. Note that even Postgres views would have to be represented as Flink tables in Flink. In order to change the message to (Postgres) "views

Re: Checkpoints aborted - Job is not in state RUNNING but FINISHED

Hi Yun, thanks for the quick reply! Great to hear that a fix has been put in place as of Flink 1.14. Since we are currently using Beam on top of Flink, we are currently limited to the Flink 1.13 runner, so I would expect the Fix not to be available to us yet. But to clarify the underlying

RE: Windows and data loss.

Actually not, because processing-time does not matter at all. Event-time timers are always compared to watermark-time progress. If system happens to be compromised for (say) 4 hours, also watermarks won’t progress, hence the windows get not evicted and wait for watermarks to pick up from when

Views support in PostgresCatalog

Hi to all, I was trying to use a view of my Postgres database through the PostgresCatalog but at the moment it seems that the current implementation ignores views. Probably this is caused by the fact that there's no way to avoid INSERT statements in Flink. However, the thrown error is somehow

Re: Windows and data loss.

But if we use event time, if a failure happens potentially those events can't be delivered in their windo they will be dropped if they come after the lateness and watermark settings no? On Fri, 26 Nov 2021 at 02:35, Schwalbe Matthias wrote: > Hi John, > > > > Going with processing time is

Re: Scala class ExecutionContext not found on my Java + Flink project

Hi Felipe, glad this helped and you found the problematic dependencies! Best, Nico On Fri, Nov 26, 2021 at 12:32 PM Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote: > hey Nico, you nailed it :tada: > > after doing "./mvnw dependency:tree | less" I saw scala 2.13 inside >

Re: Scala class ExecutionContext not found on my Java + Flink project

hey Nico, you nailed it :tada: after doing "./mvnw dependency:tree | less" I saw scala 2.13 inside spring-kafka-test. I excluded it and the error was gone. org.springframework.kafka spring-kafka-test com.fasterxml.jackson.module

Re:退订

退订发送到 user-zh-unsubscr...@flink.apache.org | | 稚远 | | drewfrank...@163.com 行远自逊，笃行不怠 | 签名由网易邮箱大师定制 On 11/26/2021 17:44，蔡玉胜 wrote： 退订

Re: Scala class ExecutionContext not found on my Java + Flink project

Hi Felipe, can you check for Scala 2.13 dependencies on your classpath (parasitic was only introduced in Scala 2.13)? Not sure if that comes from Spring Boot 2.6 or something else. If you use Flink 1.13 or lower, it will just use whatever version is on the classpath, so you have to make sure

Re: Scala class ExecutionContext not found on my Java + Flink project

Hi Nico, yes, I did. I also run from the terminal "./mvnw clean verify -Pintegration-test" and I get the same error I also invalidated cache, deleted the ~/.m2/...flink dir. ~/.idea Imported to IntelliJ from scratch I am sure that the class is in "flink-runtime_2.12" dependency. When I start my

Re: Scala class ExecutionContext not found on my Java + Flink project

Hi Felipe, just a quick question to make sure: did you do a full rebuild of your project after changing the Spring boot version? Best, Nico On Thu, Nov 25, 2021 at 8:01 PM Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote: > Hi community, > > I am using Flink 1.11 + Java 8 and I was

Re: Will Flink loss some old Keyed State when changing the parallelism

Hi, to rescale, you should take a savepoint, stop the job, then restart from the savepoint with your new desired parallelism. This way, no data will be lost. Best, Nico On Thu, Nov 25, 2021 at 10:53 AM 杨浩 wrote: > Will Flink loss some old Keyed State when changing the parallelism, like 2 > ->

退订

退订

回复：Flink kafka自定义metrics在influxdb上解析失败

你好，我没有自己开发连接器，我用的是kafka connector，influxdb只是作为一个metrics信息存储端，你是需要一个influxdb的连接器？我只是用到了metrics统计体系而已，只是自定义了Counter对象，和连接器没有什么关系 发自 网易邮箱大师 回复的原邮件 | 发件人 | 信华哺 | | 日期 | 2021年11月26日 17:22 | | 收件人 | user-zh@flink.apache.org | | 抄送至 | | | 主题 | 回复：Flink kafka自定义metrics在influxdb上解析失败 |

退订

退订 | | alwaysonlinedj | | alwaysonlin...@163.com | 签名由网易邮箱大师定制

Stateful function endpoint self-signed certificate problem

Hi, We have been running a simple stateful functions (version 3.0.0) job, which simply forwards the incoming messages via Kafka source to an HTTPS endpoint on AWS. Our HTTP endpoint is behind a Load Balancer on AWS and this Load Balancer is listening on 443 for incoming HTTPS traffic. Certificate

回复：Flink kafka自定义metrics在influxdb上解析失败

你好： 我想问一下，你用的flink sql连接器是自己开发的么？ 我在网上只能找到一个datastream的influxdb连接器 在2021年7月23日 10:11，Jimmy Zhang 写道： 大家好，Flink版本1.13.1。 我在FlinkKafkaProducer的invork方法中，创建了一个counter，它采用sinkTableName+_receive_count作为最终的metric 表名，并且采用influxDB作为上报终端，其中，sinkTableName是从catalog信息里拿到的，我从构造中一层层传递进去。

Re: mutual authentication with ssl

Hi Raul, On all systems keystore is needed normally on the server side and truststore on client side. As a result it's highly advised to use different config files in these places. It's easy to see why it would be a security leak when keystore would be available in client side (client can fake a

Re: [DISCUSS] Drop Zookeeper 3.4

According to this SO question [1], it seems that Zk 3.5 clients cannot talk to 3.4 servers. I also tried it out with a local deployment and Flink was not able to start. Newer Zk versions can talk to older Zk servers if no new APIs are used [2]. [1] https://stackoverflow.com/a/61630617/4815083

Re: Is there a way to print key and state metadata/types for a job?

Hi Dan, AFAIK, there is no built-in way to solve your problem. You could check whether state processor API could help you. Or you could add log in your program. Best, Jing Zhang Dan Hill 于2021年11月26日周五