Flink kubernets operator delete HA metadata after resuming from suspend

2023-09-11 Thread Evgeniy Lyutikov
Hi all! After updating the operator to version 1.6.0, suspended and resuming flink jobs stopped working. When job resumes, the high availability metadata is removed. Suspend job: 2023-09-11 06:01:41,548 o.a.f.k.o.l.AuditUtils [INFO ][rec-job/rec-job] >>> Event | Info| SPECCHANGED

Re: Incompatible KafkaProducer version

2023-09-11 Thread Hang Ruan
Hi, Krzysztof. I find that this part has been changed in PR[1] when updating the kafka client version to 3.4.0. This fix is not released yet. Maybe you can package and check it by yourself. Best, Hang [1] https://github.com/apache/flink-connector-kafka/pull/11 Krzysztof Jankiewicz

e2e tests with flink

2023-09-11 Thread Oscar Perez via user
Hi, I have a flink job which I want to test e2e. In the test I start flink minicluster and this reads from kafka topics in testcontainers. I m facing a problem that for some topics I have starting offset as latest and I want to publish these messages just after the job has been completely

Feedback on Testing Guidelines for Data Stream Processing Applications

2023-09-11 Thread Alexandre Strapacao Guedes Vianna
Greetings, I hope this message finds you well. As part of my PhD research, I've developed guidelines tailored to assist professionals in planning testing of Data Stream Processing applications. If you've worked directly with stream processing or have experience with similar systems, like

Re: Flink kubernets operator delete HA metadata after resuming from suspend

2023-09-11 Thread Gyula Fóra
Hi! I could not reproduce your issue, last-state suspend/restore seems to work as before. However these 2 logs seem very suspicious: 2023-09-11 06:02:07,481 o.a.f.k.o.o.d.ApplicationObserver [INFO ][rec-job/rec-job] Observing JobManager deployment. Previous status: MISSING 2023-09-11

Re: Flink kubernets operator delete HA metadata after resuming from suspend

2023-09-11 Thread Evgeniy Lyutikov
Hi! No, no one could restart jobmanager, I monitored the pods in real time, they all deleted when suspended as expected. От: Gyula Fóra Отправлено: 11 сентября 2023 г. 20:34:52 Кому: Evgeniy Lyutikov Копия: user@flink.apache.org Тема: Re: Flink kubernets

Re: Flink kubernets operator delete HA metadata after resuming from suspend

2023-09-11 Thread Evgeniy Lyutikov
Why we need to use latest CRD version with older operator version? От: Gyula Fóra Отправлено: 12 сентября 2023 г. 0:36:26 Кому: Evgeniy Lyutikov Копия: user@flink.apache.org Тема: Re: Flink kubernets operator delete HA metadata after resuming from suspend Do

Re: observedGeneration field in FlinkDeployment

2023-09-11 Thread Tony Chen
I think it should be fine for now. I can compare the following 2 fields to check if the spec has changed: - metadata.generation: 4 - status.reconciliationStatus.lastStableSpec: '{"resource_metadata": {"metadata": {"generation": 2}' On Sat, Sep 9, 2023 at 2:48 AM Gyula Fóra wrote: >

Re: Flink kubernets operator delete HA metadata after resuming from suspend

2023-09-11 Thread Gyula Fóra
Do not change the CRD but you can roll back the operator itself I believe Gyula On Mon, 11 Sep 2023 at 18:52, Evgeniy Lyutikov wrote: > Is it safe to rollback the operator version with replace to old CRDs? > -- > *От:* Evgeniy Lyutikov > *Отправлено:* 11 сентября

Re: Flink kubernets operator delete HA metadata after resuming from suspend

2023-09-11 Thread Evgeniy Lyutikov
Is it safe to rollback the operator version with replace to old CRDs? От: Evgeniy Lyutikov Отправлено: 11 сентября 2023 г. 23:50:26 Кому: Gyula Fóra Копия: user@flink.apache.org Тема: Re: Flink kubernets operator delete HA metadata after resuming from suspend

Re: Flink kubernets operator delete HA metadata after resuming from suspend

2023-09-11 Thread Gyula Fóra
You don’t need it but you can really mess up clusters by rolling back CRD changes… On Mon, 11 Sep 2023 at 19:42, Evgeniy Lyutikov wrote: > Why we need to use latest CRD version with older operator version? > -- > *От:* Gyula Fóra > *Отправлено:* 12 сентября 2023 г.

Reading parquet files using Flink

2023-09-11 Thread Hou, Lijuan via user
Hi team, Is there any defined way to read Parquet files for flink 1.17.1? I did some search, and found this

Re: e2e tests with flink

2023-09-11 Thread Feng Jin
Hi Oscar You can refer to the unit tests in flink-connector-kafka. https://github.com/apache/flink-connector-kafka/blob/d6525c1481fc2d2821f361d2d5ce48f97221dd74/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaITCase.java#L152 Best, Feng On Mon, Sep 11,

Re: Incompatible KafkaProducer version

2023-09-11 Thread Krzysztof Jankiewicz
Hi, Hang. There have been a lot of changes made to the TransactionsManager in the Kafka client in 2022. (e.g. https://github.com/apache/kafka/commit/3ea7b418fb3d7e9fc74c27751c1b02b04877f197 ). Version 3.2.3 was the last one when the TransactionsManager class contained attributes (e.g.,

Re: 退订

2023-09-11 Thread Hang Ruan
Hi, 请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Please send email to user-zh-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user-zh@flink.apache.org , and you can refer [1][2] for more details. Best,

Re: 退订

2023-09-11 Thread Hang Ruan
Hi, 请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Please send email to user-zh-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user-zh@flink.apache.org , and you can refer [1][2] for more details. Best,

回复：flink-metrics如何获取applicationid

2023-09-11 Thread 吴先生
请问好使吗，怎么使用的 | | 吴先生 | | 15951914...@163.com | 回复的原邮件 | 发件人 | allanqinjy | | 发送日期 | 2023年8月30日 20:02 | | 收件人 | user-zh@flink.apache.org | | 主题 | 回复：flink-metrics如何获取applicationid | 多谢了，明天改一下代码试试 回复的原邮件 | 发件人 | Feng Jin | | 发送日期 | 2023年08月30日 19:42 | | 收件人 | user-zh | | 主题 |