[jira] [Created] (FLINK-33156) Remove flakiness from tests in OperatorStateBackendTest.java

2023-09-25 Thread Asha Boyapati (Jira)
Asha Boyapati created FLINK-33156: - Summary: Remove flakiness from tests in OperatorStateBackendTest.java Key: FLINK-33156 URL: https://issues.apache.org/jira/browse/FLINK-33156 Project: Flink

回复: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

2023-09-25 Thread Chen Zhanghao
Hi Jing, I've updated Compatibility, Deprecation, and Migration Plan section to list all the potential compatibility issues for users who want to upgrade an existing job to use this feature: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150. Best, Zhanghao Chen

Re: [ANNOUNCE] Apache Flink Stateful Functions Release 3.3.0 released

2023-09-25 Thread Martijn Visser
Hi Frans, Good remark, I still need to provide the images to those who have access to the Dockerhub, but I haven't been able to done that yet. Hopefully I can do that at the end of the week. Best regards, Martijn On Mon, Sep 25, 2023 at 2:04 PM wrote: > > Hi Martijn. > > Thanks for this.

Re: [VOTE] FLIP-362: Support minimum resource limitation

2023-09-25 Thread Weihua Hu
+1 (binding) Best, Weihua On Tue, Sep 26, 2023 at 10:00 AM Shammon FY wrote: > +1(binding), thanks for the proposal. > > Best, > Shammon FY > > On Mon, Sep 25, 2023 at 11:20 PM Jing Ge > wrote: > > > +1(binding) Thanks! > > > > Best regards, > > Jing > > > > On Mon, Sep 25, 2023 at 3:48 AM

Re: Re: [DISCUSS] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

2023-09-25 Thread Dong Lin
Hi all, Thanks for the review! Becket and I discussed this FLIP offline and we agreed on several things that need to be improved with this FLIP. I will summarize our discussion with the problems and TODOs. We will update the FLIP and let you know once the FLIP is ready for review again. 1)

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

2023-09-25 Thread Leonard Xu
+1(binding) Best, Leonard > On Sep 26, 2023, at 9:59 AM, Feng Jin wrote: > > +1(no-binding) > > > thanks for driving this proposal > > > Best, > Feng > > On Mon, Sep 25, 2023 at 11:19 PM Jing Ge wrote: > >> +1(binding) Thanks! >> >> Best Regards, >> Jing >> >> On Sun, Sep 24, 2023 at

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

2023-09-25 Thread Feng Jin
+1(no-binding) thanks for driving this proposal Best, Feng On Mon, Sep 25, 2023 at 11:19 PM Jing Ge wrote: > +1(binding) Thanks! > > Best Regards, > Jing > > On Sun, Sep 24, 2023 at 10:30 PM Shammon FY wrote: > > > Hi devs, > > > > Thanks for all the feedback on FLIP-314: Support

Re: [VOTE] FLIP-362: Support minimum resource limitation

2023-09-25 Thread Shammon FY
+1(binding), thanks for the proposal. Best, Shammon FY On Mon, Sep 25, 2023 at 11:20 PM Jing Ge wrote: > +1(binding) Thanks! > > Best regards, > Jing > > On Mon, Sep 25, 2023 at 3:48 AM Chen Zhanghao > wrote: > > > Thanks for driving this. +1 (non-binding) > > > > Best, > > Zhanghao Chen > >

Re: [DISCUSS] FLIP-328: Allow source operators to determine isProcessingBacklog based on watermark lag

2023-09-25 Thread Dong Lin
Hi Piotr, Thanks for the comments. Let me try to explain it below. Overall, the two competing options differ in how an invocation of `#setIsProcessingBacklog(false)` affects the backlog status for the given source (corresponding to the SplitEnumeratorContext instance on which this method is

RE: [ANNOUNCE] Apache Flink Stateful Functions Release 3.3.0 released

2023-09-25 Thread frans.king
Hi Martijn. Thanks for this. Should there also be docker images available? https://hub.docker.com/r/apache/flink-statefun/tags goes up to 3.2.0. Frans -Original Message- From: Martijn Visser Sent: Tuesday, September 19, 2023 11:37 AM To: dev@flink.apache.org; user ; user-zh ;

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-25 Thread Jing Ge
Hi Ryan, Thanks for reaching out. It is fine to include it but we need to wait until the CI passes. I am not sure how long it will take, since there seems to be some infra issues. Best regards, Jing On Mon, Sep 25, 2023 at 11:34 AM Ryan Skraba wrote: > Hello! There's a security fix that

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-25 Thread Ryan Skraba
Hello! There's a security fix that probably should be applied to 1.18 in the next RC1 : https://github.com/apache/flink/pull/23461 (bump to snappy-java). Do you think this would be possible to include? All my best, Ryan [1]: https://issues.apache.org/jira/browse/FLINK-33149 "Bump snappy-java

Re: [VOTE] FLIP-327: Support switching from batch to stream mode to improve throughput when processing backlog data

2023-09-25 Thread Ahmed Hamdy
+1(non binding) Best regards Ahmed Hamdy On Mon, 25 Sep 2023, 19:57 Venkatakrishnan Sowrirajan, wrote: > +1 (non-binding) > > On Sun, Sep 24, 2023, 6:49 PM Xintong Song wrote: > > > +1 (binding) > > > > Best, > > > > Xintong > > > > > > > > On Sat, Sep 23, 2023 at 10:16 PM Yuepeng Pan > >

Re: [VOTE] FLIP-327: Support switching from batch to stream mode to improve throughput when processing backlog data

2023-09-25 Thread Venkatakrishnan Sowrirajan
+1 (non-binding) On Sun, Sep 24, 2023, 6:49 PM Xintong Song wrote: > +1 (binding) > > Best, > > Xintong > > > > On Sat, Sep 23, 2023 at 10:16 PM Yuepeng Pan > wrote: > > > +1(non-binding), thank you for driving this proposal. > > > > Best, > > Yuepeng Pan. > > At 2023-09-22 14:07:45, "Dong

Re: [VOTE] FLIP-362: Support minimum resource limitation

2023-09-25 Thread Jing Ge
+1(binding) Thanks! Best regards, Jing On Mon, Sep 25, 2023 at 3:48 AM Chen Zhanghao wrote: > Thanks for driving this. +1 (non-binding) > > Best, > Zhanghao Chen > > 发件人: xiangyu feng > 发送时间: 2023年9月25日 17:38 > 收件人: dev@flink.apache.org > 主题: [VOTE] FLIP-362:

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

2023-09-25 Thread Jing Ge
+1(binding) Thanks! Best Regards, Jing On Sun, Sep 24, 2023 at 10:30 PM Shammon FY wrote: > Hi devs, > > Thanks for all the feedback on FLIP-314: Support Customized Job Lineage > Listener [1] in thread [2]. > > I would like to start a vote for it. The vote will be opened for at least > 72

Re: [Discuss] FLIP-362: Support minimum resource limitation

2023-09-25 Thread Jing Ge
Hi Yangze, Thanks for the clarification. The example of two batch jobs team up with one streaming job is interesting. Best regards, Jing On Wed, Sep 20, 2023 at 7:19 PM Yangze Guo wrote: > Thanks for the comments, Jing. > > > Will the minimum resource configuration also take effect for

Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

2023-09-25 Thread Jing Ge
Hi Zhanghao, Thanks for driving the FLIP. This is a nice feature users are looking for. >From users' perspective, would you like to add explicit description about any potential(or none) compatibility issues if users want to use this new feature and start existing jobs with savepoints or

Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

2023-09-25 Thread Lincoln Lee
Hi Zhanghao, Thanks for your explanation! - For question 2, I can accept that the user must provide a pk when modifying the parallelism of the cdc source. - For question 1&3 and additional concerns, if there is no more general or discussable solution at the time, I agree to address part of the

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-25 Thread Jing Ge
Thanks Zakelly for the update! Appreciate it! @Piotr Nowojski If you do not have any other concerns, I will move forward to create 1.18 rc1 and start voting. WDYT? Best regards, Jing On Mon, Sep 25, 2023 at 2:20 AM Zakelly Lan wrote: > Hi Jing and everyone, > > I have conducted three rounds

[jira] [Created] (FLINK-33155) Flink ResourceManager continuously fails to start TM container on YARN when Kerberos enabled

2023-09-25 Thread Yang Wang (Jira)
Yang Wang created FLINK-33155: - Summary: Flink ResourceManager continuously fails to start TM container on YARN when Kerberos enabled Key: FLINK-33155 URL: https://issues.apache.org/jira/browse/FLINK-33155

[jira] [Created] (FLINK-33154) flink on k8s,An error occurred during consuming rocketmq

2023-09-25 Thread Monody (Jira)
Monody created FLINK-33154: -- Summary: flink on k8s,An error occurred during consuming rocketmq Key: FLINK-33154 URL: https://issues.apache.org/jira/browse/FLINK-33154 Project: Flink Issue Type:

回复: [VOTE] FLIP-362: Support minimum resource limitation

2023-09-25 Thread Chen Zhanghao
Thanks for driving this. +1 (non-binding) Best, Zhanghao Chen 发件人: xiangyu feng 发送时间: 2023年9月25日 17:38 收件人: dev@flink.apache.org 主题: [VOTE] FLIP-362: Support minimum resource limitation Hi all, I would like to start the vote for FLIP-362: Support minimum

Re: [VOTE] FLIP-362: Support minimum resource limitation

2023-09-25 Thread Ahmed Hamdy
Thanks for the proposal +1(non-binding) Best regards Ahmed On Mon, 25 Sep 2023, 13:35 Yangze Guo, wrote: > +1 (binding) > > Best, > Yangze Guo > > On Mon, Sep 25, 2023 at 5:39 PM xiangyu feng wrote: > > > > Hi all, > > > > I would like to start the vote for FLIP-362: Support minimum resource

Re: [VOTE] FLIP-362: Support minimum resource limitation

2023-09-25 Thread Yangze Guo
+1 (binding) Best, Yangze Guo On Mon, Sep 25, 2023 at 5:39 PM xiangyu feng wrote: > > Hi all, > > I would like to start the vote for FLIP-362: Support minimum resource > limitation[1]. > This FLIP was discussed in this thread [2]. > > The vote will be open for at least 72 hours unless there is

[VOTE] FLIP-362: Support minimum resource limitation

2023-09-25 Thread xiangyu feng
Hi all, I would like to start the vote for FLIP-362: Support minimum resource limitation[1]. This FLIP was discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. Regards, Xiangyu [1]

[jira] [Created] (FLINK-33153) Kafka using latest-offset maybe missing data

2023-09-25 Thread tanjialiang (Jira)
tanjialiang created FLINK-33153: --- Summary: Kafka using latest-offset maybe missing data Key: FLINK-33153 URL: https://issues.apache.org/jira/browse/FLINK-33153 Project: Flink Issue Type: Bug

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-25 Thread Zakelly Lan
Hi Jing and everyone, I have conducted three rounds of benchmarking with Java11, comparing release 1.18 (commit: deb07e99560[1]) with commit 6d62f9918ea[2]. The results are attached[3]. Most of the tests show no obvious regression. However, I did observe significant change in several tests. Upon

Re: [DISCUSS] FLIP-328: Allow source operators to determine isProcessingBacklog based on watermark lag

2023-09-25 Thread Piotr Nowojski
Hi Jarl and Dong, I'm a bit confused about the difference between the two competing options. Could one of you elaborate what's the difference between: > 2) The semantics of `#setIsProcessingBacklog(false)` is that it overrides > the effect of the previous invocation (if any) of >

[jira] [Created] (FLINK-33152) Prometheus Sink Connector - Integration tests

2023-09-25 Thread Lorenzo Nicora (Jira)
Lorenzo Nicora created FLINK-33152: -- Summary: Prometheus Sink Connector - Integration tests Key: FLINK-33152 URL: https://issues.apache.org/jira/browse/FLINK-33152 Project: Flink Issue

[DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

2023-09-25 Thread Yuepeng Pan
Hi all, I and Fan Rui(CC’ed) created the FLIP-370[1] to support balanced tasks scheduling. The current strategy of Flink to deploy tasks sometimes leads some TMs(TaskManagers) to have more tasks while others have fewer tasks, resulting in excessive resource utilization at some TMs that

[jira] [Created] (FLINK-33151) Prometheus Sink Connector - Create Github Repo

2023-09-25 Thread Danny Cranmer (Jira)
Danny Cranmer created FLINK-33151: - Summary: Prometheus Sink Connector - Create Github Repo Key: FLINK-33151 URL: https://issues.apache.org/jira/browse/FLINK-33151 Project: Flink Issue Type:

[jira] [Created] (FLINK-33150) add the processing logic for the long type

2023-09-25 Thread wenhao.yu (Jira)
wenhao.yu created FLINK-33150: - Summary: add the processing logic for the long type Key: FLINK-33150 URL: https://issues.apache.org/jira/browse/FLINK-33150 Project: Flink Issue Type: New

[jira] [Created] (FLINK-33149) Bump snappy-java to 1.1.10.4

2023-09-25 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-33149: --- Summary: Bump snappy-java to 1.1.10.4 Key: FLINK-33149 URL: https://issues.apache.org/jira/browse/FLINK-33149 Project: Flink Issue Type: Bug

Re: Future of classical remoting in Pekko

2023-09-25 Thread He Pin
The related changes are : https://github.com/apache/incubator-pekko/pull/643 https://github.com/apache/incubator-pekko/pull/651 https://github.com/apache/incubator-pekko/pull/652 https://github.com/apache/incubator-pekko/pull/656 It would be very helpful if you take another round of review,

Re: Future of classical remoting in Pekko

2023-09-25 Thread He Pin
All the relatived changes have been merged,Which is mainly for CVEs, and there are some changes relative to performance too. Currently pekko only releases snapshot version, if you decide to contains it in 1.19, will need pekko1.1.0 to release before right? On 2023/09/20 14:29:45 Ferenc Csaky