Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-10-23 Thread Tzu-Li (Gordon) Tai
21:39 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [ANNOUNCE] Release 1.18.0, release candidate #0 > Hi David, > > It’s a deliberate choice to decouple the connectors. We shouldn’t block > Flink 1.18 on connector statuses. There’s already work being done to fix > the Flink

RE: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-10-06 Thread David Radley
regression like this, Kind regards, David. From: Martijn Visser Date: Thursday, 5 October 2023 at 21:39 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [ANNOUNCE] Release 1.18.0, release candidate #0 Hi David, It’s a deliberate choice to decouple the connectors. We shouldn’t block Flink 1.18

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-10-05 Thread Martijn Visser
wanting to release a Kafka connector when there are no changes just to have > a match with the Flink core version. > > Kind regards, David. > > > > From: Jing Ge > Date: Wednesday, 4 October 2023 at 17:36 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [ANNOUNCE]

RE: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-10-05 Thread David Radley
October 2023 at 17:36 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [ANNOUNCE] Release 1.18.0, release candidate #0 Hi David, First of all, we should have enough time to wait for those issues to be resolved. Secondly, it makes less sense to block upstream release by downstream build issues. In case

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-10-04 Thread Jing Ge
and > https://issues.apache.org/jira/browse/FLINK-33017 are resolved before > releasing 1.18? > > Kind regards, David. > > > From: Jing Ge > Date: Wednesday, 27 September 2023 at 15:11 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [ANNOUNCE] Release 1.18.0,

RE: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-10-04 Thread David Radley
? Kind regards, David. From: Jing Ge Date: Wednesday, 27 September 2023 at 15:11 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [ANNOUNCE] Release 1.18.0, release candidate #0 Hi Folks, @Ryan FYI: CI passed and the PR has been merged. Thanks! If there are no more other concerns, I will

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-27 Thread Jing Ge
Hi Folks, @Ryan FYI: CI passed and the PR has been merged. Thanks! If there are no more other concerns, I will start publishing 1.18-rc1. Best regards, Jing On Mon, Sep 25, 2023 at 1:40 PM Jing Ge wrote: > Hi Ryan, > > Thanks for reaching out. It is fine to include it but we need to wait > un

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-25 Thread Jing Ge
Hi Ryan, Thanks for reaching out. It is fine to include it but we need to wait until the CI passes. I am not sure how long it will take, since there seems to be some infra issues. Best regards, Jing On Mon, Sep 25, 2023 at 11:34 AM Ryan Skraba wrote: > Hello! There's a security fix that proba

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-25 Thread Ryan Skraba
Hello! There's a security fix that probably should be applied to 1.18 in the next RC1 : https://github.com/apache/flink/pull/23461 (bump to snappy-java). Do you think this would be possible to include? All my best, Ryan [1]: https://issues.apache.org/jira/browse/FLINK-33149 "Bump snappy-java to

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-25 Thread Jing Ge
Thanks Zakelly for the update! Appreciate it! @Piotr Nowojski If you do not have any other concerns, I will move forward to create 1.18 rc1 and start voting. WDYT? Best regards, Jing On Mon, Sep 25, 2023 at 2:20 AM Zakelly Lan wrote: > Hi Jing and everyone, > > I have conducted three rounds o

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-25 Thread Zakelly Lan
Hi Jing and everyone, I have conducted three rounds of benchmarking with Java11, comparing release 1.18 (commit: deb07e99560[1]) with commit 6d62f9918ea[2]. The results are attached[3]. Most of the tests show no obvious regression. However, I did observe significant change in several tests. Upon r

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-23 Thread ConradJam
+1 for testing with Java 17 Jing Ge 于2023年9月24日周日 09:40写道: > +1 for testing with Java 17 too. Thanks Zakelly for your effort! > > Best regards, > Jing > > On Fri, Sep 22, 2023 at 1:01 PM Zakelly Lan wrote: > > > Hi Jing, > > > > I agree we could wait for the result with Java 11. And it should b

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-23 Thread Jing Ge
+1 for testing with Java 17 too. Thanks Zakelly for your effort! Best regards, Jing On Fri, Sep 22, 2023 at 1:01 PM Zakelly Lan wrote: > Hi Jing, > > I agree we could wait for the result with Java 11. And it should be > available next Monday. > Additionally, I could also build a pipeline with J

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-22 Thread Zakelly Lan
Hi Jing, I agree we could wait for the result with Java 11. And it should be available next Monday. Additionally, I could also build a pipeline with Java 17 later since it is supported in 1.18[1]. Best regards, Zakelly [1] https://github.com/apache/flink/commit/9c1318ca7fa5b2e7b11827068ad12884

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-22 Thread Jing Ge
Hi Zakelly, Thanks for your effort and the update! Since Java 8 has been deprecated[1], let's wait for the result with Java 11. It should be available after the weekend and there should be no big surprise. WDYT? Best regards, Jing [1] https://nightlies.apache.org/flink/flink-docs-master/release-

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-22 Thread Zakelly Lan
Hi everyone, I want to provide an update on the benchmark results that I have been working on. After spending some time preparing the environment and adjusting the benchmark script, I finally got a comparison between release 1.18 (commit: 2aeb99804ba[1]) and the commit before the old codespeed ser

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Yuxin Tan
Hi, Zakelly, No benchmark tests currently are affected by this issue. We may add benchmarks to guard it later. Thanks. Best, Yuxin Zakelly Lan 于2023年9月21日周四 11:56写道: > Hi Jing, > > Sure, I will run the benchmark with this fix. > > Hi Yunxin, > > I'm not familiar with the hybrid shuffle. Is th

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Zakelly Lan
Hi Jing, Sure, I will run the benchmark with this fix. Hi Yunxin, I'm not familiar with the hybrid shuffle. Is there any specific benchmark test that may be affected by this issue? I will pay special attention to it. Thanks. Best, Zakelly On Thu, Sep 21, 2023 at 10:08 AM Yuxin Tan wrote: > >

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Yuxin Tan
Hi, Jing, Qingsheng, Thanks a lot. The fix has been backported. Best, Yuxin Jing Ge 于2023年9月21日周四 00:42写道: > Hi Lijie, > > Thanks for reaching out. Please backport it to release-1.18. > > Best regards, > Jing > > On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang > wrote: > > > Hi community and rele

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Jing Ge
Hi Lijie, Thanks for reaching out. Please backport it to release-1.18. Best regards, Jing On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang wrote: > Hi community and release managers: > > We found a critical bug[1] of the rest client a few days ago, which may > cause the inode to be used up. Now the

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Yuxin Tan
Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing. The fix

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Qingsheng Ren
Thanks for the update, Lijie! The patch will be included in the next RC. Best, Qingsheng On Wed, Sep 20, 2023 at 10:12 PM Lijie Wang wrote: > Hi community and release managers: > > We found a critical bug[1] of the rest client a few days ago, which may > cause the inode to be used up. Now the f

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Yuxin Tan
Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing. The fix

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Yuxin Tan
Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing. The fix

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Yuxin Tan
Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing. The fix

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Qingsheng Ren
Thanks for letting us know, Yuxin. Please go ahead and backport it to 1.18. We will include the patch in the next release candidate. Best, Qingsheng On Wed, Sep 20, 2023 at 8:54 PM Yuxin Tan wrote: > Hi, dear community and release managers, > > Thanks for bringing this up. > > When testing the

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Lijie Wang
Hi community and release managers: We found a critical bug[1] of the rest client a few days ago, which may cause the inode to be used up. Now the fix-PR[2] is ready for merging, I hope to backport it to release-1.18. Please let me know if you have any concerns. Thanks. [1] https://issues.apache.

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Jing Ge
Hi Yuxin, Thanks for the update. LGTM. @Zakelly Please please be aware of that and make sure the benchmarks test will include that fix. Thanks! Best regards, Jing On Wed, Sep 20, 2023 at 8:51 AM Yuxin Tan wrote: > Hi, dear community and release managers, > > Thanks for bringing this up. > > Wh

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Yuxin Tan
Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing. The fix

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-19 Thread Zakelly Lan
Hi Yuan and Jing, Thank you for sharing your thoughts. I completely agree that it is our top priority to ensure that there are no regressions from the last commit the previous benchmark pipeline covered to the final commit of this release. I will try to get this result first. Best, Zakelly On T

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-19 Thread Jing Ge
Hi Thanks Zakelly and Yuan for your effort and update. Since we changed the hardware, IMHO, if we are able to reach a consensus in the community that there is no regression with the benchmarks, we could consider releasing rc1 without waiting for the new baseline scores which might take days. Best

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-19 Thread Yuan Mei
Hey Zakelly, Thanks very much for the efforts to re-build the entire benchmark environment. As long as we have 1) the pipeline set up and ready (no need for the entire portal ready), 2) get benchmark comparison numbers (comparing with the commit just before the benchmark pipeline is down) and 3)

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-19 Thread Zakelly Lan
Hi everyone, I am working on rebuilding the benchmark pipeline and it's almost done. However, due to the change in machines for benchmarking, I will need a few more days to run tests and gather the baseline scores for further comparison. Once the pipeline is fully ready, we will proceed with the p