Hi ,
As release 1.18 removes the kafka connector from the core Flink repository, I
assume we will wait until the kafka connector nightly build issues
https://issues.apache.org/jira/browse/FLINK-33104 and
https://issues.apache.org/jira/browse/FLINK-33017 are resolved before releasing
1.18?
Kind regards, David.
From: Jing Ge <j...@ververica.com.INVALID>
Date: Wednesday, 27 September 2023 at 15:11
To: dev@flink.apache.org <dev@flink.apache.org>
Subject: [EXTERNAL] Re: [ANNOUNCE] Release 1.18.0, release candidate #0
Hi Folks,
@Ryan FYI: CI passed and the PR has been merged. Thanks!
If there are no more other concerns, I will start publishing 1.18-rc1.
Best regards,
Jing
On Mon, Sep 25, 2023 at 1:40 PM Jing Ge <j...@ververica.com> wrote:
> Hi Ryan,
>
> Thanks for reaching out. It is fine to include it but we need to wait
> until the CI passes. I am not sure how long it will take, since there seems
> to be some infra issues.
>
> Best regards,
> Jing
>
> On Mon, Sep 25, 2023 at 11:34 AM Ryan Skraba <ryan.skr...@aiven.io.invalid>
> wrote:
>
>> Hello! There's a security fix that probably should be applied to 1.18
>> in the next RC1 : https://github.com/apache/flink/pull/23461 (bump to
>> snappy-java). Do you think this would be possible to include?
>>
>> All my best, Ryan
>>
>> [1]: https://issues.apache.org/jira/browse/FLINK-33149 "Bump
>> snappy-java to 1.1.10.4"
>>
>>
>>
>> On Mon, Sep 25, 2023 at 3:54 PM Jing Ge <j...@ververica.com.invalid>
>> wrote:
>> >
>> > Thanks Zakelly for the update! Appreciate it!
>> >
>> > @Piotr Nowojski <pnowoj...@apache.org> If you do not have any other
>> > concerns, I will move forward to create 1.18 rc1 and start voting. WDYT?
>> >
>> > Best regards,
>> > Jing
>> >
>> > On Mon, Sep 25, 2023 at 2:20 AM Zakelly Lan <zakelly....@gmail.com>
>> wrote:
>> >
>> > > Hi Jing and everyone,
>> > >
>> > > I have conducted three rounds of benchmarking with Java11, comparing
>> > > release 1.18 (commit: deb07e99560[1]) with commit 6d62f9918ea[2]. The
>> > > results are attached[3]. Most of the tests show no obvious regression.
>> > > However, I did observe significant change in several tests. Upon
>> > > reviewing the historical results from the previous pipeline, I also
>> > > discovered a substantial variance in those tests, as shown in the
>> > > timeline pictures included in the sheet[3]. I believe this variance
>> > > has existed for a long time and requires further investigation, and
>> > > fully measuring the variance requires more rounds (15 or more). I
>> > > think for now it is not a blocker for release 1.18. WDYT?
>> > >
>> > >
>> > > Best,
>> > > Zakelly
>> > >
>> > > [1]
>> > >
>> https://github.com/apache/flink/commit/deb07e99560b45033a629afc3f90666ad0a32feb
>> > > [2]
>> > >
>> https://github.com/apache/flink/commit/6d62f9918ea2cbb8a10c705a25a4ff6deab60711
>> > > [3]
>> > >
>> https://docs.google.com/spreadsheets/d/1V0-duzNTgu7H6R7kioF-TAPhlqWl7Co6Q9ikTBuaULo/edit?usp=sharing
>> > >
>> > > On Sun, Sep 24, 2023 at 11:29 AM ConradJam <jam.gz...@gmail.com>
>> wrote:
>> > > >
>> > > > +1 for testing with Java 17
>> > > >
>> > > > Jing Ge <j...@ververica.com.invalid> 于2023年9月24日周日 09:40写道：
>> > > >
>> > > > > +1 for testing with Java 17 too. Thanks Zakelly for your effort!
>> > > > >
>> > > > > Best regards,
>> > > > > Jing
>> > > > >
>> > > > > On Fri, Sep 22, 2023 at 1:01 PM Zakelly Lan <
>> zakelly....@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi Jing,
>> > > > > >
>> > > > > > I agree we could wait for the result with Java 11. And it
>> should be
>> > > > > > available next Monday.
>> > > > > > Additionally, I could also build a pipeline with Java 17 later
>> since
>> > > > > > it is supported in 1.18[1].
>> > > > > >
>> > > > > >
>> > > > > > Best regards,
>> > > > > > Zakelly
>> > > > > >
>> > > > > > [1]
>> > > > > >
>> > > > >
>> > >
>> https://github.com/apache/flink/commit/9c1318ca7fa5b2e7b11827068ad1288483aaa464#diff-8310c97396d60e96766a936ca8680f1e2971ef486cfc2bc55ec9ca5a5333c47fR53
>> > > > > >
>> > > > > > On Fri, Sep 22, 2023 at 5:57 PM Jing Ge
>> <j...@ververica.com.invalid>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > Hi Zakelly,
>> > > > > > >
>> > > > > > > Thanks for your effort and the update! Since Java 8 has been
>> > > > > > deprecated[1],
>> > > > > > > let's wait for the result with Java 11. It should be available
>> > > after
>> > > > > the
>> > > > > > > weekend and there should be no big surprise. WDYT?
>> > > > > > >
>> > > > > > > Best regards,
>> > > > > > > Jing
>> > > > > > >
>> > > > > > > [1]
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.15/#jdk-upgrade
>> > > > > > >
>> > > > > > > On Fri, Sep 22, 2023 at 11:26 AM Zakelly Lan <
>> > > zakelly....@gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi everyone,
>> > > > > > > >
>> > > > > > > > I want to provide an update on the benchmark results that I
>> have
>> > > been
>> > > > > > > > working on. After spending some time preparing the
>> environment
>> > > and
>> > > > > > > > adjusting the benchmark script, I finally got a comparison
>> > > between
>> > > > > > > > release 1.18 (commit: 2aeb99804ba[1]) and the commit before
>> the
>> > > old
>> > > > > > > > codespeed server went down (commit: 6d62f9918ea[2]) on
>> openjdk8.
>> > > The
>> > > > > > > > report is attached[3]. Note that the test has only run once
>> on
>> > > jdk8,
>> > > > > > > > so the impact of single-test fluctuations is not ruled out.
>> > > > > > > > Additionally, I have noticed some significant fluctuations
>> in
>> > > > > specific
>> > > > > > > > tests when reviewing previous benchmark scores, which I
>> have also
>> > > > > > > > noted in the report. Taking all of these factors into
>> > > consideration,
>> > > > > I
>> > > > > > > > think there is no obvious regression in release 1.18 *for
>> now*.
>> > > More
>> > > > > > > > tests including the one on openjdk11 are on the way. Hope it
>> > > does not
>> > > > > > > > delay the release procedure.
>> > > > > > > >
>> > > > > > > > Please let me know if you have any concerns.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > Zakelly
>> > > > > > > >
>> > > > > > > > [1]
>> > > > > > > >
>> > > > > >
>> > > > >
>> > >
>> https://github.com/apache/flink/commit/2aeb99804ba56c008df0a1730f3246d3fea856b9
>> > > > > > > > [2]
>> > > > > > > >
>> > > > > >
>> > > > >
>> > >
>> https://github.com/apache/flink/commit/6d62f9918ea2cbb8a10c705a25a4ff6deab60711
>> > > > > > > > [3]
>> > > > > > > >
>> > > > > >
>> > > > >
>> > >
>> https://docs.google.com/spreadsheets/d/1-3Y974jYq_WrQNzLN-y_6lOU-NGXaIDTQBYTZd04tJ0/edit?usp=sharing
>> > > > > > > >
>> > > > > > > > The new environment for benchmark:
>> > > > > > > > ECS on Aliyun
>> > > > > > > > CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz (8 Core
>> > > available)
>> > > > > > > > Memory: 64GB
>> > > > > > > > OS: Alibaba Cloud Linux 3.2104 LTS 64bit
>> > > > > > > > Kernel: 5.10.134-15.al8.x86_64
>> > > > > > > > OpenJDK8 version: 1.8.0_372
>> > > > > > > >
>> > > > > > > > On Thu, Sep 21, 2023 at 12:04 PM Yuxin Tan <
>> > > tanyuxinw...@gmail.com>
>> > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > Hi, Zakelly,
>> > > > > > > > >
>> > > > > > > > > No benchmark tests currently are affected by this issue.
>> We
>> > > > > > > > > may add benchmarks to guard it later. Thanks.
>> > > > > > > > >
>> > > > > > > > > Best,
>> > > > > > > > > Yuxin
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Zakelly Lan <zakelly....@gmail.com> 于2023年9月21日周四
>> 11:56写道：
>> > > > > > > > >
>> > > > > > > > > > Hi Jing,
>> > > > > > > > > >
>> > > > > > > > > > Sure, I will run the benchmark with this fix.
>> > > > > > > > > >
>> > > > > > > > > > Hi Yunxin,
>> > > > > > > > > >
>> > > > > > > > > > I'm not familiar with the hybrid shuffle. Is there any
>> > > specific
>> > > > > > > > > > benchmark test that may be affected by this issue? I
>> will pay
>> > > > > > special
>> > > > > > > > > > attention to it.
>> > > > > > > > > > Thanks.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Best,
>> > > > > > > > > > Zakelly
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Sep 21, 2023 at 10:08 AM Yuxin Tan <
>> > > > > tanyuxinw...@gmail.com
>> > > > > > >
>> > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > Hi, Jing, Qingsheng,
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks a lot.
>> > > > > > > > > > > The fix has been backported.
>> > > > > > > > > > >
>> > > > > > > > > > > Best,
>> > > > > > > > > > > Yuxin
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > Jing Ge <j...@ververica.com.invalid> 于2023年9月21日周四
>> > > 00:42写道：
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi Lijie,
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks for reaching out. Please backport it to
>> > > release-1.18.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > Jing
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang <
>> > > > > > > > wangdachui9...@gmail.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi community and release managers:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > We found a critical bug[1] of the rest client a
>> few
>> > > days
>> > > > > ago,
>> > > > > > > > which
>> > > > > > > > > > may
>> > > > > > > > > > > > > cause the inode to be used up. Now the fix-PR[2]
>> is
>> > > ready
>> > > > > for
>> > > > > > > > > > merging, I
>> > > > > > > > > > > > > hope to backport it to release-1.18.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Please let me know if you have any concerns.
>> Thanks.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > [1]
>> https://issues.apache.org/jira/browse/FLINK-32974
>> > > > > > > > > > > > > [2] https://github.com/apache/flink/pull/23363
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > Lijie
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Zakelly Lan <zakelly....@gmail.com> 于2023年9月19日周二
>> > > 17:26写道：
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi Yuan and Jing,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thank you for sharing your thoughts. I
>> completely
>> > > agree
>> > > > > > that
>> > > > > > > > it is
>> > > > > > > > > > our
>> > > > > > > > > > > > > > top priority to ensure that there are no
>> regressions
>> > > from
>> > > > > > the
>> > > > > > > > last
>> > > > > > > > > > > > > > commit the previous benchmark pipeline covered
>> to the
>> > > > > final
>> > > > > > > > commit
>> > > > > > > > > > of
>> > > > > > > > > > > > > > this release. I will try to get this result
>> first.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > Zakelly
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Tue, Sep 19, 2023 at 4:55 PM Jing Ge
>> > > > > > > > <j...@ververica.com.invalid
>> > > > > > > > > > >
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hi
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Thanks Zakelly and Yuan for your effort and
>> update.
>> > > > > > Since we
>> > > > > > > > > > changed
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > hardware, IMHO, if we are able to reach a
>> > > consensus in
>> > > > > > the
>> > > > > > > > > > community
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > there is no regression with the benchmarks, we
>> > > could
>> > > > > > consider
>> > > > > > > > > > > > releasing
>> > > > > > > > > > > > > > rc1
>> > > > > > > > > > > > > > > without waiting for the new baseline scores
>> which
>> > > might
>> > > > > > take
>> > > > > > > > > > days.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > Jing
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei <
>> > > > > > > > > > yuanmei.w...@gmail.com>
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Hey Zakelly,
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Thanks very much for the efforts to
>> re-build the
>> > > > > entire
>> > > > > > > > > > benchmark
>> > > > > > > > > > > > > > > > environment.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > As long as we have
>> > > > > > > > > > > > > > > > 1) the pipeline set up and ready (no need
>> for the
>> > > > > > entire
>> > > > > > > > portal
>> > > > > > > > > > > > > ready),
>> > > > > > > > > > > > > > > > 2) get benchmark comparison numbers
>> (comparing
>> > > with
>> > > > > the
>> > > > > > > > commit
>> > > > > > > > > > just
>> > > > > > > > > > > > > > before
>> > > > > > > > > > > > > > > > the benchmark pipeline is down) and
>> > > > > > > > > > > > > > > > 3) confirmed no-regression, it should be
>> good
>> > > enough.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Thanks again!
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Best
>> > > > > > > > > > > > > > > > Yuan
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan
>> <
>> > > > > > > > > > zakelly....@gmail.com
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Hi everyone,
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > I am working on rebuilding the benchmark
>> > > pipeline
>> > > > > and
>> > > > > > > > it's
>> > > > > > > > > > almost
>> > > > > > > > > > > > > > > > > done. However, due to the change in
>> machines
>> > > for
>> > > > > > > > > > benchmarking, I
>> > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > need a few more days to run tests and
>> gather
>> > > the
>> > > > > > baseline
>> > > > > > > > > > scores
>> > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > further comparison. Once the pipeline is
>> fully
>> > > > > > ready, we
>> > > > > > > > will
>> > > > > > > > > > > > > proceed
>> > > > > > > > > > > > > > > > > with the performance test for release
>> 1.18.0.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Please let me know if you have any
>> concerns.
>> > > Thank
>> > > > > > you
>> > > > > > > > all
>> > > > > > > > > > for
>> > > > > > > > > > > > your
>> > > > > > > > > > > > > > > > > patience.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > Zakelly
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge
>> > > > > > > > > > > > <j...@ververica.com.invalid
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Hi everyone,
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > The RC0 for Apache Flink 1.18.0 has been
>> > > created.
>> > > > > > This
>> > > > > > > > RC
>> > > > > > > > > > is
>> > > > > > > > > > > > > > currently
>> > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > preview only to facilitate the
>> integrated
>> > > testing
>> > > > > > > > since the
>> > > > > > > > > > > > > > benchmark
>> > > > > > > > > > > > > > > > > tests
>> > > > > > > > > > > > > > > > > > are not available yet[1] and the release
>> > > > > > announcement
>> > > > > > > > is
>> > > > > > > > > > still
>> > > > > > > > > > > > > > under
>> > > > > > > > > > > > > > > > > > review. The RC1 will be released after
>> all
>> > > > > > benchmarks
>> > > > > > > > > > tests are
>> > > > > > > > > > > > > > passed.
>> > > > > > > > > > > > > > > > > The
>> > > > > > > > > > > > > > > > > > related voting process will be triggered
>> > > once the
>> > > > > > > > > > announcement
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > ready.
>> > > > > > > > > > > > > > > > > > The RC0 has all the artifacts that we
>> would
>> > > > > > typically
>> > > > > > > > have
>> > > > > > > > > > for
>> > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > release,
>> > > > > > > > > > > > > > > > > > except for the release note and the
>> website
>> > > pull
>> > > > > > > > request
>> > > > > > > > > > for
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > release
>> > > > > > > > > > > > > > > > > > announcement.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > The following contents are available
>> for your
>> > > > > > review:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > - The preview source release and binary
>> > > > > convenience
>> > > > > > > > > > releases
>> > > > > > > > > > > > [2],
>> > > > > > > > > > > > > > which
>> > > > > > > > > > > > > > > > > > are signed with the key with fingerprint
>> > > > > > > > > > 96AE0E32CBE6E0753CE6
>> > > > > > > > > > > > > [3].
>> > > > > > > > > > > > > > > > > > - all artifacts that would normally be
>> > > deployed
>> > > > > to
>> > > > > > the
>> > > > > > > > > > Maven
>> > > > > > > > > > > > > > > > > > Central Repository [4].
>> > > > > > > > > > > > > > > > > > - source code tag "release-1.18.0-rc0"
>> [5]
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Your help testing the release will be
>> greatly
>> > > > > > > > appreciated!
>> > > > > > > > > > And
>> > > > > > > > > > > > > > we'll
>> > > > > > > > > > > > > > > > > > create the rc1 release and the voting
>> thread
>> > > as
>> > > > > > soon as
>> > > > > > > > > > all the
>> > > > > > > > > > > > > > efforts
>> > > > > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > finished.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > [1]
>> > > > > > https://issues.apache.org/jira/browse/FLINK-33052
>> > > > > > > > > > > > > > > > > > [2]
>> > > > > > > > > > > > >
>> > > > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
>> > > > > > > > > > > > > > > > > > [3]
>> > > > > > > > https://dist.apache.org/repos/dist/release/flink/KEYS
>> > > > > > > > > > > > > > > > > > [4]
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > >
>> https://repository.apache.org/content/repositories/orgapacheflink-1656/
>> > > > > > > > > > > > > > > > > > [5]
>> > > > > > > > > > > > > >
>> > > > > > > >
>> https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > > Qingsheng, Sergei, Konstantin and Jing
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Best
>> > > >
>> > > > ConradJam
>> > >
>>
>
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
