Re: [VOTE] Release 1.19.1, release candidate #1

[Rui Fan]

+1(binding)

- Reviewed the flink-web PR (Left some comments)
- Checked Github release tag
- Verified signatures
- Verified sha512 (hashsums)
- The source archives do not contain any binaries
- Build the source with Maven 3 and java8 (Checked the license as well)
- Start the cluster locally with jdk8, and run the StateMachineExample job,
it works fine.

Best,
Rui

On Thu, Jun 6, 2024 at 11:39 PM Hong Liang  wrote:

> Hi everyone,
> Please review and vote on the release candidate #1 for the flink v1.19.1,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint B78A5EA1 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.19.1-rc1" [5],
> * website pull request listing the new release and adding announcement blog
> post [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Hong
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354399
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.19.1-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1736/
> [5] https://github.com/apache/flink/releases/tag/release-1.19.1-rc1
> [6] https://github.com/apache/flink-web/pull/745
>

Re: [VOTE] Release flink-connector-gcp-pubsub v3.1.0, release candidate #1

[Rui Fan]

Thanks Danny for the hard work!

+1(binding)

- Verified signatures
- Verified sha512 (hashsums)
- The source archives do not contain any binaries
- Build the source with Maven 3 and java8 (Checked the license as well)
- Checked Github release tag
- Reviewed the flink-web PR

Best,
Rui

On Wed, May 22, 2024 at 8:07 PM Leonard Xu  wrote:

>
> +1 (binding)
>
> - verified signatures
> - verified hashsums
> - built from source code with java 1.8 succeeded
> - checked Github release tag
> - checked release notes
> - reviewed the web PR
>
> Best,
> Leonard
>
> > 2024年4月21日 下午9:52，Hang Ruan  写道：
> >
> > +1 (non-binding)
> >
> > - Validated checksum hash
> > - Verified signature
> > - Verified that no binaries exist in the source archive
> > - Build the source with Maven and jdk8
> > - Verified web PR
> > - Check that the jar is built by jdk8
> >
> > Best,
> > Hang
> >
> > Ahmed Hamdy  于2024年4月18日周四 20:01写道：
> >
> >> Hi Danny,
> >> +1 (non-binding)
> >>
> >> -  verified hashes and checksums
> >> - verified signature
> >> - verified source contains no binaries
> >> - tag exists in github
> >> - reviewed web PR
> >>
> >> Best Regards
> >> Ahmed Hamdy
> >>
> >>
> >> On Thu, 18 Apr 2024 at 11:32, Danny Cranmer 
> >> wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> Please review and vote on release candidate #1 for
> >>> flink-connector-gcp-pubsub v3.1.0, as follows:
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>
> >>> This release supports Flink 1.18 and 1.19.
> >>>
> >>> The complete staging area is available for your review, which includes:
> >>> * JIRA release notes [1],
> >>> * the official Apache source release to be deployed to dist.apache.org
> >>> [2],
> >>> which are signed with the key with fingerprint 125FD8DB [3],
> >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>> * source code tag v3.1.0-rc1 [5],
> >>> * website pull request listing the new release [6].
> >>> * CI build of the tag [7].
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by majority
> >>> approval, with at least 3 PMC affirmative votes.
> >>>
> >>> Thanks,
> >>> Danny
> >>>
> >>> [1]
> >>>
> >>>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353813
> >>> [2]
> >>>
> >>>
> >>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-gcp-pubsub-3.1.0-rc1
> >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >>> [4]
> >> https://repository.apache.org/content/repositories/orgapacheflink-1720
> >>> [5]
> >>>
> >>>
> >>
> https://github.com/apache/flink-connector-gcp-pubsub/releases/tag/v3.1.0-rc1
> >>> [6] https://github.com/apache/flink-web/pull/736/files
> >>> [7]
> >>>
> >>>
> >>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8735952883
> >>>
> >>
>
>

Re: [VOTE] Release flink-connector-opensearch v1.2.0, release candidate #1

[Rui Fan]

Thanks Sergey for the hard work!

+1(binding)

- Verified signatures
- Verified sha512 (hashsums)
- The source archives do not contain any binaries
- Build the source with Maven 3 and java8 (Checked the license as well)
- Checked Github release tag
- Reviewed the flink-web PR

Best,
Rui

On Thu, May 30, 2024 at 8:22 PM weijie guo 
wrote:

> Thanks Sergey for driving this release!
>
> +1(non-binding)
>
> 1. Verified signatures and hash sums
> 2. Build from source with 1.8.0_291 succeeded
> 3. Checked RN.
>
> Best regards,
>
> Weijie
>
>
> Yuepeng Pan  于2024年5月30日周四 10:08写道：
>
> > +1 (non-binding)
> >
> > - Built from source code with JDK 1.8 on MaxOS- Run examples locally.-
> > Checked release notes Best, Yuepeng Pan
> >
> >
> > At 2024-05-28 22:53:10, "gongzhongqiang" 
> > wrote:
> > >+1(non-binding)
> > >
> > >- Verified signatures and hash sums
> > >- Reviewed the web PR
> > >- Built from source code with JDK 1.8 on Ubuntu 22.04
> > >- Checked release notes
> > >
> > >Best,
> > >Zhongqiang Gong
> > >
> > >
> > >Sergey Nuyanzin  于2024年5月16日周四 06:03写道：
> > >
> > >> Hi everyone,
> > >> Please review and vote on release candidate #1 for
> > >> flink-connector-opensearch v1.2.0, as follows:
> > >> [ ] +1, Approve the release
> > >> [ ] -1, Do not approve the release (please provide specific comments)
> > >>
> > >>
> > >> The complete staging area is available for your review, which
> includes:
> > >> * JIRA release notes [1],
> > >> * the official Apache source release to be deployed to
> dist.apache.org
> > >> [2],
> > >> which are signed with the key with fingerprint
> > >> F7529FAE24811A5C0DF3CA741596BBF0726835D8 [3],
> > >> * all artifacts to be deployed to the Maven Central Repository [4],
> > >> * source code tag v1.2.0-rc1 [5],
> > >> * website pull request listing the new release [6].
> > >> * CI build of the tag [7].
> > >>
> > >> The vote will be open for at least 72 hours. It is adopted by majority
> > >> approval, with at least 3 PMC affirmative votes.
> > >>
> > >> Note that this release is for Opensearch v1.x
> > >>
> > >> Thanks,
> > >> Release Manager
> > >>
> > >> [1] https://issues.apache.org/jira/projects/FLINK/versions/12353812
> > >> [2]
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-opensearch-1.2.0-rc1
> > >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > >> [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1734
> > >> [5]
> > >>
> > >>
> >
> https://github.com/apache/flink-connector-opensearch/releases/tag/v1.2.0-rc1
> > >> [6] https://github.com/apache/flink-web/pull/740
> > >> [7]
> > >>
> > >>
> >
> https://github.com/apache/flink-connector-opensearch/actions/runs/9102334125
> > >>
> >
>

Re: [VOTE] Release flink-connector-opensearch v2.0.0, release candidate #1

[Rui Fan]

Thanks Sergey for the hard work!

+1(binding)

- Verified signatures
- Verified sha512 (hashsums)
- The source archives do not contain any binaries
- Build the source with Maven 3 and java8 (Checked the license as well)
- Checked Github release tag
- Reviewed the flink-web PR

Best,
Rui

On Mon, May 27, 2024 at 5:35 PM Hang Ruan  wrote:

> +1 (non-binding)
>
> - verified signatures
> - verified hashsums
> - built from source code with JDK 11 succeed
> - checked release notes
> - reviewed the web PR
>
> Best,
> Hang
>
> Leonard Xu  于2024年5月22日周三 21:02写道：
>
> >
> > > +1 (binding)
> > >
> > > - verified signatures
> > > - verified hashsums
> > > - built from source code with JDK 1.8 succeeded
> > > - checked Github release tag
> > > - checked release notes
> > > - reviewed the web PR
> >
> > Supply more information about build from source code with JDK 1.8
> >
> > > - built from source code with JDK 1.8 succeeded
> > It’s correct as we don’t activate opensearch2 profile by default.
> >
> > - built from source code with JDK 1.8 and -Popensearch2 failed
> > - built from source code with JDK 11 and -Popensearch2 succeeded
> >
> > Best,
> > Leonard
> >
> >
> > >
> > >
> > >> 2024年5月16日 上午6:58，Andrey Redko  写道：
> > >>
> > >> +1 (non-binding), thanks Sergey!
> > >>
> > >> On Wed, May 15, 2024, 6:00 p.m. Sergey Nuyanzin 
> > wrote:
> > >>
> > >>> Hi everyone,
> > >>> Please review and vote on release candidate #1 for
> > >>> flink-connector-opensearch v2.0.0, as follows:
> > >>> [ ] +1, Approve the release
> > >>> [ ] -1, Do not approve the release (please provide specific comments)
> > >>>
> > >>>
> > >>> The complete staging area is available for your review, which
> includes:
> > >>> * JIRA release notes [1],
> > >>> * the official Apache source release to be deployed to
> dist.apache.org
> > >>> [2],
> > >>> which are signed with the key with fingerprint
> > >>> F7529FAE24811A5C0DF3CA741596BBF0726835D8 [3],
> > >>> * all artifacts to be deployed to the Maven Central Repository [4],
> > >>> * source code tag v2.0.0-rc1 [5],
> > >>> * website pull request listing the new release [6].
> > >>> * CI build of the tag [7].
> > >>>
> > >>> The vote will be open for at least 72 hours. It is adopted by
> majority
> > >>> approval, with at least 3 PMC affirmative votes.
> > >>>
> > >>> Note that this release is for Opensearch v2.x
> > >>>
> > >>> Thanks,
> > >>> Release Manager
> > >>>
> > >>> [1] https://issues.apache.org/jira/projects/FLINK/versions/12354674
> > >>> [2]
> > >>>
> > >>>
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-opensearch-2.0.0-rc1
> > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > >>> [4]
> > >>>
> > https://repository.apache.org/content/repositories/orgapacheflink-1735/
> > >>> [5]
> > >>>
> > >>>
> >
> https://github.com/apache/flink-connector-opensearch/releases/tag/v2.0.0-rc1
> > >>> [6] https://github.com/apache/flink-web/pull/741
> > >>> [7]
> > >>>
> > >>>
> >
> https://github.com/apache/flink-connector-opensearch/actions/runs/9102980808
> > >>>
> > >
> >
> >
>

Re: [VOTE] Release flink-connector-jdbc v3.2.0, release candidate #2

[Rui Fan]

Thanks Danny for the hard work!

+1(binding)

- Verified signatures
- Verified sha512 (hashsums)
- The source archives do not contain any binaries
- Build the source with Maven 3 and java8 (Checked the license as well)
- Checked Github release tag
- Reviewed the flink-web PR

Best,
Rui

On Tue, Jun 4, 2024 at 1:31 PM gongzhongqiang 
wrote:

> +1 (non-binding)
>
> - Validated checksum hash and signature.
> - Confirmed that no binaries exist in the source archive.
> - Built the source with JDK 8.
> - Verified the web PR.
> - Ensured the JAR is built by JDK 8.
>
> Best,
> Zhongqiang Gong
>
> Danny Cranmer  于2024年4月18日周四 18:20写道：
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version 3.2.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > This release supports Flink 1.18 and 1.19.
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > [2],
> > which are signed with the key with fingerprint 125FD8DB [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag v3.2.0-rc1 [5],
> > * website pull request listing the new release [6].
> > * CI run of tag [7].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Danny
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353143
> > [2]
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-jdbc-3.2.0-rc2
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1718/
> > [5]
> https://github.com/apache/flink-connector-jdbc/releases/tag/v3.2.0-rc2
> > [6] https://github.com/apache/flink-web/pull/734
> > [7]
> https://github.com/apache/flink-connector-jdbc/actions/runs/8736019099
> >
>

Re: [VOTE] Release flink-connector-cassandra v3.2.0, release candidate #1

[Rui Fan]

Thanks Danny for the hard work!

+1(binding)

- Verified signatures
- Verified sha512 (hashsums)
- The source archives do not contain any binaries
- Build the source with Maven 3 and java8 (Checked the license as well)
- Checked Github release tag
- Reviewed the flink-web PR

Best,
Rui

On Wed, May 22, 2024 at 8:01 PM Leonard Xu  wrote:

> +1 (binding)
>
> - verified signatures
> - verified hashsums
> - built from source code with java 1.8 succeeded
> - checked Github release tag
> - checked release notes status which only left one issue is used for
> release tracking
> - reviewed the web PR
>
> Best,
> Leonard
>
> > 2024年5月22日 下午6:10，weijie guo  写道：
> >
> > +1(non-binding)
> >
> > -Validated checksum hash
> > -Verified signature
> > -Build from source
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Hang Ruan  于2024年5月22日周三 10:12写道：
> >
> >> +1 (non-binding)
> >>
> >> - Validated checksum hash
> >> - Verified signature
> >> - Verified that no binaries exist in the source archive
> >> - Build the source with Maven and jdk8
> >> - Verified web PR
> >> - Check that the jar is built by jdk8
> >>
> >> Best,
> >> Hang
> >>
> >> Muhammet Orazov  于2024年5月22日周三 04:15写道：
> >>
> >>> Hey all,
> >>>
> >>> Could we please get some more votes to proceed with the release?
> >>>
> >>> Thanks and best,
> >>> Muhammet
> >>>
> >>> On 2024-04-22 13:04, Danny Cranmer wrote:
>  Hi everyone,
> 
>  Please review and vote on release candidate #1 for
>  flink-connector-cassandra v3.2.0, as follows:
>  [ ] +1, Approve the release
>  [ ] -1, Do not approve the release (please provide specific comments)
> 
>  This release supports Flink 1.18 and 1.19.
> 
>  The complete staging area is available for your review, which
> includes:
>  * JIRA release notes [1],
>  * the official Apache source release to be deployed to
> dist.apache.org
>  [2],
>  which are signed with the key with fingerprint 125FD8DB [3],
>  * all artifacts to be deployed to the Maven Central Repository [4],
>  * source code tag v3.2.0-rc1 [5],
>  * website pull request listing the new release [6].
>  * CI build of the tag [7].
> 
>  The vote will be open for at least 72 hours. It is adopted by majority
>  approval, with at least 3 PMC affirmative votes.
> 
>  Thanks,
>  Danny
> 
>  [1]
> 
> >>>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353148
>  [2]
> 
> >>>
> >>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.2.0-rc1
>  [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>  [4]
> 
> https://repository.apache.org/content/repositories/orgapacheflink-1722
>  [5]
> 
> >>>
> >>
> https://github.com/apache/flink-connector-cassandra/releases/tag/v3.2.0-rc1
>  [6] https://github.com/apache/flink-web/pull/737
>  [7]
> 
> >>>
> >>
> https://github.com/apache/flink-connector-cassandra/actions/runs/8784310241
> >>>
> >>
>
>

Re: [VOTE] Release flink-connector-aws v4.3.0, release candidate #2

[Rui Fan]

Thanks Danny for the hard work!

+1(binding)

- Verified signatures
- Verified sha512 (hashsums)
- The source archives do not contain any binaries
- Build the source with Maven 3 and java8 (Checked the license as well)
- Checked Github release tag
- Reviewed the flink-web PR

Best,
Rui

On Fri, May 31, 2024 at 11:47 AM gongzhongqiang 
wrote:

> +1 (non-binding)
>
> - Validated the checksum hash and signature.
> - No binaries exist in the source archive.
> - Built the source with JDK 8 succeed.
> - Verified the flink-web PR.
> - Ensured the JAR is built by JDK 8.
>
> Best,
> Zhongqiang Gong
>
> Danny Cranmer  于2024年4月19日周五 18:08写道：
>
> > Hi everyone,
> >
> > Please review and vote on release candidate #2 for flink-connector-aws
> > v4.3.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > This version supports Flink 1.18 and 1.19.
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > [2],
> > which are signed with the key with fingerprint 125FD8DB [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag v4.3.0-rc2 [5],
> > * website pull request listing the new release [6].
> > * CI build of the tag [7].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353793
> > [2]
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-4.3.0-rc2
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1721/
> > [5]
> https://github.com/apache/flink-connector-aws/releases/tag/v4.3.0-rc2
> > [6] https://github.com/apache/flink-web/pull/733
> > [7]
> https://github.com/apache/flink-connector-aws/actions/runs/8751694197
> >
>

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

[Rui Fan]

Thanks Matthias for driving this proposal!

This proposal can reduce the amount of data that is processed repeatedly
after rescaling, so this proposal makes sense to me.

I have some questions:
1. The public change only includes the "New Configuration Parameters" part,
right?
2. jobmanager.adaptive-scheduler.rescale-on-failed-checkpoints-count is
obviously
  a config option for users. But I'm not sure whether
  jobmanager.adaptive-scheduler.max-delay-for-rescale-trigger is a config
option
 or an internal logic? I saw it's computed by
rescale-on-failed-checkpoints-count.
3. I'm not sure if the default value of rescale-on-failed-checkpoints-count
should
   be 1 or is greater than 1 better?
   If 1 as the default value, when the checkpoint fails occasionally, and
rescale happens,
   flink job will process a series of repeated data as well.
   If 2 as the default value, when the checkpoint fails occasionally, and
the next
   checkpoint succeeds, the flink job won't process repeated data.
4. The description of rescale-on-failed-checkpoints-count is
  "The number of subsequent failed checkpoints that will initiate
rescaling."
  IIUC, the "consecutive" is more accurate than subsequent here. WDYT?
5. Proposed Changes part is specific implementation, I'm not sure whether
   all internal interfaces are best for the current version. So I cannot
give
  any suggestion or feedback for now. But I'm happy to review them when
  your PR is ready if I have time.
  Feel free to cc me (I'm interested in Adaptive Scheduler)
6. This proposal aims to improve one logic inside of Adaptive Scheduler.
   Would you mind mentioning Adaptive Scheduler in the FLIP title? It will
   be useful for users to understand which component this proposal belongs
to.

Also, I also don't understand why this proposal needs to care about the
checkpoint type is unaligned checkpoint or aligned checkpoint.

Please correct me if anything is wrong, thanks.

Best,
Rui

On Wed, Jun 5, 2024 at 3:01 PM Matthias Pohl  wrote:

> Hi ConradJam,
> thanks for your response.
>
> The CheckpointStatsTracker gets notified about the checkpoint completion
> after the checkpoint is finalized, i.e. all its data is persisted and the
> metadata is written to the CompletedCheckpointStore. At this moment, the
> checkpoint is considered for restoring a job and, therefore, becomes
> available for restarts. This workflow also applies to unaligned
> checkpoints. But I see how this context might be helpful for understanding
> the change. I will add it to the FLIP. So far, I don't see a reason
> to disable the feature for unaligned checkpoints. Do you see other issues
> that might make it necessary to disable this feature for this type of
> checkpoints?
>
> Can you elaborate a bit more what you mean by "checkpoints that do not
> check it"? I do not fully understand what you are referring to with "it"
> here.
>
> Best,
> Matthias
>
> On Wed, Jun 5, 2024 at 4:46 AM ConradJam  wrote:
>
> > I have a few questions:
> > Unaligned checkpoints Do we need to enable this feature? Whether this
> > feature should be disabled for checkpoints that do not check it
> >
> > Matthias Pohl  于2024年6月4日周二 18:03写道：
> >
> > > Hi everyone,
> > > I'd like to discuss FLIP-461 [1]. The FLIP proposes the synchronization
> > of
> > > rescaling and the completion of checkpoints. The idea is to reduce the
> > > amount of data that needs to be processed after rescaling happened. A
> > more
> > > detailed motivation can be found in FLIP-461.
> > >
> > > I'm looking forward to feedback and suggestions.
> > >
> > > Best,
> > > Matthias
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-461%3A+Synchronize+rescaling+with+checkpoint+creation+to+minimize+reprocessing
> > >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

[Rui Fan]

Congratulations Weijie, well deserved!

Best,
Rui

On Tue, Jun 4, 2024 at 2:47 PM Xintong Song  wrote:

> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce that Weijie Guo has joined
> the Flink PMC!
>
> Weijie has been an active member of the Apache Flink community for many
> years. He has made significant contributions in many components, including
> runtime, shuffle, sdk, connectors, etc. He has driven / participated in
> many FLIPs, authored and reviewed hundreds of PRs, been consistently active
> on mailing lists, and also helped with release management of 1.20 and
> several other bugfix releases.
>
> Congratulations and welcome Weijie!
>
> Best,
>
> Xintong (on behalf of the Flink PMC)
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Rui Fan]

Thanks Yuxin for driving this proposal!

Kindly reminder: the image of CIP-6[1] cannot be loaded.

[1]
https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn

Best,
Rui

On Wed, May 29, 2024 at 9:03 AM Xintong Song  wrote:

> +1 for this proposal.
>
> We have been prototyping this feature internally at Alibaba for a couple of
> months. Yuxin, I think we can also publish the prototype codes so the
> community can better understand and help with it.
>
> Best,
>
> Xintong
>
>
>
> On Tue, May 28, 2024 at 8:34 PM Yuxin Tan  wrote:
>
> > Hi all,
> >
> > I would like to start a discussion on FLIP-459 Support Flink hybrid
> shuffle
> > integration with
> > Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> > memory, disk, and
> > remote storage to improve performance and job stability. Concurrently,
> > Apache Celeborn
> > provides a stable, performant, scalable remote shuffle service. This
> > integration proposal is to
> > harness the benefits from both hybrid shuffle and Celeborn
> simultaneously.
> >
> > Note that this proposal has two parts.
> > 1. The Flink-side modifications are in FLIP-459[1].
> > 2. The Celeborn-side changes are in CIP-6[2].
> >
> > Looking forward to everyone's feedback and suggestions. Thank you!
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> >
> > Best,
> > Yuxin
> >
>

[jira] [Created] (FLINK-35471) Kubernetes operator bumps the flink version to 1.19

[Rui Fan (Jira)]

Rui Fan created FLINK-35471:
---

 Summary: Kubernetes operator bumps the flink version to 1.19
 Key: FLINK-35471
 URL: https://issues.apache.org/jira/browse/FLINK-35471
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.9.0


Kubernetes operator bumps the flink version to 1.19 after 1.19.1 is released.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] The performance of serializerHeavyString starts regress since April 3

[Rui Fan]

Thanks Sam for the comment!

> It looks like the most recent run of JDK 11 saw a big improvement of the
> performance of the test.

I guess that improvement is a fluctuation. You can double check the
performance results[1] of the last few days. The performance isn't
recovered.

> That improvement seems related to which is a fix for FLINK-35215.

I fixed an issue related to kryo serialization in FLINK-35215. IIUC,
serializerHeavyString doesn't use the kryo serialization. I try to
run serializerHeavyString demo locally, and didn't see the
kryo serialization related code is called.

Please correct me if I'm wrong, thanks~

[1]
http://flink-speed.xyz/timeline/#/?exe=6=serializerHeavyString=on=on=off=3=200

Best,
Rui

On Thu, May 23, 2024 at 1:27 PM Sam Barker  wrote:

> It looks like the most recent run of JDK 11 saw a big improvement[1] of the
> performance of the test. That improvement seems related to [2] which is a
> fix for FLINK-35215 [3]. That suggests to me that the test isn't as
> isolated to the performance of the code its trying to test as would be
> ideal. However I've only just started looking at the test suite and trying
> to run locally so I'm not very well placed to judge.
>
> It does however suggest that this shouldn't be a blocker for the release.
>
>
>
> [1] http://flink-speed.xyz/changes/?rev=c1baf07d76=6=3
> [2]
>
> https://github.com/apache/flink/commit/c1baf07d7601a683f42997dc35dfaef4e41bc928
> [3] https://issues.apache.org/jira/browse/FLINK-35215
>
> On Wed, 22 May 2024 at 00:15, Piotr Nowojski  wrote:
>
> > Hi,
> >
> > Given what you wrote, that you have investigated the issue and couldn't
> > find any easy explanation, I would suggest closing this ticket as "Won't
> > do" or "Can not reproduce" and ignoring the problem.
> >
> > In the past there have been quite a bit of cases where some benchmark
> > detected a performance regression. Sometimes those can not be reproduced,
> > other times (as it's the case here), some seemingly unrelated change is
> > causing the regression. The same thing happened in this benchmark many
> > times in the past [1], [2], [3], [4]. Generally speaking this benchmark
> has
> > been in the spotlight a couple of times [5].
> >
> > Note that there have been cases where this benchmark did detect a
> > performance regression :)
> >
> > My personal suspicion is that after that commons-io version bump,
> > something poked JVM/JIT to compile the code a bit differently for string
> > serialization causing this regression. We have a couple of benchmarks
> that
> > seem to be prone to such semi intermittent issues. For example the same
> > benchmark was subject to this annoying pattern [6], that I've spotted in
> > quite a bit of benchmarks over the years [6]:
> >
> > [image: image.png]
> > (https://imgur.com/a/AoygmWS)
> >
> > Where benchmark results are very stable within a single JVM fork. But
> > between two forks, they can reach two different "stable" levels. Here it
> > looks like 50% of the chance of getting stable "200 records/ms" and 50%
> > chances of "250 records/ms".
> >
> > A small interlude. Each of our benchmarks run in 3 different JVM forks,
> 10
> > warm up iterations and 10 measurement iterations. Each iteration
> > lasts/invokes the benchmarking method at least for one second. So by
> "very
> > stable" results, I mean that for example after the 2nd or 3rd warm up
> > iteration, the results stabilize < +/-1%, and stay on that level for the
> > whole duration of the fork.
> >
> > Given that we are repeating the same benchmark in 3 different forks, we
> > can have by pure chance:
> > - 3 slow fork - total average 200 records/ms
> > - 2 slow fork, 1 fast fork - average 216 r/ms
> > - 1 slow fork, 2 fast forks - average 233 r/ms
> > - 3 fast forks - average 250 r/ms
> >
> > So this benchmark is susceptible to enter some different semi stable
> > states. As I wrote above, I guess something with the commons-io version
> > bump just swayed it to a different semi stable state :( I have never
> gotten
> > desperate enough to actually dig further what's exactly causing this kind
> > of issues.
> >
> > Best,
> > Piotrek
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18684
> > [2] https://issues.apache.org/jira/browse/FLINK-27133
> > [3] https://issues.apache.org/jira/browse/FLINK-27165
> > [4] https://issues.apache.org/jira/browse/FLINK-31745
> > [5]
> >
> https://issues.apache.org/jira/browse/FLINK-35040?jql=project%20%3D%20FLINK%20AND%20status%20in%

Re: [DISCUSS] Flink 1.19.1 release

[Rui Fan]

+1 for release 1.19.1 and Hong as release manager.

We are really looking forward to this version, thank you.

Best,
Rui

On Mon, May 27, 2024 at 11:05 AM Xintong Song  wrote:

> +1 for the 1.19.1 release and +1 for Hong as the release manager.
>
> Hong, if you need help on PMC-related steps and Danny is not available,
> please feel free to reach out to me on slack.
>
> Best,
>
> Xintong
>
>
>
> On Mon, May 27, 2024 at 9:44 AM Leonard Xu  wrote:
>
> > +1 for the 1.19.1 release and +1 for Hong as release manager.
> >
> > Best,
> > Leonard
> >
> > > 2024年5月25日 上午2:55，Danny Cranmer  写道：
> > >
> > > +1 for the 1.19.1 release and +1 for Hong as release manager.
> >
> >
>

Re: [VOTE] FLIP-457: Improve Table/SQL Configuration for Flink 2.0

[Rui Fan]

+1(binding)

Best,
Rui

On Fri, May 24, 2024 at 1:45 PM Leonard Xu  wrote:

> +1
>
> Best,
> Leonard
>
> > 2024年5月24日 下午1:27，weijie guo  写道：
> >
> > +1(binding)
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Lincoln Lee  于2024年5月24日周五 12:20写道：
> >
> >> +1(binding)
> >>
> >> Best,
> >> Lincoln Lee
> >>
> >>
> >> Jane Chan  于2024年5月24日周五 09:52写道：
> >>
> >>> Hi all,
> >>>
> >>> I'd like to start a vote on FLIP-457[1] after reaching a consensus
> >> through
> >>> the discussion thread[2].
> >>>
> >>> The vote will be open for at least 72 hours unless there is an
> objection
> >> or
> >>> insufficient votes.
> >>>
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=307136992
> >>> [2] https://lists.apache.org/thread/1sthbv6q00sq52pp04n2p26d70w4fqj1
> >>>
> >>> Best,
> >>> Jane
> >>>
> >>
>
>

Re: [VOTE] FLIP-443: Interruptible watermark processing

[Rui Fan]

+1(binding)

Best,
Rui

On Fri, May 24, 2024 at 12:01 PM Yanfei Lei  wrote:

> Thanks for driving this!
>
> +1 (binding)
>
> Best,
> Yanfei
>
> Zakelly Lan  于2024年5月24日周五 10:13写道：
>
> >
> > +1 (binding)
> >
> > Best,
> > Zakelly
> >
> > On Thu, May 23, 2024 at 8:21 PM Piotr Nowojski 
> wrote:
> >
> > > Hi all,
> > >
> > > After reaching what looks like a consensus in the discussion thread
> [1], I
> > > would like to put FLIP-443 [2] to the vote.
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection or
> > > insufficient votes.
> > >
> > > [1] https://lists.apache.org/thread/flxm7rphvfgqdn2gq2z0bb7kl007olpz
> > > [2] https://cwiki.apache.org/confluence/x/qgn9EQ
> > >
> > > Bets,
> > > Piotrek
> > >
>

Re: [SUMMARY] Flink 1.20 Release Sync 05/21/2024

[Rui Fan]

Thanks Weijie for the summary!

Best,
Rui

On Wed, May 22, 2024 at 2:52 PM weijie guo 
wrote:

> Dear devs,
>
>
> This is the fourth meeting for Flink 1.20 release cycle.
>
>
> I'd like to share the information synced in the meeting.
>
>
>
> - Feature Freeze
>
>
>   It is worth noting that there are only 4 weeks left until the feature 
> freeze time(June 15, 2024),
> and developers need to pay attention to the feature freeze time.
>
>
>
> - Features:
>
>
>   So far we've had 10 flips/features, there are two FLIPs that
> may not be completed in 1.20, and the status of the other
> eight FLIPs is good.
>
>   It is encouraged to continuously updating
> the 1.20 wiki page[1] for contributors.
>
>
>
> - Blockers:
>
>
>   - [Closed] FLINK-35041
> IncrementalRemoteKeyedStateHandleTest.testSharedStateReRegistration failed
>
>
>
>   - [Doing] FLINK-35040 The performance of serializerHeavyString regresses 
> since April 3
>
>
> - we have started a discussion[2] about this to dev mail list to collect 
> some valuable suggestion. We will accept this regression if we don't have any 
> suggestion.
>
>
>
>   - [In Verification] FLINK-35215 The performance of serializerKryo and 
> serializerKryoWithoutRegistration are regressed
>
> - Rui provided an alternative change without performance regression, we 
> will follow the benchmark result in the following days, and close this JIRA
> after the performance is recovered.
>
>
>
> - Sync meeting[3]:
>
>
>
>  The next meeting is 06/04/2024 10am (UTC+2) and 4pm (UTC+8), please feel 
> free to join us. After this, the release sync will be adjusted to weekly as 
> we approaching the feature freeze date.
>
>
>  Lastly, we encourage attendees to fill out the topics to be discussed at
> the bottom of 1.20 wiki page[1] a day in advance, to make it easier for
> everyone to understand the background of the topics, thanks!
>
>
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/1.20+Release
>
> [2] https://lists.apache.org/thread/moo2b67qk77ysoofs9j1ojzjk0rhrr9h
>
> [3] https://meet.google.com/mtj-huez-apu
>
>
>
> Best,
>
> Robert, Rui, Ufuk, Weijie
>

Re: [DISCUSS] The performance of serializerHeavyString starts regress since April 3

[Rui Fan]

 gotten
>> desperate enough to actually dig further what's exactly causing this kind
>> of issues.
>>
>> Best,
>> Piotrek
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-18684
>> [2] https://issues.apache.org/jira/browse/FLINK-27133
>> [3] https://issues.apache.org/jira/browse/FLINK-27165
>> [4] https://issues.apache.org/jira/browse/FLINK-31745
>> [5]
>> https://issues.apache.org/jira/browse/FLINK-35040?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20Resolved%2C%20Closed)%20AND%20text%20~%20%22serializerHeavyString%22
>> [6]
>> http://flink-speed.xyz/timeline/#/?exe=1=serializerHeavyString=on=on=off=2=1000
>>
>> wt., 21 maj 2024 o 12:50 Rui Fan <1996fan...@gmail.com> napisał(a):
>>
>>> Hi devs:
>>>
>>> We(release managers of flink 1.20) wanna update one performance
>>> regresses to the flink dev mail list.
>>>
>>> # Background:
>>>
>>> The performance of serializerHeavyString starts regress since April 3,
>>> and we created FLINK-35040[1] to follow it.
>>>
>>> In brief:
>>> - The performance only regresses for jdk 11, and Java 8 and Java 17 are
>>> fine.
>>> - The regression reason is upgrading commons-io version from 2.11.0 to
>>> 2.15.1
>>>   - This upgrading is done in FLINK-34955[2].
>>>   - The performance can be recovered after reverting the commons-io
>>> version
>>> to 2.11.0
>>>
>>> You can get more details from FLINK-35040[1].
>>>
>>> # Problem
>>>
>>> We try to generate the flame graph (wall mode) to analyze why upgrading
>>> the commons-io version affects the performance. These flamegraphs can
>>> be found in FLINK-35040[1]. (Many thanks to Zakelly for generating these
>>> flamegraphs from the benchmark server).
>>>
>>> Unfortunately, we cannot find any code of commons-io dependency is
>>> called.
>>> Also, we try to analyze if any other dependencies are changed during
>>> upgrading
>>> commons-io version. The result is no, other dependencies are totally the
>>> same.
>>>
>>> # Request
>>>
>>> After the above analysis, we cannot find why the performance of
>>> serializerHeavyString
>>> starts to regress for jdk11.
>>>
>>> We are looking forward to hearing valuable suggestions from the Flink
>>> community.
>>> Thanks everyone in advance.
>>>
>>> Note:
>>> 1. I cannot reproduce the regression on my Mac with jdk11, and we suspect
>>>   this regression may be caused by the benchmark environment.
>>> 2. We will accept this regression if the issue still cannot be solved.
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-35040
>>> [2] https://issues.apache.org/jira/browse/FLINK-34955
>>>
>>> Best,
>>> Weijie, Ufuk, Robert and Rui
>>>
>>

[DISCUSS] The performance of serializerHeavyString starts regress since April 3

[Rui Fan]

Hi devs:

We(release managers of flink 1.20) wanna update one performance
regresses to the flink dev mail list.

# Background:

The performance of serializerHeavyString starts regress since April 3,
and we created FLINK-35040[1] to follow it.

In brief:
- The performance only regresses for jdk 11, and Java 8 and Java 17 are
fine.
- The regression reason is upgrading commons-io version from 2.11.0 to
2.15.1
  - This upgrading is done in FLINK-34955[2].
  - The performance can be recovered after reverting the commons-io version
to 2.11.0

You can get more details from FLINK-35040[1].

# Problem

We try to generate the flame graph (wall mode) to analyze why upgrading
the commons-io version affects the performance. These flamegraphs can
be found in FLINK-35040[1]. (Many thanks to Zakelly for generating these
flamegraphs from the benchmark server).

Unfortunately, we cannot find any code of commons-io dependency is called.
Also, we try to analyze if any other dependencies are changed during
upgrading
commons-io version. The result is no, other dependencies are totally the
same.

# Request

After the above analysis, we cannot find why the performance of
serializerHeavyString
starts to regress for jdk11.

We are looking forward to hearing valuable suggestions from the Flink
community.
Thanks everyone in advance.

Note:
1. I cannot reproduce the regression on my Mac with jdk11, and we suspect
  this regression may be caused by the benchmark environment.
2. We will accept this regression if the issue still cannot be solved.

[1] https://issues.apache.org/jira/browse/FLINK-35040
[2] https://issues.apache.org/jira/browse/FLINK-34955

Best,
Weijie, Ufuk, Robert and Rui

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

[Rui Fan]

Hi Piotr,

> we checked in the firing timers benchmark [1] and we didn't observe any
> performance regression.

Thanks for the feedback, it's good news to hear that. I didn't notice
we already have fireProcessingTimers benchmark.

If so, we can follow it after this FLIP is merged.

+1 for this FLIP.

Best,
Rui

On Thu, May 16, 2024 at 5:13 PM Piotr Nowojski  wrote:

> Hi Zakelly,
>
> > I'm suggesting skipping the continuation mail during draining of async
> state access.
>
> I see. That makes sense to me now. I will later update the FLIP.
>
> > the code path will become more complex after this FLIP
> due to the addition of shouldIntterupt() checks, right?
>
> Yes, that's correct.
>
> > If so, it's better to add a benchmark to check whether the job
> > performance regresses when one job has a lot of timers.
> > If the performance regresses too much, we need to re-consider it.
> > Of course, I hope the performance is fine.
>
> I had the same concerns when initially David Moravek proposed this
> solution,
> but we checked in the firing timers benchmark [1] and we didn't observe any
> performance regression.
>
> Best,
> Piotrek
>
> [1] http://flink-speed.xyz/timeline/?ben=fireProcessingTimers=3
>
>
>
> wt., 7 maj 2024 o 09:47 Rui Fan <1996fan...@gmail.com> napisał(a):
>
> > Hi Piotr,
> >
> > Overall this FLIP is fine for me. I have a minor concern:
> > IIUC, the code path will become more complex after this FLIP
> > due to the addition of shouldIntterupt() checks, right?
> >
> > If so, it's better to add a benchmark to check whether the job
> > performance regresses when one job has a lot of timers.
> > If the performance regresses too much, we need to re-consider it.
> > Of course, I hope the performance is fine.
> >
> > Best,
> > Rui
> >
> > On Mon, May 6, 2024 at 6:30 PM Zakelly Lan 
> wrote:
> >
> > > Hi Piotr,
> > >
> > > I'm saying the scenario where things happen in the following order:
> > > 1. advance watermark and process timers.
> > > 2. the cp arrives and interrupts the timer processing, after this the
> > > continuation mail is in the mailbox queue.
> > > 3. `snapshotState` is called, where the async state access responses
> will
> > > be drained by calling `tryYield()` [1]. —— What if the continuation
> mail
> > is
> > > triggered by `tryYield()`?
> > >
> > > I'm suggesting skipping the continuation mail during draining of async
> > > state access.
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/flink/blob/1904b215e36e4fd48e48ece7ffdf2f1470653130/flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java#L305
> > >
> > > Best,
> > > Zakelly
> > >
> > >
> > > On Mon, May 6, 2024 at 6:00 PM Piotr Nowojski 
> > > wrote:
> > >
> > > > Hi Zakelly,
> > > >
> > > > Can you elaborate a bit more on what you have in mind? How marking
> > mails
> > > as
> > > > interruptible helps with something? If an incoming async state access
> > > > response comes, it could just request to interrupt any currently
> > ongoing
> > > > computations, regardless the currently executed mail is or is not
> > > > interruptible.
> > > >
> > > > Best,
> > > > Piotrek
> > > >
> > > > pon., 6 maj 2024 o 06:33 Zakelly Lan 
> > napisał(a):
> > > >
> > > > > Hi Piotr,
> > > > >
> > > > > Thanks for the improvement, overall +1 for this. I'd leave a minor
> > > > comment:
> > > > >
> > > > > 1. I'd suggest also providing `isInterruptable()` in `Mail`, and
> the
> > > > > continuation mail will return true. The FLIP-425 will leverage this
> > > queue
> > > > > to execute some state requests, and when the cp arrives, the
> operator
> > > may
> > > > > call `yield()` to drain. It may happen that the continuation mail
> is
> > > > called
> > > > > again in `yield()`. By checking `isInterruptable()`, we can skip
> this
> > > > mail
> > > > > and re-enqueue.
> > > > >
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > On Wed, May 1, 2024 at 4:35 PM Yanfei Lei 
> > wrote:
> > > > >

Re: [RESULT][VOTE] FLIP-449: Reorganization of flink-connector-jdbc

[Rui Fan]

Thanks Joao Boto for driving the FLIP. We need 3 +1(binding)
votes according to Flink Bylaws[1] before the community accepts it.

You can search "Consensus" and "FLIP (Major Change)" keywords
in this wiki.

[1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws

Best,
Rui

On Thu, May 16, 2024 at 2:16 AM Joao Boto  wrote:

> Hi,
>
> I am happy to say that FLIP-449: Reorganization of flink-connector-jdbc [1]
> has been accepted.
>
> The proposal has been accepted with 5 approving votes (1 binding) and there
> are no disapproval (voted on this thread [2]):
> Rui Fan (binding)
> Yuepeng Pan (non-binding)
> Aleksandr Pilipenko (non-binding)
> Muhammet Orazov (non-binding)
> Jeyhun Karimov (non-binding)
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
> [2] https://lists.apache.org/thread/mfqx711zoxgs3sbojr2slqrt2xv2h5q9
>
> Thanks to all involved.
>
> Best,
> Joao Boto
>

Re: [VOTE] FLIP-450: Improve Runtime Configuration for Flink 2.0

[Rui Fan]

+1(binding)

Best,
Rui

On Wed, May 15, 2024 at 5:01 PM Xuannan Su  wrote:

> Hi everyone,
>
> Thanks for all the feedback about the FLIP-450: Improve Runtime
> Configuration for Flink 2.0 [1] [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours(excluding weekends,until MAY 20, 12:00AM GMT) unless there is an
> objection or an insufficient number of votes.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0
> [2] https://lists.apache.org/thread/20mkzd31607posls793hxy7mht40xp2x
>
>
> Best regards,
> Xuannan
>

Re: [VOTE] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction

[Rui Fan]

Thanks Martijn for driving this proposal!

+1(binding)

Best,
Rui

On Tue, May 14, 2024 at 3:30 PM Ferenc Csaky 
wrote:

> +1 (non-binding)
>
> Thanks,
> Ferenc
>
>
>
>
> On Tuesday, 14 May 2024 at 08:51, weijie guo 
> wrote:
>
> >
> >
> > Thanks Martijn for the effort!
> >
> > +1(binding)
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Martijn Visser martijnvis...@apache.org 于2024年5月14日周二 14:45写道：
> >
> > > Hi everyone,
> > >
> > > With no more discussions being open in the thread [1] I would like to
> start
> > > a vote on FLIP-453: Promote Unified Sink API V2 to Public and Deprecate
> > > SinkFunction [2]
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection or
> > > insufficient votes.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > [1] https://lists.apache.org/thread/hod6bg421bzwhbfv60lwsck7r81dvo59
> > > [2]
> > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-453%3A+Promote+Unified+Sink+API+V2+to+Public+and+Deprecate+SinkFunction
>

Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0

[Rui Fan]

Thanks Xuannan for the update!

LGTM, +1 for this proposal.

Best,
Rui

On Sat, May 11, 2024 at 10:20 AM Xuannan Su  wrote:

> Hi Rui,
>
> Thanks for the suggestion!
>
> I updated the description of
> taskmanager.network.memory.max-overdraft-buffers-per-gate and
> hard-coded it to 20.
>
> Best regards,
> Xuannan
>
> On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Thanks Xuannan for driving this proposal!
> >
> > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be
> removed
> > and hard-coded to either 10 or 20.
> >
> > Currently, it's a public option. Could we determine the value of
> > the overdraft buffer in the current FLIP?
> >
> > I vote 20 as the hard code value due to 2 reasons:
> > - Removing this option means users cannot change it, it might be better
> to
> > turn it up.
> > - Most of tasks don't use the overdraft buffer, so increasing it doesn't
> > introduce more risk.
> >
> > Best,
> > Rui
> >
> > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan 
> wrote:
> >
> > > Thanks for the effort, Xuannan.
> > >
> > > +1 for the proposal.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Xintong Song  于2024年4月29日周一 15:40写道：
> > >
> > > > Thanks for driving this effort, Xuannan.
> > > >
> > > > +1 for the proposed changes.
> > > >
> > > > Just one suggestion: Some of the proposed changes involve not solely
> > > > changing the configuration options, but are bound to changing /
> removal
> > > of
> > > > certain features. E.g., the removal of hash-blocking shuffle and
> legacy
> > > > hybrid shuffle mode, and the behavior change of overdraft network
> > > buffers.
> > > > Therefore, it might be nicer to provide an implementation plan with a
> > > list
> > > > of related tasks in the FLIP. This should not block the FLIP though.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su 
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start a discussion on FLIP-450: Improve Runtime
> > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have
> > > > > revisited all runtime configurations and identified several
> > > > > improvements to enhance user-friendliness and maintainability. In
> this
> > > > > FLIP, we aim to refine the runtime configuration.
> > > > >
> > > > > Looking forward to everyone's feedback and suggestions. Thank you!
> > > > >
> > > > > Best regards,
> > > > > Xuannan
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0
> > > > >
> > > >
> > >
>

Re: [VOTE] FLIP-449: Reorganization of flink-connector-jdbc

[Rui Fan]

Thanks for driving this proposal!

+1(binding)

Best,
Rui

On Fri, May 10, 2024 at 9:50 AM Yuepeng Pan  wrote:

> Hi, Boto.
>
> +1 (non-binding).
>
> Thanks for your driving it.
>
> Best regards,
> Yuepeng Pan.
>
> On 2024/05/09 12:01:04 Joao Boto wrote:
> > Hi everyone,
> >
> > Thanks for all the feedback, I'd like to start a vote on the FLIP-449:
> > Reorganization of flink-connector-jdbc [1].
> > The discussion thread is here [2].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> or
> > insufficient votes.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
> > [2] https://lists.apache.org/thread/jc1yvvo35xwqzlxl5mj77qw3hq6f5sgr
> >
> > Best
> > Joao Boto
> >
>

[jira] [Created] (FLINK-35315) MemoryManagerConcurrentModReleaseTest executes more than 15 minutes

[Rui Fan (Jira)]

Rui Fan created FLINK-35315:
---

 Summary: MemoryManagerConcurrentModReleaseTest executes more than 
15 minutes
 Key: FLINK-35315
 URL: https://issues.apache.org/jira/browse/FLINK-35315
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network, Tests
Affects Versions: 1.20.0
Reporter: Rui Fan
 Attachments: image-2024-05-09-11-53-10-037.png

[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59395=results]

 

It seems 

MemoryManagerConcurrentModReleaseTest.testConcurrentModificationWhileReleasing 
executes more than 15 minutes.

The root cause may be {color:#e1dfdd}ConcurrentModificationException{color}

 

[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59395=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=10060]

 

!image-2024-05-09-11-53-10-037.png!

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35306) Flink cannot compile with jdk17

[Rui Fan (Jira)]

Rui Fan created FLINK-35306:
---

 Summary: Flink cannot compile with jdk17
 Key: FLINK-35306
 URL: https://issues.apache.org/jira/browse/FLINK-35306
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.20.0
 Attachments: image-2024-05-08-11-48-04-161.png

!image-2024-05-08-11-48-04-161.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[SUMMARY] Flink 1.20 Release Sync 05/07/2024

[Rui Fan]

Dear devs,

Today is the third meeting for Flink 1.20 release cycle.
I'd like to share the information synced in the meeting.

- Feature Freeze

  As we discussed before, the feature freeze time is
  June 15, 2024, 00:00 CEST(UTC+2). It is worth noting that
  there are only 6 weeks left until the feature freeze time,
  and developers need to pay attention to the feature freeze time.

  Also, if there are no other changes, 1.20 will be the last version
  before 2.0. Relevant developers also need to check whether
  the Public or Public Evolving API related to Flink 2.0 deprecation
  has been completed.

- Features:
  So far we've had 8 flips/features, there are two FLIPs that
  may not be completed in 1.20, and the status of the other
  six FLIPs is good. It is encouraged to continuously updating
  the 1.20 wiki page[1] for contributors.

- Blockers:
  - [Closed] FLINK-34997 PyFlink YARN per-job on Docker test failed on azure
  - [Doing] FLINK-35041
IncrementalRemoteKeyedStateHandleTest.testSharedStateReRegistration failed
- It fails frequently, Rui has pinged @feifan, he will check it
this week with high priority
  - [Doing] FLINK-35040 The performance of serializerHeavyString regresses
since April 3
- Rui needs to generate a flamegraph from the benchmark server.
  - [Doing] FLINK-35215 The performance of serializerKryo and
serializerKryoWithoutRegistration are regressed
- Rui provided an alternative change without performance regression, we
need someone who familiar with kryo serialization to review

- Sync meeting[2]:
 The next meeting is 05/21/2024 10am (UTC+2) and 4pm (UTC+8), please feel
free to join us.

Lastly, we encourage attendees to fill out the topics to be discussed at
the bottom of 1.20 wiki page[1] a day in advance, to make it easier for
everyone to understand the background of the topics, thanks!

[1] https://cwiki.apache.org/confluence/display/FLINK/1.20+Release
[2] https://meet.google.com/mtj-huez-apu

Best,

Weijie, Ufuk, Robert and Rui

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

[Rui Fan]

Hi Piotr,

Overall this FLIP is fine for me. I have a minor concern:
IIUC, the code path will become more complex after this FLIP
due to the addition of shouldIntterupt() checks, right?

If so, it's better to add a benchmark to check whether the job
performance regresses when one job has a lot of timers.
If the performance regresses too much, we need to re-consider it.
Of course, I hope the performance is fine.

Best,
Rui

On Mon, May 6, 2024 at 6:30 PM Zakelly Lan  wrote:

> Hi Piotr,
>
> I'm saying the scenario where things happen in the following order:
> 1. advance watermark and process timers.
> 2. the cp arrives and interrupts the timer processing, after this the
> continuation mail is in the mailbox queue.
> 3. `snapshotState` is called, where the async state access responses will
> be drained by calling `tryYield()` [1]. —— What if the continuation mail is
> triggered by `tryYield()`?
>
> I'm suggesting skipping the continuation mail during draining of async
> state access.
>
>
> [1]
>
> https://github.com/apache/flink/blob/1904b215e36e4fd48e48ece7ffdf2f1470653130/flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java#L305
>
> Best,
> Zakelly
>
>
> On Mon, May 6, 2024 at 6:00 PM Piotr Nowojski 
> wrote:
>
> > Hi Zakelly,
> >
> > Can you elaborate a bit more on what you have in mind? How marking mails
> as
> > interruptible helps with something? If an incoming async state access
> > response comes, it could just request to interrupt any currently ongoing
> > computations, regardless the currently executed mail is or is not
> > interruptible.
> >
> > Best,
> > Piotrek
> >
> > pon., 6 maj 2024 o 06:33 Zakelly Lan  napisał(a):
> >
> > > Hi Piotr,
> > >
> > > Thanks for the improvement, overall +1 for this. I'd leave a minor
> > comment:
> > >
> > > 1. I'd suggest also providing `isInterruptable()` in `Mail`, and the
> > > continuation mail will return true. The FLIP-425 will leverage this
> queue
> > > to execute some state requests, and when the cp arrives, the operator
> may
> > > call `yield()` to drain. It may happen that the continuation mail is
> > called
> > > again in `yield()`. By checking `isInterruptable()`, we can skip this
> > mail
> > > and re-enqueue.
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Wed, May 1, 2024 at 4:35 PM Yanfei Lei  wrote:
> > >
> > > > Thanks for your answers, Piotrek. I got it now.  +1 for this
> > improvement.
> > > >
> > > > Best,
> > > > Yanfei
> > > >
> > > > Stefan Richter  于2024年4月30日周二
> 21:30写道：
> > > > >
> > > > >
> > > > > Thanks for the improvement proposal, I’m +1 for the change!
> > > > >
> > > > > Best,
> > > > > Stefan
> > > > >
> > > > >
> > > > >
> > > > > > On 30. Apr 2024, at 15:23, Roman Khachatryan 
> > > wrote:
> > > > > >
> > > > > > Thanks for the proposal, I definitely see the need for this
> > > > improvement, +1.
> > > > > >
> > > > > > Regards,
> > > > > > Roman
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 30, 2024 at 3:11 PM Piotr Nowojski <
> > pnowoj...@apache.org
> > > > > wrote:
> > > > > >
> > > > > >> Hi Yanfei,
> > > > > >>
> > > > > >> Thanks for the feedback!
> > > > > >>
> > > > > >>> 1. Currently when AbstractStreamOperator or
> > > AbstractStreamOperatorV2
> > > > > >>> processes a watermark, the watermark will be sent to
> downstream,
> > if
> > > > > >>> the `InternalTimerServiceImpl#advanceWatermark` is interrupted,
> > > when
> > > > > >>> is the watermark sent downstream?
> > > > > >>
> > > > > >> The watermark would be outputted by an operator only once all
> > > relevant
> > > > > >> timers are fired.
> > > > > >> In other words, if firing of timers is interrupted a
> continuation
> > > > mail to
> > > > > >> continue firing those
> > > > > >> interrupted timers is created. Watermark will be emitted
> > downstream
> > > > at the
> > > > > >> end of that
> > > > > >> continuation mail.
> > > > > >>
> > > > > >>> 2. IIUC, processing-timer's firing is also encapsulated into
> mail
> > > and
> > > > > >>> executed in mailbox. Is processing-timer allowed to be
> > interrupted?
> > > > > >>
> > > > > >> Yes, both firing processing and even time timers share the same
> > code
> > > > and
> > > > > >> both will
> > > > > >> support interruptions in the same way. Actually I've renamed the
> > > FLIP
> > > > from
> > > > > >>
> > > > > >>> Interruptible watermarks processing
> > > > > >>
> > > > > >> to:
> > > > > >>
> > > > > >>> Interruptible timers firing
> > > > > >>
> > > > > >> to make this more clear.
> > > > > >>
> > > > > >> Best,
> > > > > >> Piotrek
> > > > > >>
> > > > > >> wt., 30 kwi 2024 o 06:08 Yanfei Lei 
> > > napisał(a):
> > > > > >>
> > > > > >>> Hi Piotrek,
> > > > > >>>
> > > > > >>> Thanks for this proposal. It looks like it will shorten the
> > > > checkpoint
> > > > > >>> duration, especially in the case of back pressure. +1 for it!
> > I'd
> > > > > >>> like to ask some questions to understand your thoughts more
> > > > 

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

[Rui Fan]

+1 (binding)

Best,
Rui

On Mon, May 6, 2024 at 11:01 AM Yanfei Lei  wrote:

> +1 (binding)
>
> Best,
> Yanfei
>
> Zakelly Lan  于2024年5月6日周一 11:00写道：
> >
> > +1 (binding)
> >
> > Thanks for driving this!
> >
> >
> > Best,
> > Zakelly
> >
> > On Mon, May 6, 2024 at 10:54 AM yue ma  wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for all the feedback, I'd like to start a vote on the FLIP-447:
> > > Upgrade FRocksDB from 6.20.3 to 8.10.0 [1]. The discussion thread is
> here
> > > [2].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection or
> > > insufficient votes.
> > >
> > > [1]
> > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0
> > > [2] https://lists.apache.org/thread/lrxjfpjjwlq4sjzm1oolx58n1n8r48hw
> > >
> > > --
> > > Best,
> > > Yue
> > >
>

Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0

[Rui Fan]

Thanks Xuannan for driving this proposal!

> taskmanager.network.memory.max-overdraft-buffers-per-gate will be removed
and hard-coded to either 10 or 20.

Currently, it's a public option. Could we determine the value of
the overdraft buffer in the current FLIP?

I vote 20 as the hard code value due to 2 reasons:
- Removing this option means users cannot change it, it might be better to
turn it up.
- Most of tasks don't use the overdraft buffer, so increasing it doesn't
introduce more risk.

Best,
Rui

On Mon, May 6, 2024 at 10:47 AM Yuxin Tan  wrote:

> Thanks for the effort, Xuannan.
>
> +1 for the proposal.
>
> Best,
> Yuxin
>
>
> Xintong Song  于2024年4月29日周一 15:40写道：
>
> > Thanks for driving this effort, Xuannan.
> >
> > +1 for the proposed changes.
> >
> > Just one suggestion: Some of the proposed changes involve not solely
> > changing the configuration options, but are bound to changing / removal
> of
> > certain features. E.g., the removal of hash-blocking shuffle and legacy
> > hybrid shuffle mode, and the behavior change of overdraft network
> buffers.
> > Therefore, it might be nicer to provide an implementation plan with a
> list
> > of related tasks in the FLIP. This should not block the FLIP though.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su 
> wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a discussion on FLIP-450: Improve Runtime
> > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have
> > > revisited all runtime configurations and identified several
> > > improvements to enhance user-friendliness and maintainability. In this
> > > FLIP, we aim to refine the runtime configuration.
> > >
> > > Looking forward to everyone's feedback and suggestions. Thank you!
> > >
> > > Best regards,
> > > Xuannan
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0
> > >
> >
>

Re: [VOTE] FLIP-445: Support dynamic parallelism inference for HiveSource

[Rui Fan]

+1(binding)

Best,
Rui

On Fri, Apr 26, 2024 at 10:26 AM Muhammet Orazov
 wrote:

> Hey Xia,
>
> +1 (non-binding)
>
> Thanks and best,
> Muhammet
>
> On 2024-04-26 02:21, Xia Sun wrote:
> > Hi everyone,
> >
> > I'd like to start a vote on FLIP-445: Support dynamic parallelism
> > inference
> > for HiveSource[1] which has been discussed in this thread [2].
> >
> > The vote will be open for at least 72 hours unless there is an
> > objection or
> > not enough votes.
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-445%3A+Support+dynamic+parallelism+inference+for+HiveSource
> > [2] https://lists.apache.org/thread/4k1qx6lodhbkknkhjyl0lq9bx8fcpjvn
> >
> >
> > Best,
> > Xia
>

Re: [VOTE] FLIP-446: Kubernetes Operator State Snapshot CRD

[Rui Fan]

+1(binding)

Best,
Rui

On Wed, Apr 24, 2024 at 4:03 PM Mate Czagany  wrote:

> Hi everyone,
>
> I'd like to start a vote on the FLIP-446: Kubernetes Operator State
> Snapshot CRD [1]. The discussion thread is here [2].
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-446%3A+Kubernetes+Operator+State+Snapshot+CRD
> [2] https://lists.apache.org/thread/q5dzjwj0qk34rbg2sczyypfhokxoc3q7
>
> Regards,
> Mate
>

[jira] [Created] (FLINK-35227) Remove execution-mode in ExecutionConfigInfo

[Rui Fan (Jira)]

Rui Fan created FLINK-35227:
---

 Summary: Remove execution-mode in ExecutionConfigInfo
 Key: FLINK-35227
 URL: https://issues.apache.org/jira/browse/FLINK-35227
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / REST
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 2.0.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35226) Deprecate execution-mode in ExecutionConfigInfo related rest api

[Rui Fan (Jira)]

Rui Fan created FLINK-35226:
---

 Summary: Deprecate execution-mode in ExecutionConfigInfo related 
rest api
 Key: FLINK-35226
 URL: https://issues.apache.org/jira/browse/FLINK-35226
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Web Frontend
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.20.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35225) Remove Execution mode in Flink WebUI

[Rui Fan (Jira)]

Rui Fan created FLINK-35225:
---

 Summary: Remove Execution mode in Flink WebUI
 Key: FLINK-35225
 URL: https://issues.apache.org/jira/browse/FLINK-35225
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Web Frontend
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.20.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35223) Add jobType in JobDetailsInfo related rest api

[Rui Fan (Jira)]

Rui Fan created FLINK-35223:
---

 Summary: Add jobType in JobDetailsInfo related rest api
 Key: FLINK-35223
 URL: https://issues.apache.org/jira/browse/FLINK-35223
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Web Frontend
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.20.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35224) Show the JobType on Flink WebUI

[Rui Fan (Jira)]

Rui Fan created FLINK-35224:
---

 Summary: Show the JobType on Flink WebUI
 Key: FLINK-35224
 URL: https://issues.apache.org/jira/browse/FLINK-35224
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Web Frontend
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.20.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35222) Adding getJobType for AccessExecutionGraph

[Rui Fan (Jira)]

Rui Fan created FLINK-35222:
---

 Summary: Adding getJobType for AccessExecutionGraph
 Key: FLINK-35222
 URL: https://issues.apache.org/jira/browse/FLINK-35222
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Web Frontend
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.20.0


Adding getJobType for AccessExecutionGraph interface, and all implementations 
need to overrite it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [SUMMARY] Flink 1.20 Release Sync 04/23/2024

[Rui Fan]

Hi Muhammet,

Thanks for your reminder.

Each meeting will start from 10am (UTC+2) and 4pm (UTC+8).
And the release sync happens bi-weekly at first, and will be
adjusted to weekly as we approach the feature freeze date.

Best,
Rui

On Tue, Apr 23, 2024 at 5:34 PM Muhammet Orazov 
wrote:

> Hey Rui,
>
> Thanks for the summary!
>
> Could you please write what time are you meeting on 5th May?
>
> Thanks and best,
> Muhammet
>
> On 2024-04-23 09:13, Rui Fan wrote:
> > Dear devs,
> >
> > Today is the second meeting for Flink 1.20 release cycle. I'd like to
> > share
> > the information synced in the meeting.
> >
> > - Features:
> >   There're 8 weeks until the feature freeze date, so far we've had 6
> >   flips/features and the status looks good. It is encouraged to
> > continuously
> >   updating the 1.20 wiki page[1] for contributors.
> >
> > - Blockers:
> >   - [Closed] Change on getTransitivePredecessors breaks connectors
> > FLINK-35009
> >   - [Doing] 2 tests fail FLINK-35041, FLINK-34997
> >   - [Doing] 2 benchmarks performance regression FLINK-35040,
> > FLINK-35215
> >
> > - Sync meeting[2]:
> >  The next meeting is 05/07/2024, please feel free to join us.
> >
> > Lastly, we encourage attendees to fill out the topics to be discussed
> > at
> > the bottom of 1.20 wiki page[1] a day in advance, to make it easier for
> > everyone to understand the background of the topics.
> >
> > [1] https://cwiki.apache.org/confluence/display/FLINK/1.20+Release
> > [2] https://meet.google.com/mtj-huez-apu
> >
> > Best,
> >
> > Weijie, Ufuk, Robert and Rui
>

[SUMMARY] Flink 1.20 Release Sync 04/23/2024

[Rui Fan]

Dear devs,

Today is the second meeting for Flink 1.20 release cycle. I'd like to share
the information synced in the meeting.

- Features:
  There're 8 weeks until the feature freeze date, so far we've had 6
  flips/features and the status looks good. It is encouraged to continuously
  updating the 1.20 wiki page[1] for contributors.

- Blockers:
  - [Closed] Change on getTransitivePredecessors breaks connectors
FLINK-35009
  - [Doing] 2 tests fail FLINK-35041, FLINK-34997
  - [Doing] 2 benchmarks performance regression FLINK-35040, FLINK-35215

- Sync meeting[2]:
 The next meeting is 05/07/2024, please feel free to join us.

Lastly, we encourage attendees to fill out the topics to be discussed at
the bottom of 1.20 wiki page[1] a day in advance, to make it easier for
everyone to understand the background of the topics.

[1] https://cwiki.apache.org/confluence/display/FLINK/1.20+Release
[2] https://meet.google.com/mtj-huez-apu

Best,

Weijie, Ufuk, Robert and Rui

[jira] [Created] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed

[Rui Fan (Jira)]

Rui Fan created FLINK-35215:
---

 Summary: The performance of serializerKryo and 
serializerKryoWithoutRegistration are regressed
 Key: FLINK-35215
 URL: https://issues.apache.org/jira/browse/FLINK-35215
 Project: Flink
  Issue Type: Bug
  Components: API / Type Serialization System
Affects Versions: 1.20.0
Reporter: Rui Fan


The performance of serializerKryo and serializerKryoWithoutRegistration are 
regressed[1][2], I checked recent commits, and found FLINK-34954 changed 
related logic.

 

[1] 
[http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerKryo=on=on=off=3=50]

[2] 
http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerKryoWithoutRegistration=on=on=off=3=50

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

[Rui Fan]

Thanks Yue for driving this proposal!

This upgrade makes sense, +1 for that.

Best,
Rui

On Tue, Apr 23, 2024 at 12:06 PM Roman Khachatryan  wrote:

> Hi,
>
> Thanks for writing the proposal and preparing the upgrade.
>
> FRocksDB  definitely needs to be kept in sync with the upstream and the new
> APIs are necessary for faster rescaling.
> We're already using a similar version internally.
>
> I reviewed the FLIP and it looks good to me (disclaimer: I took part in
> some steps of this effort).
>
>
> Regards,
> Roman
>
> On Mon, Apr 22, 2024, 08:11 yue ma  wrote:
>
> > Hi Flink devs,
> >
> > I would like to start a discussion on FLIP-447: Upgrade FRocksDB from
> > 6.20.3 to 8.10.0
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0
> >
> > This FLIP proposes upgrading the version of FRocksDB in the Flink Project
> > from 6.20.3 to 8.10.0.
> > The FLIP mainly introduces the main benefits of upgrading FRocksDB,
> > including the use of IngestDB which can improve Rescaling performance by
> > more than 10 times in certain scenarios, as well as other potential
> > optimization points such as async_io, blob db, and tiered storage.The
> > FLIP also presented test results based on RocksDB 8.10, including
> > StateBenchmark and Nexmark tests.
> > Overall, upgrading FRocksDB may result in a small regression of write
> > performance( which is a very small part of the overall overhead), but it
> > can bring many important performance benefits.
> > So we hope to upgrade the version of FRocksDB through this FLIP.
> >
> > Looking forward to everyone's feedback and suggestions. Thank you!
> > --
> > Best regards,
> > Yue
> >
>

Re: [VOTE] FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines

[Rui Fan]

+1(binding)

Best,
Rui

On Wed, Apr 17, 2024 at 2:45 PM Feng Jin  wrote:

> +1 (non-binding)
>
> Best,
> Feng Jin
>
> On Wed, Apr 17, 2024 at 2:28 PM Ron liu  wrote:
>
> > +1(binding)
> >
> > Best,
> > Ron
> >
> > Ron liu  于2024年4月17日周三 14:27写道：
> >
> > > Hi Dev,
> > >
> > > Thank you to everyone for the feedback on FLIP-435: Introduce a New
> > > Materialized Table for Simplifying Data Pipelines[1][2].
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours unless there is an objection or not enough votes.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-435%3A+Introduce+a+New+Materialized+Table+for+Simplifying+Data+Pipelines
> > > [2] https://lists.apache.org/thread/c1gnn3bvbfs8v1trlf975t327s4rsffs
> > >
> > > Best,
> > > Ron
> > >
> >
>

Re: [VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0

[Rui Fan]

+1(binding)

Best,
Rui

On Wed, Apr 17, 2024 at 1:02 PM Xuannan Su  wrote:

> Hi everyone,
>
> Thanks for all the feedback about the FLIP-442: General Improvement to
> Configuration for Flink 2.0 [1] [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours(excluding weekends,until APR 22, 12:00AM GMT) unless there is an
> objection or an insufficient number of votes.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-442%3A+General+Improvement+to+Configuration+for+Flink+2.0
> [2] https://lists.apache.org/thread/15k0stwyoknhxvd643ctwjw3fd17pqwk
>
>
> Best regards,
> Xuannan
>

[jira] [Created] (FLINK-35105) Support setting default Autoscaler options at autoscaler standalone level

[Rui Fan (Jira)]

Rui Fan created FLINK-35105:
---

 Summary: Support setting default Autoscaler options at autoscaler 
standalone level
 Key: FLINK-35105
 URL: https://issues.apache.org/jira/browse/FLINK-35105
 Project: Flink
  Issue Type: Sub-task
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.9.0


Currently, autoscaler standalone doesn't support set [autoscaler 
options|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.8/docs/operations/configuration/#autoscaler-configuration].
 We must set them at job level when we use autoscaler standalone. It's not 
convenient if platform administrator wanna change the default value for some 
autoscaler options, such as:
 * job.autoscaler.enabled
 * job.autoscaler.metrics.window
 * etc

This Jira supports setting Autoscaler options at autoscaler standalone level, 
it's similar with flink kubernetes operator.

The  autoscaler options of autoscaler standalone will be as the base 
configuration, and the configuration at job-level can override the default 
value provided in the autoscaler standalone.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

[Rui Fan]

Hi devs,

FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI[1]
has been accepted and voted through this thread[2].

Thank you all for your discussions and votes.

The proposal received 14  approving votes, 6 of which are binding, and
there is no disapproval.

- Feifan Wang
- Samrat Deb
- gongzhongqiang
- Zhu Zhu (binding)
- Ahmed Hamdy
- Muhammet Orazov
- Jinzhong Li
- Hangxiang Yu (binding)
- Yanfei Lei (binding)
- Yuepeng Pan
- Zakelly Lan (binding)
- Lijie Wang (binding)
- Junrui Lee
- Rui Fan (binding)

[1] https://cwiki.apache.org/confluence/x/agrPEQ
[2] https://lists.apache.org/thread/3oqodwoobhrtsrcrvvhm8p5klznx32c0

Best,
Rui

Re: [VOTE] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

[Rui Fan]

+1(binding)

And close this voting discussion as it has been going on for 5 days.
I will announce this result in a new thread.

Best,
Rui

On Fri, Apr 12, 2024 at 1:28 PM Junrui Lee  wrote:

> +1 （non-binding)
>
> Best,
> Junrui
>
> Lijie Wang  于2024年4月12日周五 12:43写道：
>
> > +1 (binding)
> >
> > Thanks for driving.
> >
> > Best,
> > Lijie
> >
> > Zakelly Lan  于2024年4月12日周五 11:08写道：
> >
> > > +1 non-binding
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Fri, Apr 12, 2024 at 11:05 AM Yuepeng Pan 
> > > wrote:
> > >
> > > > Hi Rui,
> > > >
> > > > Thanks for driving it!
> > > > +1  (non-binding)
> > > > Best,
> > > > Yuepeng Pan
> > > >
> > > > At 2024-04-12 10:31:19, "Yanfei Lei"  wrote:
> > > > >Hi Rui,
> > > > >
> > > > >Thanks for driving it!
> > > > >
> > > > >+1  (binding)
> > > > >
> > > > >Hangxiang Yu  于2024年4月12日周五 10:26写道：
> > > > >>
> > > > >> +1  (binding)
> > > > >>
> > > > >> On Fri, Apr 12, 2024 at 10:22 AM Jinzhong Li <
> > > lijinzhong2...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > +1  (non binding)
> > > > >> >
> > > > >> > Bests,
> > > > >> > Jinzhong
> > > > >> >
> > > > >> > On Thu, Apr 11, 2024 at 7:26 AM Muhammet Orazov
> > > > >> >  wrote:
> > > > >> >
> > > > >> > > Hey Rui,
> > > > >> > >
> > > > >> > > +1 (non-binding).
> > > > >> > >
> > > > >> > > Thanks for driving it!
> > > > >> > >
> > > > >> > > Best,
> > > > >> > > Muhammet
> > > > >> > >
> > > > >> > > On 2024-04-10 04:36, Rui Fan wrote:
> > > > >> > > > Hi devs,
> > > > >> > > >
> > > > >> > > > Thank you to everyone for the feedback on FLIP-441: Show
> > > > >> > > > the JobType and remove Execution Mode on Flink WebUI[1]
> > > > >> > > > which has been discussed in this thread [2].
> > > > >> > > >
> > > > >> > > > I would like to start a vote for it. The vote will be open
> for
> > > at
> > > > least
> > > > >> > > > 72
> > > > >> > > > hours unless there is an objection or not enough votes.
> > > > >> > > >
> > > > >> > > > [1] https://cwiki.apache.org/confluence/x/agrPEQ
> > > > >> > > > [2]
> > > > https://lists.apache.org/thread/0s52w17w24x7m2zo6ogl18t1fy412vcd
> > > > >> > > >
> > > > >> > > > Best,
> > > > >> > > > Rui
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Best,
> > > > >> Hangxiang.
> > > > >
> > > > >
> > > > >
> > > > >--
> > > > >Best,
> > > > >Yanfei
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

[Rui Fan]

Congratulations!

Best,
Rui

On Mon, Apr 15, 2024 at 11:01 AM Congxian Qiu 
wrote:

> Congratulations!
>
> Best,
> Congxian
>
>
> Ron liu  于2024年4月15日周一 11:00写道：
>
> > Congratulations!
> >
> > Best,
> > Ron
> >
> > Yuan Mei  于2024年4月15日周一 10:51写道：
> >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm happy to let you know that Zakelly Lan has
> > become
> > > a new Flink Committer!
> > >
> > > Zakelly has been continuously contributing to the Flink project since
> > 2020,
> > > with a focus area on Checkpointing, State as well as frocksdb (the
> > default
> > > on-disk state db).
> > >
> > > He leads several FLIPs to improve checkpoints and state APIs, including
> > > File Merging for Checkpoints and configuration/API reorganizations. He
> is
> > > also one of the main contributors to the recent efforts of
> "disaggregated
> > > state management for Flink 2.0" and drives the entire discussion in the
> > > mailing thread, demonstrating outstanding technical depth and breadth
> of
> > > knowledge.
> > >
> > > Beyond his technical contributions, Zakelly is passionate about helping
> > the
> > > community in numerous ways. He spent quite some time setting up the
> Flink
> > > Speed Center and rebuilding the benchmark pipeline after the original
> one
> > > was out of lease. He helps build frocksdb and tests for the upcoming
> > > frocksdb release (bump rocksdb from 6.20.3->8.10).
> > >
> > > Please join me in congratulating Zakelly for becoming an Apache Flink
> > > committer!
> > >
> > > Best,
> > > Yuan (on behalf of the Flink PMC)
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

[Rui Fan]

Congratulations, Jing~

Best,
Rui

On Fri, Apr 12, 2024 at 4:28 PM Yun Tang  wrote:

> Congratulations, Jing!
>
> Best
> Yun Tang
> 
> From: Jark Wu 
> Sent: Friday, April 12, 2024 16:02
> To: dev 
> Cc: gej...@gmail.com 
> Subject: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge
>
> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce that Jing Ge has
> joined the Flink PMC!
>
> Jing has been contributing to Apache Flink for a long time. He continuously
> works on SQL, connectors, Source, and Sink APIs, test, and document
> modules while contributing lots of code and insightful discussions. He is
> one of the maintainers of Flink CI infra. He is also willing to help a lot
> in the
> community work, such as being the release manager for both 1.18 and 1.19,
> verifying releases, and answering questions on the mailing list. Besides
> that,
> he is continuously helping with the expansion of the Flink community and
> has
> given several talks about Flink at many conferences, such as Flink Forward
> 2022 and 2023.
>
> Congratulations and welcome Jing!
>
> Best,
> Jark (on behalf of the Flink PMC)
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

[Rui Fan]

Congratulations, Lincoln!

Best,
Rui

On Fri, Apr 12, 2024 at 3:59 PM Jark Wu  wrote:

> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce that Lincoln Lee has
> joined the Flink PMC!
>
> Lincoln has been an active member of the Apache Flink community for
> many years. He mainly works on Flink SQL component and has driven
> /pushed many FLIPs around SQL, including FLIP-282/373/415/435 in
> the recent versions. He has a great technical vision of Flink SQL and
> participated in plenty of discussions in the dev mailing list. Besides
> that,
> he is community-minded, such as being the release manager of 1.19,
> verifying releases, managing release syncs, writing the release
> announcement etc.
>
> Congratulations and welcome Lincoln!
>
> Best,
> Jark (on behalf of the Flink PMC)
>

Re: [DISCUSSION] FLIP-442: FLIP-442: General Improvement to Configuration for Flink 2.0

[Rui Fan]

Hi Xuannan,

> Regarding the new classes, we propose updating the
> `ConfigOptionsDocGenerator` to throw an exception and fail the build
> if it detects any new class missing the proper annotation in the
> future.

Thanks for your feedback, it sounds good to me.

Best,
Rui

On Fri, Apr 12, 2024 at 9:35 AM Xuannan Su  wrote:

> Hi Rui,
>
> If I understand correctly, all classes without annotations are
> non-public by default. I'm concerned that adding an exception to this
> rule will make it harder to understand.
>
> Regarding the new classes, we propose updating the
> `ConfigOptionsDocGenerator` to throw an exception and fail the build
> if it detects any new class missing the proper annotation in the
> future.
>
> Best regards,
> Xuannan
>
>
> On Wed, Apr 10, 2024 at 2:09 PM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Thanks Xuannan for driving this proposal!
> >
> > > Ensure all the ConfigOptions are properly annotated as PublicEvolving
> >
> > Could we add a specification directly? All XxxOptions classes are
> > PublicEvolving by default. I'm afraid some new classes still miss
> > PublicEvolving in the future.
> >
> > If we have a specification, it will be clear. And we don't need to
> > add PublicEvolving for each XxxOptions.
> >
> > Best,
> > Rui
> >
> > On Wed, Apr 10, 2024 at 1:54 PM Muhammet Orazov
>  wrote:
> >>
> >> Hey Xuannan,
> >>
> >> Thanks for the FLIP and your efforts!
> >>
> >> Minor clarification from my side:
> >>
> >> > We will relocate these ConfigOptions to a class that is included
> >> > in the documentation generation.
> >>
> >> Would it make sense to define also in the FLIP the options class for
> >> these variables? For example, GPUDriverOptions?
> >>
> >> Best,
> >> Muhammet
> >>
> >> On 2024-04-09 08:20, Xuannan Su wrote:
> >> > Hi all,
> >> >
> >> > I'd like to start a discussion on FLIP-442: General Improvement to
> >> > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we aim to
> >> > provide users with a better experience with the existing
> >> > configuration. This FLIP proposes several general improvements to the
> >> > current configuration.
> >> >
> >> > Looking forward to everyone's feedback and suggestions. Thank you!
> >> >
> >> > Best regards,
> >> > Xuannan
> >> >
> >> > [1]
> >> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-442%3A+General+Improvement+to+Configuration+for+Flink+2.0
>

Re: [DISCUSSION] FLIP-442: FLIP-442: General Improvement to Configuration for Flink 2.0

[Rui Fan]

Thanks Xuannan for driving this proposal!

> Ensure all the ConfigOptions are properly annotated as PublicEvolving

Could we add a specification directly? All XxxOptions classes are
PublicEvolving by default. I'm afraid some new classes still miss
PublicEvolving in the future.

If we have a specification, it will be clear. And we don't need to
add PublicEvolving for each XxxOptions.

Best,
Rui

On Wed, Apr 10, 2024 at 1:54 PM Muhammet Orazov
 wrote:

> Hey Xuannan,
>
> Thanks for the FLIP and your efforts!
>
> Minor clarification from my side:
>
> > We will relocate these ConfigOptions to a class that is included
> > in the documentation generation.
>
> Would it make sense to define also in the FLIP the options class for
> these variables? For example, GPUDriverOptions?
>
> Best,
> Muhammet
>
> On 2024-04-09 08:20, Xuannan Su wrote:
> > Hi all,
> >
> > I'd like to start a discussion on FLIP-442: General Improvement to
> > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we aim to
> > provide users with a better experience with the existing
> > configuration. This FLIP proposes several general improvements to the
> > current configuration.
> >
> > Looking forward to everyone's feedback and suggestions. Thank you!
> >
> > Best regards,
> > Xuannan
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-442%3A+General+Improvement+to+Configuration+for+Flink+2.0
>

[VOTE] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

[Rui Fan]

Hi devs,

Thank you to everyone for the feedback on FLIP-441: Show
the JobType and remove Execution Mode on Flink WebUI[1]
which has been discussed in this thread [2].

I would like to start a vote for it. The vote will be open for at least 72
hours unless there is an objection or not enough votes.

[1] https://cwiki.apache.org/confluence/x/agrPEQ
[2] https://lists.apache.org/thread/0s52w17w24x7m2zo6ogl18t1fy412vcd

Best,
Rui

Re: [DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

[Rui Fan]

Sorry, it's a typo. It should be FLINK-32558[1].

[1] https://issues.apache.org/jira/browse/FLINK-32558

Best,
Rui

On Mon, Apr 8, 2024 at 6:44 PM Ahmed Hamdy  wrote:

> Hi Rui,
> Thanks for the proposal.
> Is the deprecation Jira mentioned (FLINK-32258) correct?
> Best Regards
> Ahmed Hamdy
>
>
> On Sun, 7 Apr 2024 at 03:37, Rui Fan <1996fan...@gmail.com> wrote:
>
> > If there are no extra comments, I will start voting in three days, thank
> > you~
> >
> > Best,
> > Rui
> >
> > On Thu, Mar 28, 2024 at 4:46 PM Muhammet Orazov
> >  wrote:
> >
> > > Hey Rui,
> > >
> > > Thanks for the detailed explanation and updating the FLIP!
> > >
> > > It is much clearer definitely, thanks for the proposal.
> > >
> > > Best,
> > > Muhammet
> > >
> > > On 2024-03-28 07:37, Rui Fan wrote:
> > > > Hi Muhammet,
> > > >
> > > > Thanks for your reply!
> > > >
> > > >> The execution mode is also used for the DataStream API [1],
> > > >> would that also affect/hide the DataStream execution mode
> > > >> if we remove it from the WebUI?
> > > >
> > > > Sorry, I didn't describe it clearly in FLIP-441[2], I have updated
> it.
> > > > Let me clarify the Execution Mode here:
> > > >
> > > > 1. Flink 1.19 website[3] also mentions the Execution mode, but it
> > > > actually matches the JobType[4] in the Flink code. Both of them
> > > > have 2 types: STREAMING and BATCH.
> > > > 2. execution.runtime-mode can be set to 3 types: STREAMING,
> > > > BATCH and AUTOMATIC. But the jobType will be inferred as
> > > > STREAMING or BATCH when execution.runtime-mode is set
> > > > to AUTOMATIC.
> > > > 3. The ExecutionMode I describe is: code link[5] , as we can
> > > > see, ExecutionMode has 4 enums: PIPELINED,
> > > > PIPELINED_FORCED, BATCH and BATCH_FORCED.
> > > > And we can see a flink streaming job from Flink WebUI,
> > > > the Execution mode is PIPELINE instead of STREAMING.
> > > > I attached a screenshot to the FLIP doc[2], you can see it there.
> > > > 4. What this proposal wants to do is to remove the ExecutionMode
> > > > with four enumerations on Flink WebUI and introduce the
> > > > JobType with two enumerations (STREAMING or BATCH).
> > > > STREAMING or BATCH is clearer and more accurate for users.
> > > >
> > > > Please let me know if it's not clear or anything is wrong, thanks a
> > > > lot!
> > > >
> > > > [1]
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/execution_mode/
> > > > [2] https://cwiki.apache.org/confluence/x/agrPEQ
> > > > [3]
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/
> > > > [4]
> > > >
> > >
> >
> https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobType.java#L22
> > > > [5]
> > > >
> > >
> >
> https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-core/src/main/java/org/apache/flink/api/common/ExecutionMode.java#L54
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Thu, Mar 28, 2024 at 1:33 AM Venkatakrishnan Sowrirajan
> > > > 
> > > > wrote:
> > > >
> > > >> Rui,
> > > >>
> > > >> I assume the current proposal would also handle the case of mixed
> mode
> > > >> (BATCH + STREAMING within the same app) in the future, right?
> > > >>
> > > >> Regards
> > > >> Venkat
> > > >>
> > > >> On Wed, Mar 27, 2024 at 10:15 AM Venkatakrishnan Sowrirajan <
> > > >> vsowr...@asu.edu> wrote:
> > > >>
> > > >>> This will be a very useful addition to Flink UI. Thanks Rui for
> > > >>> starting
> > > >>> a FLIP for this improvement.
> > > >>>
> > > >>> Regards
> > > >>> Venkata krishnan
> > > >>>
> > > >>>
> > > >>> On Wed, Mar 27, 2024 at 4:49 AM Muhammet Orazov
> > > >>>  wrote:
> > > &g

[jira] [Created] (FLINK-35040) The performance of serializerHeavyString regresses since April 3

[Rui Fan (Jira)]

Rui Fan created FLINK-35040:
---

 Summary: The performance of serializerHeavyString regresses since 
April 3
 Key: FLINK-35040
 URL: https://issues.apache.org/jira/browse/FLINK-35040
 Project: Flink
  Issue Type: Bug
  Components: Benchmarks
Affects Versions: 1.20.0
Reporter: Rui Fan
 Attachments: image-2024-04-08-10-51-07-403.png

The performance of serializerHeavyString regresses since April 3, and had not 
yet recovered on April 8th.

http://flink-speed.xyz/timeline/#/?exe=6=serializerHeavyString=on=on=off=3=200


 !image-2024-04-08-10-51-07-403.png! 





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

[Rui Fan]

If there are no extra comments, I will start voting in three days, thank
you~

Best,
Rui

On Thu, Mar 28, 2024 at 4:46 PM Muhammet Orazov
 wrote:

> Hey Rui,
>
> Thanks for the detailed explanation and updating the FLIP!
>
> It is much clearer definitely, thanks for the proposal.
>
> Best,
> Muhammet
>
> On 2024-03-28 07:37, Rui Fan wrote:
> > Hi Muhammet,
> >
> > Thanks for your reply!
> >
> >> The execution mode is also used for the DataStream API [1],
> >> would that also affect/hide the DataStream execution mode
> >> if we remove it from the WebUI?
> >
> > Sorry, I didn't describe it clearly in FLIP-441[2], I have updated it.
> > Let me clarify the Execution Mode here:
> >
> > 1. Flink 1.19 website[3] also mentions the Execution mode, but it
> > actually matches the JobType[4] in the Flink code. Both of them
> > have 2 types: STREAMING and BATCH.
> > 2. execution.runtime-mode can be set to 3 types: STREAMING,
> > BATCH and AUTOMATIC. But the jobType will be inferred as
> > STREAMING or BATCH when execution.runtime-mode is set
> > to AUTOMATIC.
> > 3. The ExecutionMode I describe is: code link[5] , as we can
> > see, ExecutionMode has 4 enums: PIPELINED,
> > PIPELINED_FORCED, BATCH and BATCH_FORCED.
> > And we can see a flink streaming job from Flink WebUI,
> > the Execution mode is PIPELINE instead of STREAMING.
> > I attached a screenshot to the FLIP doc[2], you can see it there.
> > 4. What this proposal wants to do is to remove the ExecutionMode
> > with four enumerations on Flink WebUI and introduce the
> > JobType with two enumerations (STREAMING or BATCH).
> > STREAMING or BATCH is clearer and more accurate for users.
> >
> > Please let me know if it's not clear or anything is wrong, thanks a
> > lot!
> >
> > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/execution_mode/
> > [2] https://cwiki.apache.org/confluence/x/agrPEQ
> > [3]
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/
> > [4]
> >
> https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobType.java#L22
> > [5]
> >
> https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-core/src/main/java/org/apache/flink/api/common/ExecutionMode.java#L54
> >
> > Best,
> > Rui
> >
> > On Thu, Mar 28, 2024 at 1:33 AM Venkatakrishnan Sowrirajan
> > 
> > wrote:
> >
> >> Rui,
> >>
> >> I assume the current proposal would also handle the case of mixed mode
> >> (BATCH + STREAMING within the same app) in the future, right?
> >>
> >> Regards
> >> Venkat
> >>
> >> On Wed, Mar 27, 2024 at 10:15 AM Venkatakrishnan Sowrirajan <
> >> vsowr...@asu.edu> wrote:
> >>
> >>> This will be a very useful addition to Flink UI. Thanks Rui for
> >>> starting
> >>> a FLIP for this improvement.
> >>>
> >>> Regards
> >>> Venkata krishnan
> >>>
> >>>
> >>> On Wed, Mar 27, 2024 at 4:49 AM Muhammet Orazov
> >>>  wrote:
> >>>
> >>>> Hello Rui,
> >>>>
> >>>> Thanks for the proposal! It looks good!
> >>>>
> >>>> I have minor clarification from my side:
> >>>>
> >>>> The execution mode is also used for the DataStream API [1],
> >>>> would that also affect/hide the DataStream execution mode
> >>>> if we remove it from the WebUI?
> >>>>
> >>>> Best,
> >>>> Muhammet
> >>>>
> >>>> [1]:
> >>>>
> >>>>
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/execution_mode/__;!!IKRxdwAv5BmarQ!eFyqVJyje_8hu1vMSUwKGBsj8vqsFDisEvJ5AxPV0LduhhHWF3rPKYEEE-09biA0unFbfMy5AVQZMgBv1AOa5oTHmcYlkUE$
> >>>>
> >>>>
> >>>> On 2024-03-27 06:23, Rui Fan wrote:
> >>>> > Hi flink developers,
> >>>> >
> >>>> > I'd like to start a discussion to discuss FLIP-441:
> >>>> > Show the JobType and remove Execution Mode on Flink WebUI[1].
> >>>> >
> >>>> > Currently, the jobType has 2 types in Flink: STREAMING and BATCH.
> >>>> > They work on completely different principles, suc

Re: Re: [VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State

[Rui Fan]

+1(binding)

Best,
Rui

On Thu, Mar 28, 2024 at 8:32 PM Feifan Wang  wrote:

> +1 (non-binding)
>
>
>
>
> ——
>
> Best regards,
>
> Feifan Wang
>
>
>
>
> At 2024-03-28 18:43:36, "Yuan Mei"  wrote:
> >+1 (binding)
> >
> >Best,
> >Yuan
> >
> >On Wed, Mar 27, 2024 at 7:31 PM Jinzhong Li 
> >wrote:
> >
> >> Hi devs,
> >>
> >>
> >> I'd like to start a vote on the FLIP-428: Fault Tolerance/Rescale
> >> Integration for Disaggregated State [1]. The discussion thread is here
> [2].
> >>
> >>
> >> The vote will be open for at least 72 hours unless there is an
> objection or
> >> insufficient votes.
> >>
> >> [1] https://cwiki.apache.org/confluence/x/UYp3EQ
> >>
> >> [2] https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b
> >>
> >> Best,
> >>
> >> Jinzhong
> >>
>

Re: Re: [VOTE] FLIP-426: Grouping Remote State Access

[Rui Fan]

+1(binding)

Best,
Rui

On Thu, Mar 28, 2024 at 8:31 PM Feifan Wang  wrote:

> +1 (non-binding)
>
>
>
>
> ——
>
> Best regards,
>
> Feifan Wang
>
>
>
>
> At 2024-03-28 18:43:12, "Yuan Mei"  wrote:
> >+1 (binding)
> >
> >Best,
> >Yuan
> >
> >On Wed, Mar 27, 2024 at 6:56 PM Jinzhong Li 
> >wrote:
> >
> >> Hi devs,
> >>
> >> I'd like to start a vote on the FLIP-426: Grouping Remote State Access
> [1].
> >> The discussion thread is here [2].
> >>
> >> The vote will be open for at least 72 hours unless there is an
> objection or
> >> insufficient votes.
> >>
> >>
> >> [1] https://cwiki.apache.org/confluence/x/TYp3EQ
> >>
> >> [2] https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf
> >>
> >>
> >> Best,
> >>
> >> Jinzhong
> >>
>

Re: Re: [VOTE] FLIP-424: Asynchronous State APIs

[Rui Fan]

+1(binding)

Best,
Rui

On Thu, Mar 28, 2024 at 8:31 PM Feifan Wang  wrote:

> +1 (non-binding)
>
>
>
>
> ——
>
> Best regards,
>
> Feifan Wang
>
>
>
>
> At 2024-03-28 20:21:50, "Jing Ge"  wrote:
> >+1 (binding)
> >
> >Thanks!
> >
> >Best regards,
> >Jing
> >
> >On Wed, Mar 27, 2024 at 11:23 AM Zakelly Lan 
> wrote:
> >
> >> Hi devs,
> >>
> >> I'd like to start a vote on the FLIP-424: Asynchronous State APIs [1].
> The
> >> discussion thread is here [2].
> >>
> >> The vote will be open for at least 72 hours unless there is an
> objection or
> >> insufficient votes.
> >>
> >> [1] https://cwiki.apache.org/confluence/x/SYp3EQ
> >> [2] https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864
> >>
> >>
> >> Best,
> >> Zakelly
> >>
>

Re: Re: [VOTE] FLIP-425: Asynchronous Execution Model

[Rui Fan]

+1(binding)

Best,
Rui

On Thu, Mar 28, 2024 at 10:12 PM Jing Ge  wrote:

> +1(binding)
>
> Best Regards,
> Jing
>
> On Thu, Mar 28, 2024 at 1:30 PM Feifan Wang  wrote:
>
> > +1 (non-binding)
> >
> >
> >
> > ——
> >
> > Best regards,
> >
> > Feifan Wang
> >
> >
> >
> >
> > At 2024-03-28 20:20:39, "Piotr Nowojski" 
> wrote:
> > >+1 (binding)
> > >
> > >Piotrek
> > >
> > >czw., 28 mar 2024 o 11:44 Yuan Mei  napisał(a):
> > >
> > >> +1 (binding)
> > >>
> > >> Best,
> > >> Yuan
> > >>
> > >> On Thu, Mar 28, 2024 at 4:33 PM Xuannan Su 
> > wrote:
> > >>
> > >> > +1 (non-binding)
> > >> >
> > >> > Best regards,
> > >> > Xuannan
> > >> >
> > >> > On Wed, Mar 27, 2024 at 6:28 PM Yanfei Lei 
> > wrote:
> > >> > >
> > >> > > Hi everyone,
> > >> > >
> > >> > > Thanks for all the feedback about the FLIP-425: Asynchronous
> > Execution
> > >> > > Model [1]. The discussion thread is here [2].
> > >> > >
> > >> > > The vote will be open for at least 72 hours unless there is an
> > >> > > objection or insufficient votes.
> > >> > >
> > >> > > [1] https://cwiki.apache.org/confluence/x/S4p3EQ
> > >> > > [2]
> > https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h
> > >> > >
> > >> > > Best regards,
> > >> > > Yanfei
> > >> >
> > >>
> >
>

Re: Re: [VOTE] FLIP-427: Disaggregated state Store

[Rui Fan]

+1(binding)

Best,
Rui

On Thu, Mar 28, 2024 at 10:30 PM Piotr Nowojski 
wrote:

> +1 (binding)
>
> czw., 28 mar 2024 o 13:31 Feifan Wang  napisał(a):
>
> > +1 (non-binding)
> >
> >
> >
> >
> > ——
> >
> > Best regards,
> >
> > Feifan Wang
> >
> >
> >
> >
> > At 2024-03-28 19:01:01, "Yuan Mei"  wrote:
> > >+1 (binding)
> > >
> > >Best
> > >Yuan
> > >
> > >
> > >
> > >
> > >On Wed, Mar 27, 2024 at 6:37 PM Hangxiang Yu 
> wrote:
> > >
> > >> Hi devs,
> > >>
> > >> Thanks all for your valuable feedback about FLIP-427: Disaggregated
> > state
> > >> Store [1].
> > >> I'd like to start a vote on it.  The discussion thread is here [2].
> > >>
> > >> The vote will be open for at least 72 hours unless there is an
> > objection or
> > >> insufficient votes.
> > >>
> > >> [1] https://cwiki.apache.org/confluence/x/T4p3EQ
> > >> [2] https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft
> > >>
> > >>
> > >> Best,
> > >> Hangxiang
> > >>
> >
>

Re: Re: [VOTE] FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)

[Rui Fan]

+1(binding)

Best,
Rui

On Thu, Mar 28, 2024 at 10:13 PM Jing Ge  wrote:

> +1(binding)
>
> Best Regards,
> Jing
>
> On Thu, Mar 28, 2024 at 1:28 PM Feifan Wang  wrote:
>
> > +1 (non-binding)
> >
> > ——
> >
> > Best regards,
> >
> > Feifan Wang
> >
> >
> >
> >
> > At 2024-03-28 20:20:21, "Piotr Nowojski"  wrote:
> > >+1 (binding)
> > >
> > >Piotrek
> > >
> > >czw., 28 mar 2024 o 11:43 Yuan Mei  napisał(a):
> > >
> > >> Hi devs,
> > >>
> > >> I'd like to start a vote on the FLIP-423: Disaggregated State Storage
> > and
> > >> Management (Umbrella FLIP) [1]. The discussion thread is here [2].
> > >>
> > >> The vote will be open for at least 72 hours unless there is an
> > objection or
> > >> insufficient votes.
> > >>
> > >> [1] https://cwiki.apache.org/confluence/x/R4p3EQ
> > >> [2] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
> > >>
> > >>
> > >> Best,
> > >> Yuan
> > >>
> >
>

Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

[Rui Fan]

Congratulations~

Best,
Rui

On Thu, Mar 28, 2024 at 3:55 PM Yu Li  wrote:

> CC the Flink user and dev mailing list.
>
> Paimon originated within the Flink community, initially known as Flink
> Table Store, and all our incubating mentors are members of the Flink
> Project Management Committee. I am confident that the bonds of
> enduring friendship and close collaboration will continue to unite the
> two communities.
>
> And congratulations all!
>
> Best Regards,
> Yu
>
> On Wed, 27 Mar 2024 at 20:35, Guojun Li  wrote:
> >
> > Congratulations!
> >
> > Best,
> > Guojun
> >
> > On Wed, Mar 27, 2024 at 5:24 PM wulin  wrote:
> >
> > > Congratulations~
> > >
> > > > 2024年3月27日 15:54，王刚  写道：
> > > >
> > > > Congratulations~
> > > >
> > > >> 2024年3月26日 10:25，Jingsong Li  写道：
> > > >>
> > > >> Hi Paimon community,
> > > >>
> > > >> I’m glad to announce that the ASF board has approved a resolution to
> > > >> graduate Paimon into a full Top Level Project. Thanks to everyone
> for
> > > >> your help to get to this point.
> > > >>
> > > >> I just created an issue to track the things we need to modify [2],
> > > >> please comment on it if you feel that something is missing. You can
> > > >> refer to apache documentation [1] too.
> > > >>
> > > >> And, we already completed the GitHub repo migration [3], please
> update
> > > >> your local git repo to track the new repo [4].
> > > >>
> > > >> You can run the following command to complete the remote repo
> tracking
> > > >> migration.
> > > >>
> > > >> git remote set-url origin https://github.com/apache/paimon.git
> > > >>
> > > >> If you have a different name, please change the 'origin' to your
> remote
> > > name.
> > > >>
> > > >> Please join me in celebrating!
> > > >>
> > > >> [1]
> > >
> https://incubator.apache.org/guides/transferring.html#life_after_graduation
> > > >> [2] https://github.com/apache/paimon/issues/3091
> > > >> [3] https://issues.apache.org/jira/browse/INFRA-25630
> > > >> [4] https://github.com/apache/paimon
> > > >>
> > > >> Best,
> > > >> Jingsong Lee
> > >
> > >
>

Re: [DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

[Rui Fan]

Hi Venkat,

> I assume the current proposal would also handle the case of mixed mode
(BATCH + STREAMING within the same app) in the future, right?

IIUC, currently one flink job is either STREAMING or BATCH.
And flink doesn't support mixed mode so far.

If one flink job can run with mixed mode, andthe JobType will
have the third type in the future, the WebUI can show it properly.

Best,
Rui

On Thu, Mar 28, 2024 at 3:37 PM Rui Fan <1996fan...@gmail.com> wrote:

> Hi Muhammet,
>
> Thanks for your reply!
>
> > The execution mode is also used for the DataStream API [1],
> > would that also affect/hide the DataStream execution mode
> > if we remove it from the WebUI?
>
> Sorry, I didn't describe it clearly in FLIP-441[2], I have updated it.
> Let me clarify the Execution Mode here:
>
> 1. Flink 1.19 website[3] also mentions the Execution mode, but it
> actually matches the JobType[4] in the Flink code. Both of them
> have 2 types: STREAMING and BATCH.
> 2. execution.runtime-mode can be set to 3 types: STREAMING,
> BATCH and AUTOMATIC. But the jobType will be inferred as
> STREAMING or BATCH when execution.runtime-mode is set
> to AUTOMATIC.
> 3. The ExecutionMode I describe is: code link[5] , as we can
> see, ExecutionMode has 4 enums: PIPELINED,
> PIPELINED_FORCED, BATCH and BATCH_FORCED.
> And we can see a flink streaming job from Flink WebUI,
> the Execution mode is PIPELINE instead of STREAMING.
> I attached a screenshot to the FLIP doc[2], you can see it there.
> 4. What this proposal wants to do is to remove the ExecutionMode
> with four enumerations on Flink WebUI and introduce the
> JobType with two enumerations (STREAMING or BATCH).
> STREAMING or BATCH is clearer and more accurate for users.
>
> Please let me know if it's not clear or anything is wrong, thanks a lot!
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/execution_mode/
> [2] https://cwiki.apache.org/confluence/x/agrPEQ
> [3]
> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/
> [4]
> https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobType.java#L22
> [5]
> https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-core/src/main/java/org/apache/flink/api/common/ExecutionMode.java#L54
>
> Best,
> Rui
>
> On Thu, Mar 28, 2024 at 1:33 AM Venkatakrishnan Sowrirajan <
> vsowr...@asu.edu> wrote:
>
>> Rui,
>>
>> I assume the current proposal would also handle the case of mixed mode
>> (BATCH + STREAMING within the same app) in the future, right?
>>
>> Regards
>> Venkat
>>
>> On Wed, Mar 27, 2024 at 10:15 AM Venkatakrishnan Sowrirajan <
>> vsowr...@asu.edu> wrote:
>>
>>> This will be a very useful addition to Flink UI. Thanks Rui for starting
>>> a FLIP for this improvement.
>>>
>>> Regards
>>> Venkata krishnan
>>>
>>>
>>> On Wed, Mar 27, 2024 at 4:49 AM Muhammet Orazov
>>>  wrote:
>>>
>>>> Hello Rui,
>>>>
>>>> Thanks for the proposal! It looks good!
>>>>
>>>> I have minor clarification from my side:
>>>>
>>>> The execution mode is also used for the DataStream API [1],
>>>> would that also affect/hide the DataStream execution mode
>>>> if we remove it from the WebUI?
>>>>
>>>> Best,
>>>> Muhammet
>>>>
>>>> [1]:
>>>>
>>>> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/execution_mode/__;!!IKRxdwAv5BmarQ!eFyqVJyje_8hu1vMSUwKGBsj8vqsFDisEvJ5AxPV0LduhhHWF3rPKYEEE-09biA0unFbfMy5AVQZMgBv1AOa5oTHmcYlkUE$
>>>>
>>>>
>>>> On 2024-03-27 06:23, Rui Fan wrote:
>>>> > Hi flink developers,
>>>> >
>>>> > I'd like to start a discussion to discuss FLIP-441:
>>>> > Show the JobType and remove Execution Mode on Flink WebUI[1].
>>>> >
>>>> > Currently, the jobType has 2 types in Flink: STREAMING and BATCH.
>>>> > They work on completely different principles, such as: scheduler,
>>>> > shuffle, join, etc. These differences lead to different
>>>> troubleshooting
>>>> > processes, so when users are maintaining a job or troubleshooting,
>>>> > it's needed to know whether the current job is a STREAMING or
>>>> > BATCH job. Unfortunately, Flink WebUI doesn't expose it to the
>>>> > users so far.
>>>> >
>>>> > Also, Execution Mode is related to DataSet api, it has been marked
>>>> > as @Deprecated in FLINK-32258 (1.18), but it's still shown in Flink
>>>> > WebUI.
>>>> >
>>>> > Looking forward to hearing more thoughts about it! Thank you~
>>>> >
>>>> > [1]
>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/agrPEQ__;!!IKRxdwAv5BmarQ!eFyqVJyje_8hu1vMSUwKGBsj8vqsFDisEvJ5AxPV0LduhhHWF3rPKYEEE-09biA0unFbfMy5AVQZMgBv1AOa5oTHayPyFj8$
>>>> > [2]
>>>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-32558__;!!IKRxdwAv5BmarQ!eFyqVJyje_8hu1vMSUwKGBsj8vqsFDisEvJ5AxPV0LduhhHWF3rPKYEEE-09biA0unFbfMy5AVQZMgBv1AOa5oTHftYeOLE$
>>>> >
>>>> > Best,
>>>> > Rui
>>>>
>>>

Re: [DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

[Rui Fan]

Hi Muhammet,

Thanks for your reply!

> The execution mode is also used for the DataStream API [1],
> would that also affect/hide the DataStream execution mode
> if we remove it from the WebUI?

Sorry, I didn't describe it clearly in FLIP-441[2], I have updated it.
Let me clarify the Execution Mode here:

1. Flink 1.19 website[3] also mentions the Execution mode, but it
actually matches the JobType[4] in the Flink code. Both of them
have 2 types: STREAMING and BATCH.
2. execution.runtime-mode can be set to 3 types: STREAMING,
BATCH and AUTOMATIC. But the jobType will be inferred as
STREAMING or BATCH when execution.runtime-mode is set
to AUTOMATIC.
3. The ExecutionMode I describe is: code link[5] , as we can
see, ExecutionMode has 4 enums: PIPELINED,
PIPELINED_FORCED, BATCH and BATCH_FORCED.
And we can see a flink streaming job from Flink WebUI,
the Execution mode is PIPELINE instead of STREAMING.
I attached a screenshot to the FLIP doc[2], you can see it there.
4. What this proposal wants to do is to remove the ExecutionMode
with four enumerations on Flink WebUI and introduce the
JobType with two enumerations (STREAMING or BATCH).
STREAMING or BATCH is clearer and more accurate for users.

Please let me know if it's not clear or anything is wrong, thanks a lot!

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/execution_mode/
[2] https://cwiki.apache.org/confluence/x/agrPEQ
[3]
https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/
[4]
https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobType.java#L22
[5]
https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-core/src/main/java/org/apache/flink/api/common/ExecutionMode.java#L54

Best,
Rui

On Thu, Mar 28, 2024 at 1:33 AM Venkatakrishnan Sowrirajan 
wrote:

> Rui,
>
> I assume the current proposal would also handle the case of mixed mode
> (BATCH + STREAMING within the same app) in the future, right?
>
> Regards
> Venkat
>
> On Wed, Mar 27, 2024 at 10:15 AM Venkatakrishnan Sowrirajan <
> vsowr...@asu.edu> wrote:
>
>> This will be a very useful addition to Flink UI. Thanks Rui for starting
>> a FLIP for this improvement.
>>
>> Regards
>> Venkata krishnan
>>
>>
>> On Wed, Mar 27, 2024 at 4:49 AM Muhammet Orazov
>>  wrote:
>>
>>> Hello Rui,
>>>
>>> Thanks for the proposal! It looks good!
>>>
>>> I have minor clarification from my side:
>>>
>>> The execution mode is also used for the DataStream API [1],
>>> would that also affect/hide the DataStream execution mode
>>> if we remove it from the WebUI?
>>>
>>> Best,
>>> Muhammet
>>>
>>> [1]:
>>>
>>> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/execution_mode/__;!!IKRxdwAv5BmarQ!eFyqVJyje_8hu1vMSUwKGBsj8vqsFDisEvJ5AxPV0LduhhHWF3rPKYEEE-09biA0unFbfMy5AVQZMgBv1AOa5oTHmcYlkUE$
>>>
>>>
>>> On 2024-03-27 06:23, Rui Fan wrote:
>>> > Hi flink developers,
>>> >
>>> > I'd like to start a discussion to discuss FLIP-441:
>>> > Show the JobType and remove Execution Mode on Flink WebUI[1].
>>> >
>>> > Currently, the jobType has 2 types in Flink: STREAMING and BATCH.
>>> > They work on completely different principles, such as: scheduler,
>>> > shuffle, join, etc. These differences lead to different troubleshooting
>>> > processes, so when users are maintaining a job or troubleshooting,
>>> > it's needed to know whether the current job is a STREAMING or
>>> > BATCH job. Unfortunately, Flink WebUI doesn't expose it to the
>>> > users so far.
>>> >
>>> > Also, Execution Mode is related to DataSet api, it has been marked
>>> > as @Deprecated in FLINK-32258 (1.18), but it's still shown in Flink
>>> > WebUI.
>>> >
>>> > Looking forward to hearing more thoughts about it! Thank you~
>>> >
>>> > [1]
>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/agrPEQ__;!!IKRxdwAv5BmarQ!eFyqVJyje_8hu1vMSUwKGBsj8vqsFDisEvJ5AxPV0LduhhHWF3rPKYEEE-09biA0unFbfMy5AVQZMgBv1AOa5oTHayPyFj8$
>>> > [2]
>>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-32558__;!!IKRxdwAv5BmarQ!eFyqVJyje_8hu1vMSUwKGBsj8vqsFDisEvJ5AxPV0LduhhHWF3rPKYEEE-09biA0unFbfMy5AVQZMgBv1AOa5oTHftYeOLE$
>>> >
>>> > Best,
>>> > Rui
>>>
>>

[jira] [Created] (FLINK-34957) JDBC Autoscaler event handler throws Column 'message' cannot be null

[Rui Fan (Jira)]

Rui Fan created FLINK-34957:
---

 Summary: JDBC Autoscaler event handler throws Column 'message' 
cannot be null 
 Key: FLINK-34957
 URL: https://issues.apache.org/jira/browse/FLINK-34957
 Project: Flink
  Issue Type: Bug
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.9.0
 Attachments: image-2024-03-28-11-57-35-234.png

JDBC Autoscaler event handler doesn't allow the event message is null, but the 
message may be null when we handle the exception.

We consider the exception message as the event message, but the exception 
message may be null, such as: TimeoutException. (It has been shown in following 
picture.)

Also, ecording a event without any message is meaningless. It doesn't have any 
benefit for troubleshooting.

Solution: 
* Consider the exception message as the event message when exception message 
isn't null
* The whole Exception as the event message if exception message is null.

 !image-2024-03-28-11-57-35-234.png! 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34956) The config type is wrong for Duration

[Rui Fan (Jira)]

Rui Fan created FLINK-34956:
---

 Summary: The config type is wrong for Duration
 Key: FLINK-34956
 URL: https://issues.apache.org/jira/browse/FLINK-34956
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.20.0
 Attachments: image-2024-03-28-11-21-31-802.png

The Config type is Boolean, but it should be Duration.

https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/

 !image-2024-03-28-11-21-31-802.png! 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

[Rui Fan]

Hi flink developers,

I'd like to start a discussion to discuss FLIP-441:
Show the JobType and remove Execution Mode on Flink WebUI[1].

Currently, the jobType has 2 types in Flink: STREAMING and BATCH.
They work on completely different principles, such as: scheduler,
shuffle, join, etc. These differences lead to different troubleshooting
processes, so when users are maintaining a job or troubleshooting,
it's needed to know whether the current job is a STREAMING or
BATCH job. Unfortunately, Flink WebUI doesn't expose it to the
users so far.

Also, Execution Mode is related to DataSet api, it has been marked
as @Deprecated in FLINK-32258 (1.18), but it's still shown in Flink WebUI.

Looking forward to hearing more thoughts about it! Thank you~

[1] https://cwiki.apache.org/confluence/x/agrPEQ
[2] https://issues.apache.org/jira/browse/FLINK-32558

Best,
Rui

Re: [DISCUSS] Planning Flink 1.20

[Rui Fan]

Welcome Ufuk and Robert join the release manager group!

Best,
Rui

Márton Balassi 于2024年3月25日 周一20:04写道：

> Thanks for kicking this off, folks.
>
> +1 for the timeline and the release manager candidates (Weijie, Rui
> Fan,Ufuk/Robert).
>
> On Mon, Mar 25, 2024 at 12:54 PM Leonard Xu  wrote:
>
> > Wow, happy to see Ufuk and Robert join the release managers group.
> >
> > +1 for the release manager candidates(Weijie, Rui Fan,Ufuk and Robert)
> > from my side.
> >
> >
> > Best,
> > Leonard
> >
> >
> >
> > > 2024年3月25日 下午6:09，Robert Metzger  写道：
> > >
> > > Hi, thanks for starting the discussion.
> > >
> > > +1 for the proposed timeline and the three proposed release managers.
> > >
> > > I'm happy to join the release managers group as well, as a backup for
> > Ufuk
> > > (unless there are objections about the number of release managers)
> > >
> > > On Mon, Mar 25, 2024 at 11:04 AM Ufuk Celebi  wrote:
> > >
> > >> Hey all,
> > >>
> > >> I'd like to join the release managers for 1.20 as well. I'm looking
> > >> forward to getting more actively involved again.
> > >>
> > >> Cheers,
> > >>
> > >> Ufuk
> > >>
> > >> On Sun, Mar 24, 2024, at 11:27 AM, Ahmed Hamdy wrote:
> > >>> +1 for the proposed timeline and release managers.
> > >>> Best Regards
> > >>> Ahmed Hamdy
> > >>>
> > >>>
> > >>> On Fri, 22 Mar 2024 at 07:41, Xintong Song 
> > >> wrote:
> > >>>
> > >>>> +1 for the proposed timeline and Weijie & Rui as the release
> managers.
> > >>>>
> > >>>> I think it would be welcomed if another 1-2 volunteers join as the
> > >> release
> > >>>> managers, but that's not a must. We used to have only 1-2 release
> > >> managers
> > >>>> for each release,
> > >>>>
> > >>>> Best,
> > >>>>
> > >>>> Xintong
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:
> > >>>>
> > >>>>> Thanks for kicking this off.
> > >>>>>
> > >>>>> +1 for the volunteered release managers (Weijie Guo, Rui Fan) and
> the
> > >>>>> targeting date (feature freeze: June 15).
> > >>>>>
> > >>>>> Best,
> > >>>>> Jark
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com>
> wrote:
> > >>>>>
> > >>>>>> Thanks Leonard for this feedback and help!
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Rui
> > >>>>>>
> > >>>>>> On Fri, Mar 22, 2024 at 12:36 PM weijie guo <
> > >> guoweijieres...@gmail.com
> > >>>>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks Leonard!
> > >>>>>>>
> > >>>>>>>> I'd like to help you if you need some help like permissions
> > >> from
> > >>>> PMC
> > >>>>>>> side, please feel free to ping me.
> > >>>>>>>
> > >>>>>>> Nice to know. It'll help a lot!
> > >>>>>>>
> > >>>>>>> Best regards,
> > >>>>>>>
> > >>>>>>> Weijie
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Leonard Xu  于2024年3月22日周五 12:09写道：
> > >>>>>>>
> > >>>>>>>> +1 for the proposed release managers (Weijie Guo, Rui Fan),
> > >> both the
> > >>>>> two
> > >>>>>>>> candidates are pretty active committers thus I believe they
> > >> know the
> > >>>>>>>> community development process well. The recent releases have
> > >> four
> > >>>>>> release
> > >>>>>>>> managers, and I am als

Re: [DISCUSS] Flink Website Menu Adjustment

[Rui Fan]

Thanks Zhongqiang for driving this discussion, +1 for the proposal

Best,
Rui

Jingsong Li 于2024年3月25日 周一22:47写道：

> +1 for the proposal
>
> On Mon, Mar 25, 2024 at 10:01 PM Ferenc Csaky
>  wrote:
> >
> > Suggested changes makes sense, +1 for the proposed menus and order.
> >
> > Best,
> > Ferenc
> >
> >
> >
> >
> > On Monday, March 25th, 2024 at 14:50, Gyula Fóra 
> wrote:
> >
> > >
> > >
> > > +1 for the proposal
> > >
> > > Gyula
> > >
> > > On Mon, Mar 25, 2024 at 12:49 PM Leonard Xu xbjt...@gmail.com wrote:
> > >
> > > > Thanks Zhongqiang for starting this discussion, updating
> documentation
> > > > menus according to sub-projects' activities makes sense to me.
> > > >
> > > > +1 for the proposed menus:
> > > >
> > > > > After:
> > > > >
> > > > > With Flink
> > > > > With Flink Kubernetes Operator
> > > > > With Flink CDC
> > > > > With Flink ML
> > > > > With Flink Stateful Functions
> > > > > Training Course
> > > >
> > > > Best,
> > > > Leonard
> > > >
> > > > > 2024年3月25日 下午3:48，gongzhongqiang gongzhongqi...@apache.org 写道：
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to start a discussion on adjusting the Flink website [1]
> menu to
> > > > > improve accuracy and usability.While migrating Flink CDC
> documentation
> > > > > to the website, I found outdated links, need to review and update
> menus
> > > > > for the most relevant information for our users.
> > > > >
> > > > > Proposal:
> > > > >
> > > > > - Remove Paimon [2] from the "Getting Started" and "Documentation"
> menus:
> > > > > Paimon [2] is now an independent top project of ASF. CC： jingsong
> lees
> > > > >
> > > > > - Sort the projects in the subdirectory by the activity of the
> projects.
> > > > > Here I list the number of releases for each project in the past
> year.
> > > > >
> > > > > Flink Kubernetes Operator : 7
> > > > > Flink CDC : 5
> > > > > Flink ML : 2
> > > > > Flink Stateful Functions : 1
> > > > >
> > > > > Expected Outcome :
> > > > >
> > > > > - Menu "Getting Started"
> > > > >
> > > > > Before:
> > > > >
> > > > > With Flink
> > > > >
> > > > > With Flink Stateful Functions
> > > > >
> > > > > With Flink ML
> > > > >
> > > > > With Flink Kubernetes Operator
> > > > >
> > > > > With Paimon(incubating) (formerly Flink Table Store)
> > > > >
> > > > > With Flink CDC
> > > > >
> > > > > Training Course
> > > > >
> > > > > After:
> > > > >
> > > > > With Flink
> > > > > With Flink Kubernetes Operator
> > > > >
> > > > > With Flink CDC
> > > > >
> > > > > With Flink ML
> > > > >
> > > > > With Flink Stateful Functions
> > > > >
> > > > > Training Course
> > > > >
> > > > > - Menu "Documentation" will same with "Getting Started"
> > > > >
> > > > > I look forward to hearing your thoughts and suggestions on this
> proposal.
> > > > >
> > > > > [1] https://flink.apache.org/
> > > > > [2] https://github.com/apache/incubator-paimon
> > > > > [3] https://github.com/apache/flink-statefun
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Zhongqiang Gong
>

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.8.0 released

[Rui Fan]

Congratulations! Thanks Max for the release and all involved for the great
work!

A gentle reminder to users: the maven artifact has just been released and
will take some time to complete.

Best,
Rui

On Mon, Mar 25, 2024 at 6:35 PM Maximilian Michels  wrote:

> The Apache Flink community is very happy to announce the release of
> the Apache Flink Kubernetes Operator version 1.8.0.
>
> The Flink Kubernetes Operator allows users to manage their Apache
> Flink applications on Kubernetes through all aspects of their
> lifecycle.
>
> Release highlights:
> - Flink Autotuning automatically adjusts TaskManager memory
> - Flink Autoscaling metrics and decision accuracy improved
> - Improve standalone Flink Autoscaling
> - Savepoint trigger nonce for savepoint-based restarts
> - Operator stability improvements for cluster shutdown
>
> Blog post:
> https://flink.apache.org/2024/03/21/apache-flink-kubernetes-operator-1.8.0-release-announcement/
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Flink Kubernetes Operator can be found at:
>
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>
> Official Docker image for Flink Kubernetes Operator can be found at:
> https://hub.docker.com/r/apache/flink-kubernetes-operator
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12353866=12315522
>
> We would like to thank the Apache Flink community and its contributors
> who made this release possible!
>
> Cheers,
> Max
>

Re: [DISCUSS] Planning Flink 1.20

[Rui Fan]

Thanks Leonard for this feedback and help!

Best,
Rui

On Fri, Mar 22, 2024 at 12:36 PM weijie guo 
wrote:

> Thanks Leonard!
>
> > I'd like to help you if you need some help like permissions from PMC
> side, please feel free to ping me.
>
> Nice to know. It'll help a lot!
>
> Best regards,
>
> Weijie
>
>
> Leonard Xu  于2024年3月22日周五 12:09写道：
>
>> +1 for the proposed release managers (Weijie Guo, Rui Fan), both the two
>> candidates are pretty active committers thus I believe they know the
>> community development process well. The recent releases have four release
>> managers, and I am also looking forward to having other volunteers
>>  join the management of Flink 1.20.
>>
>> +1 for targeting date (feature freeze: June 15, 2024), referring to the
>> release cycle of recent versions, release cycle of 4 months makes sense to
>> me.
>>
>>
>> I'd like to help you if you need some help like permissions from PMC
>> side, please feel free to ping me.
>>
>> Best,
>> Leonard
>>
>>
>> > 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道：
>> >
>> > Hi Weijie,
>> >
>> > Thanks for kicking off 1.20! I'd like to join you and participate in the
>> > 1.20 release.
>> >
>> > Best,
>> > Rui
>> >
>> > On Tue, Mar 19, 2024 at 5:30 PM weijie guo 
>> > wrote:
>> >
>> >> Hi everyone,
>> >>
>> >> With the release announcement of Flink 1.19, it's a good time to kick
>> off
>> >> discussion of the next release 1.20.
>> >>
>> >>
>> >> - Release managers
>> >>
>> >>
>> >> I'd like to volunteer as one of the release managers this time. It has
>> been
>> >> good practice to have a team of release managers from different
>> >> backgrounds, so please raise you hand if you'd like to volunteer and
>> get
>> >> involved.
>> >>
>> >>
>> >>
>> >> - Timeline
>> >>
>> >>
>> >> Flink 1.19 has been released. With a target release cycle of 4 months,
>> >> we propose a feature freeze date of *June 15, 2024*.
>> >>
>> >>
>> >>
>> >> - Collecting features
>> >>
>> >>
>> >> As usual, we've created a wiki page[1] for collecting new features in
>> 1.20.
>> >>
>> >>
>> >> In addition, we already have a number of FLIPs that have been voted or
>> are
>> >> in the process, including pre-works for version 2.0.
>> >>
>> >>
>> >> In the meantime, the release management team will be finalized in the
>> next
>> >> few days, and we'll continue to create Jira Boards and Sync meetings
>> >> to make it easy
>> >> for everyone to get an overview and track progress.
>> >>
>> >>
>> >>
>> >> Best regards,
>> >>
>> >> Weijie
>> >>
>> >>
>> >>
>> >> [1] https://cwiki.apache.org/confluence/display/FLINK/1.20+Release
>> >>
>>
>>

[jira] [Created] (FLINK-34907) jobRunningTs should be the timestamp that all tasks are running

[Rui Fan (Jira)]

Rui Fan created FLINK-34907:
---

 Summary: jobRunningTs should be the timestamp that all tasks are 
running
 Key: FLINK-34907
 URL: https://issues.apache.org/jira/browse/FLINK-34907
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.9.0


Currently, we consider the timestamp that JobStatus is changed to RUNNING as 
jobRunningTs. But the JobStatus will be RUNNING once job starts schedule, so it 
doesn't mean all tasks are running. 

It will let the isStabilizing or estimating restart time are not accurate.

Solution: jobRunningTs should be the timestamp that all tasks are running.

It can be got from SubtasksTimesHeaders rest api.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34906) Don't start autoscaling when some tasks are not running

[Rui Fan (Jira)]

Rui Fan created FLINK-34906:
---

 Summary: Don't start autoscaling when some tasks are not running
 Key: FLINK-34906
 URL: https://issues.apache.org/jira/browse/FLINK-34906
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.9.0
 Attachments: image-2024-03-21-17-40-23-523.png

Currently, the autoscaler will scale a job when the JobStatus is RUNNING. But 
the JobStatus will be RUNNING once job starts schedule, so it doesn't mean all 
tasks are running. Especially, when the resource isn't enough or job recovers 
from large state.

The autoscaler will throw exception and generate the AutoscalerError event when 
tasks are not ready, such as: 

 !image-2024-03-21-17-40-23-523.png! 


Solution: we only scale job that all tasks are running(some of tasks may be 
finished). 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

[Rui Fan]

Congratulations!

Best,
Rui

On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan  wrote:

> Congrattulations!
>
> Best,
> Hang
>
> Lincoln Lee  于2024年3月21日周四 09:54写道：
>
>>
>> Congrats, thanks for the great work!
>>
>>
>> Best,
>> Lincoln Lee
>>
>>
>> Peter Huang  于2024年3月20日周三 22:48写道：
>>
>>> Congratulations
>>>
>>>
>>> Best Regards
>>> Peter Huang
>>>
>>> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang  wrote:
>>>

 Congratulations



 Best,
 Huajie Wang



 Leonard Xu  于2024年3月20日周三 21:36写道：

> Hi devs and users,
>
> We are thrilled to announce that the donation of Flink CDC as a
> sub-project of Apache Flink has completed. We invite you to explore the 
> new
> resources available:
>
> - GitHub Repository: https://github.com/apache/flink-cdc
> - Flink CDC Documentation:
> https://nightlies.apache.org/flink/flink-cdc-docs-stable
>
> After Flink community accepted this donation[1], we have completed
> software copyright signing, code repo migration, code cleanup, website
> migration, CI migration and github issues migration etc.
> Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> Qingsheng Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors
> for their contributions and help during this process！
>
>
> For all previous contributors: The contribution process has slightly
> changed to align with the main Flink project. To report bugs or suggest 
> new
> features, please open tickets
> Apache Jira (https://issues.apache.org/jira).  Note that we will no
> longer accept GitHub issues for these purposes.
>
>
> Welcome to explore the new repository and documentation. Your feedback
> and contributions are invaluable as we continue to improve Flink CDC.
>
> Thanks everyone for your support and happy exploring Flink CDC!
>
> Best，
> Leonard
> [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
>
>

Re: [VOTE] FLIP-433: State Access on DataStream API V2

[Rui Fan]

+1(binding)

Thanks to Weijie for driving this proposal, which solves the problem that I
raised in FLIP-359.

Best,
Rui

On Thu, Mar 21, 2024 at 10:10 AM Hangxiang Yu  wrote:

> +1 (binding)
>
> On Thu, Mar 21, 2024 at 10:04 AM Xintong Song 
> wrote:
>
> > +1 (binding)
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Mar 20, 2024 at 8:30 PM weijie guo 
> > wrote:
> >
> > > Hi everyone,
> > >
> > >
> > > Thanks for all the feedback about the FLIP-433: State Access on
> > > DataStream API V2 [1]. The discussion thread is here [2].
> > >
> > >
> > > The vote will be open for at least 72 hours unless there is an
> > > objection or insufficient votes.
> > >
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-433%3A+State+Access+on+DataStream+API+V2
> > >
> > > [2] https://lists.apache.org/thread/8ghzqkvt99p4k7hjqxzwhqny7zb7xrwo
> > >
> >
>
>
> --
> Best,
> Hangxiang.
>

[jira] [Created] (FLINK-34744) autoscaling-dynamic cannot run

[Rui Fan (Jira)]

Rui Fan created FLINK-34744:
---

 Summary: autoscaling-dynamic cannot run
 Key: FLINK-34744
 URL: https://issues.apache.org/jira/browse/FLINK-34744
 Project: Flink
  Issue Type: Bug
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.9.0
 Attachments: image-2024-03-19-21-46-15-530.png

autoscaling-dynamic cannot run on my Mac

 !image-2024-03-19-21-46-15-530.png! 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34743) Memory tuning takes effect even if the parallelism isn't changed

[Rui Fan (Jira)]

Rui Fan created FLINK-34743:
---

 Summary: Memory tuning takes effect even if the parallelism isn't 
changed
 Key: FLINK-34743
 URL: https://issues.apache.org/jira/browse/FLINK-34743
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.9.0


Currently, the memory tuning related logic is only called when the parallelism 
is changed.

See ScalingExecutor#scaleResource to get more details.

It's better to let the memory tuning takes effect even if the parallelism isn't 
changed. For example, one flink job runs with desired parallelisms, but it 
wastes memory.





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Planning Flink 1.20

[Rui Fan]

Hi Weijie,

Thanks for kicking off 1.20! I'd like to join you and participate in the
1.20 release.

Best,
Rui

On Tue, Mar 19, 2024 at 5:30 PM weijie guo 
wrote:

> Hi everyone,
>
> With the release announcement of Flink 1.19, it's a good time to kick off
> discussion of the next release 1.20.
>
>
> - Release managers
>
>
> I'd like to volunteer as one of the release managers this time. It has been
> good practice to have a team of release managers from different
> backgrounds, so please raise you hand if you'd like to volunteer and get
> involved.
>
>
>
> - Timeline
>
>
> Flink 1.19 has been released. With a target release cycle of 4 months,
> we propose a feature freeze date of *June 15, 2024*.
>
>
>
> - Collecting features
>
>
> As usual, we've created a wiki page[1] for collecting new features in 1.20.
>
>
> In addition, we already have a number of FLIPs that have been voted or are
> in the process, including pre-works for version 2.0.
>
>
> In the meantime, the release management team will be finalized in the next
> few days, and we'll continue to create Jira Boards and Sync meetings
> to make it easy
> for everyone to get an overview and track progress.
>
>
>
> Best regards,
>
> Weijie
>
>
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/1.20+Release
>

Re: [ANNOUNCE] Apache Flink 1.19.0 released

[Rui Fan]

Congratulations, thanks for the great work!

Best,
Rui

On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee  wrote:

> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Please check out the release blog post for an overview of the improvements
> for this bugfix release:
>
> https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
>
> Best,
> Yun, Jing, Martijn and Lincoln
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.8.0, release candidate #1

[Rui Fan]

Thanks Max for driving this release!

+1(non-binding)

- Downloaded artifacts from dist ( svn co
https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/
)
- Verified SHA512 checksums : ( for i in *.tgz; do echo $i; sha512sum
--check $i.sha512; done )
- Verified GPG signatures : ( $ for i in *.tgz; do echo $i; gpg --verify
$i.asc $i )
- Build the source with java-11 and java-17 ( mvn -T 20 clean install
-DskipTests )
- Verified the license header during build the source
- Verified that chart and appVersion matches the target release (less the
index.yaml and Chart.yaml )
- RC repo works as Helm repo( helm repo add flink-operator-repo-1.8.0-rc1
https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/
)
- Verified Helm chart can be installed  ( helm install
flink-kubernetes-operator
flink-operator-repo-1.8.0-rc1/flink-kubernetes-operator --set
webhook.create=false )
- Submitted the autoscaling demo, the autoscaler works well with *memory
tuning *(kubectl apply -f autoscaling.yaml)
   - job.autoscaler.memory.tuning.enabled: "true"
- Download Autoscaler standalone: wget
https://repository.apache.org/content/repositories/orgapacheflink-1710/org/apache/flink/flink-autoscaler-standalone/1.8.0/flink-autoscaler-standalone-1.8.0.jar
- Ran Autoscaler standalone locally, it works well with rescale api and
JDBC state store/event handler

Best,
Rui

On Fri, Mar 15, 2024 at 1:45 AM Maximilian Michels  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version
> 1.8.0 of the Apache Flink Kubernetes Operator, as follows:
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Kubernetes Operator canonical source distribution (including the
> Dockerfile), to be deployed to the release repository at
> dist.apache.org
> b) Kubernetes Operator Helm Chart to be deployed to the release
> repository at dist.apache.org
> c) Maven artifacts to be deployed to the Maven Central Repository
> d) Docker image to be pushed to Dockerhub
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as
> follows, for your review:
> * All artifacts for (a), (b) can be found in the corresponding dev
> repository at dist.apache.org [1]
> * All artifacts for (c) can be found at the Apache Nexus Repository [2]
> * The docker image for (d) is staged on github [3]
>
> All artifacts are signed with the key
> DA359CBFCEB13FC302A8793FB655E6F7693D5FDE [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-1.8.0-rc1" [6]
> * PR to update the website Downloads page to include Kubernetes
> Operator links [7]
>
> **Vote Duration**
>
> The voting time will run for at least 72 hours. It is adopted by
> majority approval, with at least 3 PMC affirmative votes.
>
> **Note on Verification**
>
> You can follow the basic verification guide here [8]. Note that you
> don't need to verify everything yourself, but please make note of what
> you have tested together with your +- vote.
>
> Thanks,
> Max
>
> [1]
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/
> [2]
> https://repository.apache.org/content/repositories/orgapacheflink-1710/
> [3] ghcr.io/apache/flink-kubernetes-operator:8938658
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12353866=12315522
> [6]
> https://github.com/apache/flink-kubernetes-operator/tree/release-1.8.0-rc1
> [7] https://github.com/apache/flink-web/pull/726
> [8]
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Kubernetes+Operator+Release
>

[jira] [Created] (FLINK-34655) Autoscaler doesn't work for flink 1.15

[Rui Fan (Jira)]

Rui Fan created FLINK-34655:
---

 Summary: Autoscaler doesn't work for flink 1.15
 Key: FLINK-34655
 URL: https://issues.apache.org/jira/browse/FLINK-34655
 Project: Flink
  Issue Type: Bug
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.8.0


flink-ubernetes-operator is committed to supporting the latest 4 flink minor 
versions, and autoscaler is a part of flink-ubernetes-operator. Currently,  the 
latest 4 flink minor versions are 1.15, 1.16, 1.17 and 1.18.

But autoscaler doesn't work for  flink 1.15.

h2. Root cause: 

* FLINK-28310 added some properties in IOMetricsInfo in flink-1.16
* IOMetricsInfo is a part of JobDetailsInfo
* JobDetailsInfo is necessary for autoscaler [1]
* flink's RestClient doesn't allow miss any property during deserializing the 
json

That means that the RestClient after 1.15 cannot fetch JobDetailsInfo for 1.15 
jobs.

h2. How to fix it properly?

Flink side support ignore unknown properties.

FLINK-33268 already do it. But I try run autoscaler with flink-1.15 job, it 
still doesn't work. Because the IOMetricsInfo added some properties, they are 
primitive type.

It should disable DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES as well. 
(Not sure whether it should be a seperate FLIP or it can be a part of FLIP-401 
[2].)


h2. How to fix it in the short term?

1. Copy the latest RestMapperUtils and RestClient from master branch (It 
includes FLINK-33268) to flink-autoscaler module. (The copied class will be 
loaded first)
2. Disable DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES in 
RestMapperUtils#flexibleObjectMapper in copied class.

Based on these 2 steps, flink-1.15 works well with autoscaler. (I try it 
locally).


After DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES in 
RestMapperUtils#flexibleObjectMapper is disabled, and the corresponding code is 
released in flink side. flink-ubernetes-operator can remove these 2 copied 
classes.

[1] 
https://github.com/apache/flink-kubernetes-operator/blob/ede1a610b3375d31a2e82287eec67ace70c4c8df/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingMetricCollector.java#L109
[2] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-401%3A+REST+API+JSON+response+deserialization+unknown+field+tolerance



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release 1.19.0, release candidate #2

[Rui Fan]

+1 (non-binding)

- Verified signature
- Verified checksum
- Built source code successfully
- Reviewed the release PR and some comments are updated

Best,
Rui

On Tue, Mar 12, 2024 at 4:08 PM Xuannan Su  wrote:

> +1 (non-binding)
>
> - Verified signature and checksum
> - Verified that source distribution does not contain binaries
> - Built from source code successfully
> - Reviewed the release announcement PR
>
> Best regards,
> Xuannan
>
> On Tue, Mar 12, 2024 at 2:18 PM Hang Ruan  wrote:
> >
> > +1 (non-binding)
> >
> > - Verified signatures and checksums
> > - Verified that source does not contain binaries
> > - Build source code successfully
> > - Reviewed the release note and left a comment
> >
> > Best,
> > Hang
> >
> > Feng Jin  于2024年3月12日周二 11:23写道：
> >
> > > +1 (non-binding)
> > >
> > > - Verified signatures and checksums
> > > - Verified that source does not contain binaries
> > > - Build source code successfully
> > > - Run a simple sql query successfully
> > >
> > > Best,
> > > Feng Jin
> > >
> > >
> > > On Tue, Mar 12, 2024 at 11:09 AM Ron liu  wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > quickly verified:
> > > > - verified that source distribution does not contain binaries
> > > > - verified checksums
> > > > - built source code successfully
> > > >
> > > >
> > > > Best,
> > > > Ron
> > > >
> > > > Jeyhun Karimov  于2024年3月12日周二 01:00写道：
> > > >
> > > > > +1 (non binding)
> > > > >
> > > > > - verified that source distribution does not contain binaries
> > > > > - verified signatures and checksums
> > > > > - built source code successfully
> > > > >
> > > > > Regards,
> > > > > Jeyhun
> > > > >
> > > > >
> > > > > On Mon, Mar 11, 2024 at 3:08 PM Samrat Deb 
> > > > wrote:
> > > > >
> > > > > > +1 (non binding)
> > > > > >
> > > > > > - verified signatures and checksums
> > > > > > - ASF headers are present in all expected file
> > > > > > - No unexpected binaries files found in the source
> > > > > > - Build successful locally
> > > > > > - tested basic word count example
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Bests,
> > > > > > Samrat
> > > > > >
> > > > > > On Mon, 11 Mar 2024 at 7:33 PM, Ahmed Hamdy <
> hamdy10...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Lincoln
> > > > > > > +1 (non-binding) from me
> > > > > > >
> > > > > > > - Verified Checksums & Signatures
> > > > > > > - Verified Source dists don't contain binaries
> > > > > > > - Built source successfully
> > > > > > > - reviewed web PR
> > > > > > >
> > > > > > >
> > > > > > > Best Regards
> > > > > > > Ahmed Hamdy
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 11 Mar 2024 at 15:18, Lincoln Lee <
> lincoln.8...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Robin,
> > > > > > > >
> > > > > > > > Thanks for helping verifying the release note[1], FLINK-14879
> > > > should
> > > > > > not
> > > > > > > > have been included, after confirming this
> > > > > > > > I moved all unresolved non-blocker issues left over from
> 1.19.0
> > > to
> > > > > > 1.20.0
> > > > > > > > and reconfigured the release note [1].
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Lincoln Lee
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282
> > > > > > > >
> > > > > > > >
> > > > > > > > Robin Moffatt  于2024年3月11日周一
> > > 19:36写道：
> > > > > > > >
> > > > > > > > > Looking at the release notes [1] it lists `DESCRIBE
> DATABASE`
> > > > > > > > (FLINK-14879)
> > > > > > > > > and `DESCRIBE CATALOG` (FLINK-14690).
> > > > > > > > > When I try these in 1.19 RC2 the behaviour is as in 1.18.1,
> > > i.e.
> > > > it
> > > > > > is
> > > > > > > > not
> > > > > > > > > supported:
> > > > > > > > >
> > > > > > > > > ```
> > > > > > > > > [INFO] Execute statement succeed.
> > > > > > > > >
> > > > > > > > > Flink SQL> show catalogs;
> > > > > > > > > +-+
> > > > > > > > > |catalog name |
> > > > > > > > > +-+
> > > > > > > > > |   c_new |
> > > > > > > > > | default_catalog |
> > > > > > > > > +-+
> > > > > > > > > 2 rows in set
> > > > > > > > >
> > > > > > > > > Flink SQL> DESCRIBE CATALOG c_new;
> > > > > > > > > [ERROR] Could not execute SQL statement. Reason:
> > > > > > > > > org.apache.calcite.sql.validate.SqlValidatorException:
> Column
> > > > > 'c_new'
> > > > > > > not
> > > > > > > > > found in any table
> > > > > > > > >
> > > > > > > > > Flink SQL> show databases;
> > > > > > > > > +--+
> > > > > > > > > |database name |
> > > > > > > > > +--+
> > > > > > > > > | default_database |
> > > > > > > > > +--+
> > > > > > > > > 1 row in set
> > > > > > > > >
> > > > > > > > > Flink SQL> DESCRIBE DATABASE default_database;
> > > > > > > > > [ERROR] Could not execute SQL statement. Reason:
> > > > > > > > > 

[jira] [Created] (FLINK-34522) StateTtlConfig#cleanupInRocksdbCompactFilter still use the deprecated Time

[Rui Fan (Jira)]

Rui Fan created FLINK-34522:
---

 Summary: StateTtlConfig#cleanupInRocksdbCompactFilter still use 
the deprecated Time
 Key: FLINK-34522
 URL: https://issues.apache.org/jira/browse/FLINK-34522
 Project: Flink
  Issue Type: Improvement
  Components: API / Core
Affects Versions: 1.19.0
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.19.0, 1.20.0


FLINK-32570 deprecated the Time class and refactor all Public or PublicEvolving 
apis to use the Java's Duration.

StateTtlConfig.Builder#cleanupInRocksdbCompactFilter is still using the Time 
class. In general, we expect:
 * Mark it as @Deprecated  
 * Provide a new cleanupInRocksdbCompactFilter(long, Duration)

 

But I found this method is introduced in 1.19, so a better solution may be: 
only provide cleanupInRocksdbCompactFilter(long, Duration) and don't use Time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34521) Using the Duration instead of the deprecated Time classes

[Rui Fan (Jira)]

Rui Fan created FLINK-34521:
---

 Summary: Using the Duration instead of the deprecated Time classes
 Key: FLINK-34521
 URL: https://issues.apache.org/jira/browse/FLINK-34521
 Project: Flink
  Issue Type: Technical Debt
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.20.0


FLINK-32570 deprecated org.apache.flink.api.common.time.Time and 
org.apache.flink.streaming.api.windowing.time.Time.

We should refactor all internal callers from Time to Duration. (Public callers 
should be removed in 2.0).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Confluence access request

[Rui Fan]

Hi tanjialiang,

The Flink community recently discussed how non-flink-committer create
FLIP[1],
the process has been updated in the wiki[2], please check the "Create your
Own FLIP"
part, thank you.

[1] https://lists.apache.org/thread/rkpvlnwj9gv1hvx1dyklx6k88qpnvk2t
[2]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

Best,
Rui

On Tue, Feb 27, 2024 at 10:14 AM jialiang tan 
wrote:

> Hi, devs!
> I want to prepare a FLIP and start a discussion on the dev mailing list,
> but I find I don't have the access, can someone give me access to
> confluence?
>
> My Confluence username: tanjialiang
>
> Best regards,
> tanjialiang
>

Re: [VOTE] FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction

[Rui Fan]

+1(binding)

Best,
Rui

On Tue, Feb 27, 2024 at 9:44 AM weijie guo 
wrote:

> +1(binding)
>
> Best regards,
>
> Weijie
>
>
> Xintong Song  于2024年2月27日周二 09:38写道：
>
> > +1 (binding)
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Mon, Feb 26, 2024 at 6:09 PM weijie guo 
> > wrote:
> >
> > > Hi everyone,
> > >
> > >
> > > Thanks for all the feedback about the FLIP-409: DataStream V2 Building
> > > Blocks: DataStream, Partitioning and ProcessFunction [1]. The
> > > discussion thread is here [2].
> > >
> > >
> > > The vote will be open for at least 72 hours unless there is an
> > > objection or insufficient votes.
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-409%3A+DataStream+V2+Building+Blocks%3A+DataStream%2C+Partitioning+and+ProcessFunction
> > >
> > > [2] https://lists.apache.org/thread/cwds0bwbgy3lfdgnlqbfhm6lfvx2qbrv
> > >
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> >
>

Re: [VOTE] FLIP-410: Config, Context and Processing Timer Service of DataStream API V2

[Rui Fan]

+1(binding)

Best,
Rui

On Tue, Feb 27, 2024 at 9:45 AM weijie guo 
wrote:

> +1(binding)
>
> Best regards,
>
> Weijie
>
>
> Xintong Song  于2024年2月27日周二 09:37写道：
>
> > +1 (binding)
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Mon, Feb 26, 2024 at 6:10 PM weijie guo 
> > wrote:
> >
> > > Hi everyone,
> > >
> > >
> > > Thanks for all the feedback about the FLIP-410: Config, Context and
> > > Processing Timer Service of DataStream API V2 [1]. The discussion
> > > thread is here [2].
> > >
> > >
> > > The vote will be open for at least 72 hours unless there is an
> > > objection or insufficient votes.
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-410%3A++Config%2C+Context+and+Processing+Timer+Service+of+DataStream+API+V2
> > >
> > > [2] https://lists.apache.org/thread/70gf028c5gsdb9qhsgpht0chzyp9nogc
> > >
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> >
>

Re: [VOTE] FLIP-408: [Umbrella] Introduce DataStream API V2

[Rui Fan]

+1(binding)

Best,
Rui

On Tue, Feb 27, 2024 at 9:43 AM weijie guo 
wrote:

> +1(binding)
>
> Best regards,
>
> Weijie
>
>
> Xintong Song  于2024年2月27日周二 09:36写道：
>
> > +1 (binding)
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Mon, Feb 26, 2024 at 6:08 PM weijie guo 
> > wrote:
> >
> > > Hi everyone,
> > >
> > >
> > > Thanks for all the feedback about the FLIP-408: [Umbrella] Introduce
> > > DataStream API V2 [1]. The discussion thread is here [2].
> > >
> > >
> > > The vote will be open for at least 72 hours unless there is an
> > > objection or insufficient votes.
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-408%3A+%5BUmbrella%5D+Introduce+DataStream+API+V2
> > >
> > > [2] https://lists.apache.org/thread/w8olky9s7fo5h8fl3nj3qbym307zk2l0
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> >
>

[jira] [Created] (FLINK-34504) Avoid the parallelism adjutment when the upstream shuffle type doesn't has keyBy

[Rui Fan (Jira)]

Rui Fan created FLINK-34504:
---

 Summary: Avoid the parallelism adjutment when the upstream shuffle 
type doesn't has keyBy
 Key: FLINK-34504
 URL: https://issues.apache.org/jira/browse/FLINK-34504
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.8.0


JobVertexScaler#scale has a optimization: Try to adjust the parallelism such 
that it divides the number of key groups without a remainder =>  data is evenly 
spread across subtasks.

 

It's only useful when the upstream shuffle type has keyBy. We should avoid this 
optimization when the upstream shuffle type doesn't has keyBy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34502) Support calculating network memory for forward and rescale edge

[Rui Fan (Jira)]

Rui Fan created FLINK-34502:
---

 Summary: Support calculating network memory for forward and 
rescale edge
 Key: FLINK-34502
 URL: https://issues.apache.org/jira/browse/FLINK-34502
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan


This is follow up Jira of FLINK-34471.

FLINK-34471 assuming all connections type are ALL_TO_ALL. This Jira will 
optimize it to save some network memory for forward and rescale connection.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34471) Tune the network memroy in Autoscaler

[Rui Fan (Jira)]

Rui Fan created FLINK-34471:
---

 Summary: Tune the network memroy in Autoscaler
 Key: FLINK-34471
 URL: https://issues.apache.org/jira/browse/FLINK-34471
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler
Reporter: Rui Fan


Design doc: 
https://docs.google.com/document/d/19HYamwMaYYYOeH3NRbk6l9P-bBLBfgzMYjfGEPWEbeo/edit?usp=sharing



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-parent 1.1.0 release candidate #2

[Rui Fan]

Thanks for driving this, Etienne!

+1 (non-binding)

- Verified checksum and signature
- Verified pom content
- Build source on my Mac with jdk8
- Verified no binaries in source
- Checked staging repo on Maven central
- Checked source code tag
- Reviewed web PR

Best,
Rui

On Tue, Feb 20, 2024 at 10:33 AM Qingsheng Ren  wrote:

> Thanks for driving this, Etienne!
>
> +1 (binding)
>
> - Checked release note
> - Verified checksum and signature
> - Verified pom content
> - Verified no binaries in source
> - Checked staging repo on Maven central
> - Checked source code tag
> - Reviewed web PR
> - Built Kafka connector from source with parent pom in staging repo
>
> Best,
> Qingsheng
>
> On Tue, Feb 20, 2024 at 1:34 AM Etienne Chauchot 
> wrote:
>
> > Hi everyone,
> > Please review and vote on the release candidate #2 for the version
> > 1.1.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > [2], which are signed with the key with fingerprint
> > D1A76BA19D6294DD0033F6843A019F0B8DD163EA [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag v1.1.0-rc2 [5],
> > * website pull request listing the new release [6].
> >
> > * confluence wiki: connector parent upgrade to version 1.1.0 that will
> > be validated after the artifact is released (there is no PR mechanism on
> > the wiki) [7]
> >
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Etienne
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353442
> > [2]
> >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-parent-1.1.0-rc2
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1707
> > [5]
> >
> >
> https://github.com/apache/flink-connector-shared-utils/releases/tag/v1.1.0-rc2
> >
> > [6] https://github.com/apache/flink-web/pull/717
> >
> > [7]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Jiabao Sun

[Rui Fan]

Congratulations, Jiabao!

Best,
Rui

On Mon, Feb 19, 2024 at 6:07 PM weijie guo 
wrote:

> Congratulations, Jiabao :)
>
> Best regards,
>
> Weijie
>
>
> Hang Ruan  于2024年2月19日周一 18:04写道：
>
> > Congratulations, Jiabao!
> >
> > Best,
> > Hang
> >
> > Qingsheng Ren  于2024年2月19日周一 17:53写道：
> >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm happy to announce Jiabao Sun as a new Flink
> > > Committer.
> > >
> > > Jiabao began contributing in August 2022 and has contributed 60+
> commits
> > > for Flink main repo and various connectors. His most notable
> contribution
> > > is being the core author and maintainer of MongoDB connector, which is
> > > fully functional in DataStream and Table/SQL APIs. Jiabao is also the
> > > author of FLIP-377 and the main contributor of JUnit 5 migration in
> > runtime
> > > and table planner modules.
> > >
> > > Beyond his technical contributions, Jiabao is an active member of our
> > > community, participating in the mailing list and consistently
> > volunteering
> > > for release verifications and code reviews with enthusiasm.
> > >
> > > Please join me in congratulating Jiabao for becoming an Apache Flink
> > > committer!
> > >
> > > Best,
> > > Qingsheng (on behalf of the Flink PMC)
> > >
> >
>

Re: [DISCUSS] Kubernetes Operator 1.8.0 release planning

[Rui Fan]

Thanks Max and Ryan for the volunteering.

To Ryan:

I'm not sure whether non-flink-committers have permission to release.
If I remember correctly, multiple steps of the release process[1] need
the apache account, such as: Apache GPG key and Apache Nexus.

If the release process needs the committer permission, feel free to
assist this release, thanks~

To all:

Max is one of the very active contributors to the
flink-kuberneters-operator
project, and he didn't release before. So Max as the release manager
makes sense to me.

I can assist this release if all of you don't mind. In particular,
Autoscaler Standalone 1.8.0 is much improved compared to 1.7.0,
and I can help write the related Release note. Besides, I can help
check and test this release.

[1]
https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Kubernetes+Operator+Release

Best,
Rui

On Wed, Feb 7, 2024 at 11:01 PM Ryan van Huuksloot <
ryan.vanhuuksl...@shopify.com> wrote:

> I can volunteer to be a release manager. I haven't done it for
> Apache/Flink or the operator before so I may be a good candidate.
>
> Ryan van Huuksloot
> Sr. Production Engineer | Streaming Platform
> [image: Shopify]
> <https://www.shopify.com/?utm_medium=salessignatures_source=hs_email>
>
>
> On Wed, Feb 7, 2024 at 6:06 AM Maximilian Michels  wrote:
>
>> It's very considerate that you want to volunteer to be the release
>> manager, but given that you have already managed one release, I would
>> ideally like somebody else to do it. Personally, I haven't managed an
>> operator release, although I've done it for Flink itself in the past.
>> Nevertheless, it would be nice to have somebody new to the process.
>>
>> Anyone reading this who wants to try being a release manager, please
>> don't be afraid to volunteer. Of course we'll be able to assist. That
>> would also be a good opportunity for us to update the docs regarding
>> the release process.
>>
>> Cheers,
>> Max
>>
>>
>> On Wed, Feb 7, 2024 at 10:08 AM Rui Fan <1996fan...@gmail.com> wrote:
>> >
>> > If the release is postponed 1-2 more weeks, I could volunteer
>> > as the one of the release managers.
>> >
>> > Best,
>> > Rui
>> >
>> > On Wed, Feb 7, 2024 at 4:54 AM Gyula Fóra  wrote:
>> >>
>> >> Given the proposed timeline was a bit short / rushed I agree with Max
>> that
>> >> we could wait 1-2 more weeks to wrap up the current outstanding bigger
>> >> features around memory tuning and the JDBC state store.
>> >>
>> >> In the meantime it would be great to involve 1-2 new committers (or
>> other
>> >> contributors) in the operator release process so that we have some
>> fresh
>> >> eyes on the process.
>> >> Would anyone be interested in volunteering to help with the next
>> release?
>> >>
>> >> Cheers,
>> >> Gyula
>> >>
>> >> On Tue, Feb 6, 2024 at 4:35 PM Maximilian Michels 
>> wrote:
>> >>
>> >> > Thanks for starting the discussion Gyula!
>> >> >
>> >> > It comes down to how important the outstanding changes are for the
>> >> > release. Both the memory tuning as well as the JDBC changes probably
>> >> > need 1-2 weeks realistically to complete the initial spec. For the
>> >> > memory tuning, I would prefer merging it in the current state as an
>> >> > experimental feature for the release which comes disabled out of the
>> >> > box. The reason is that it can already be useful to users who want to
>> >> > try it out; we have seen some interest in it. Then for the next
>> >> > release we will offer a richer feature set and might enable it by
>> >> > default.
>> >> >
>> >> > Cheers,
>> >> > Max
>> >> >
>> >> > On Tue, Feb 6, 2024 at 10:53 AM Rui Fan <1996fan...@gmail.com>
>> wrote:
>> >> > >
>> >> > > Thanks Gyula for driving this release!
>> >> > >
>> >> > > Release 1.8.0 sounds make sense to me.
>> >> > >
>> >> > > As you said, I'm developing the JDBC event handler.
>> >> > > Since I'm going on vacation starting this Friday, and I have some
>> >> > > other work before I go on vacation. After evaluating my time today,
>> >> > > I found that I cannot complete the development, testing, and
>> merging
>> >> > > of the JDBC event handler this week. So I 

Re: [VOTE] FLIP-418: Show data skew score on Flink Dashboard

[Rui Fan]

+1(binding)

Thanks for driving this proposal!

Best,
Rui

On Thu, 15 Feb 2024 at 17:47, Danny Cranmer  wrote:

> Thanks for driving thie Emre,
>
> +1 (binding)
>
> Thanks,
> Danny.
>
> On Mon, Feb 12, 2024 at 9:42 PM Hong Liang  wrote:
>
> > +1 (binding)
> >
> > Thank you for driving this Emre! This is a good step towards better user
> > experience when diagnosing performance issues with Flink jobs.
> >
> > Best,
> > Hong
> >
> > On Wed, Jan 31, 2024 at 3:00 AM Aleksandr Pilipenko 
> > wrote:
> >
> > > Thanks for the FLIP!
> > >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Aleksandr
> > >
> > > On Mon, 29 Jan 2024 at 10:11, Kartoglu, Emre
>  > >
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I'd like to call votes on FLIP-418: Show data skew score on Flink
> > > > Dashboard.
> > > >
> > > > FLIP:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-418%3A+Show+data+skew+score+on+Flink+Dashboard
> > > > Discussion:
> > > > https://lists.apache.org/thread/m5ockoork0h2zr78h77dcrn71rbt35ql
> > > >
> > > > Kind regards,
> > > > Emre
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Kubernetes Operator 1.8.0 release planning

[Rui Fan]

If the release is postponed 1-2 more weeks, I could volunteer
as the one of the release managers.

Best,
Rui

On Wed, Feb 7, 2024 at 4:54 AM Gyula Fóra  wrote:

> Given the proposed timeline was a bit short / rushed I agree with Max that
> we could wait 1-2 more weeks to wrap up the current outstanding bigger
> features around memory tuning and the JDBC state store.
>
> In the meantime it would be great to involve 1-2 new committers (or other
> contributors) in the operator release process so that we have some fresh
> eyes on the process.
> Would anyone be interested in volunteering to help with the next release?
>
> Cheers,
> Gyula
>
> On Tue, Feb 6, 2024 at 4:35 PM Maximilian Michels  wrote:
>
> > Thanks for starting the discussion Gyula!
> >
> > It comes down to how important the outstanding changes are for the
> > release. Both the memory tuning as well as the JDBC changes probably
> > need 1-2 weeks realistically to complete the initial spec. For the
> > memory tuning, I would prefer merging it in the current state as an
> > experimental feature for the release which comes disabled out of the
> > box. The reason is that it can already be useful to users who want to
> > try it out; we have seen some interest in it. Then for the next
> > release we will offer a richer feature set and might enable it by
> > default.
> >
> > Cheers,
> > Max
> >
> > On Tue, Feb 6, 2024 at 10:53 AM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > Thanks Gyula for driving this release!
> > >
> > > Release 1.8.0 sounds make sense to me.
> > >
> > > As you said, I'm developing the JDBC event handler.
> > > Since I'm going on vacation starting this Friday, and I have some
> > > other work before I go on vacation. After evaluating my time today,
> > > I found that I cannot complete the development, testing, and merging
> > > of the JDBC event handler this week. So I tend to put the JDBC
> > > event handler in the next version.
> > >
> > > Best,
> > > Rui
> > >
> > > On Mon, Feb 5, 2024 at 11:42 PM Gyula Fóra 
> wrote:
> > >
> > > > Hi all!
> > > >
> > > > I would like to kick off the release planning for the operator 1.8.0
> > > > release. The last operator release was November 22 last year. Since
> > then we
> > > > have added a number of fixes and improvements to both the operator
> and
> > the
> > > > autoscaler logic.
> > > >
> > > > There are a few outstanding PRs currently, including some larger
> > features
> > > > for the Autoscaler (JDBC event handler, Heap tuning), we have to
> make a
> > > > decision regarding those as well whether to include in the release or
> > not. @Maximilian
> > > > Michels  , @Rui Fan <1996fan...@gmail.com> what's
> your
> > > > take regarding those PRs? I generally like to be a bit more
> > conservative
> > > > with large new features to avoid introducing last minute
> instabilities.
> > > >
> > > > My proposal would be to aim for the end of this week as the freeze
> date
> > > > (Feb 9) and then we can prepare RC1 on monday.
> > > >
> > > > I am happy to volunteer as a release manager but I am of course open
> to
> > > > working together with someone on this.
> > > >
> > > > What do you think?
> > > >
> > > > Cheers,
> > > > Gyula
> > > >
> >
>

Re: [DISCUSS] FLIP-418: Show data skew score on Flink Dashboard

[Rui Fan]

Thanks Emre for the feedback!

I still think max/mean is more simple and easy to understand
for users. But I don’t have a strong opinion about it.

This proposal is absolutely useful for flink users! In order to
ensure the value for users, would you mind if we wait for
a while and check if there is more feedback from the community.
Also, would you mind sharing these 2 solutions to the
user[1] & user-zh[2] mail list as well? Flink users may give some
valuable feedback there, thanks~

[1] u...@flink.apache.org
[2] user...@flink.apache.org

Best,
Rui

On Thu, Feb 1, 2024 at 5:52 PM Kartoglu, Emre 
wrote:

> Hi Rui,
>
> Thanks for the useful feedback and caring about the user experience.
> I will update the FLIP based on 1 comment. I consider this a minor update.
>
> Please find my detailed responses below.
>
> "numRecordsInPerSecond sounds make sense to me, and I think
> it's necessary to mention it in the FLIP wiki. It will let other developers
> to easily understand. WDYT?"
>
> I feel like this might be touching implementation details. No objections
> though,
>  I will update the FLIP with this as one of the ways in which we can
> achieve the proposal.
>
>
> "After I detailed read the FLIP and Average_absolute_deviation, we know
> 0% is the best, 100% is worst."
>
> Correct.
>
>
> "I guess it is difficult for users who have not read the documentation to
> know the meaning of 50%. We hope that the designed Data skew will
> be easy for users to understand without reading or learning a series
> of backgrounds."
>
> I think I understand where you're coming from. My thought is that the user
> won't have to
> know exactly how the skew percentage/score is calculated. But this score
> will
> act as a warning sign for them. Upon seeing a skew score of 80% for an
> operator, as a user
> I will go and click on the operator to see many of my subtasks are not
> receiving any data at all.
> So it acts as a metric to get the user's attention to the skewed operator
> and fix issues.
>
>
> "For example, as you mentioned before, flink has a metric:
> numRecordsInPerSecond.
> I believe users know what numRecordsInPerSecond means even if they
> didn't read any documentation."
>
> The FLIP suggests that we will provide an explanation of the data skew
> score
> under the proposed Data Skew tab. I would like the exact wording to be
> left to
> the code review process to prevent these from blocking the implementation
> work/progress.
> This will be a user-friendly explanation with an option for the curious
> user to see the exact formula.
>
>
> Kind regards,
> Emre
>
>
> On 01/02/2024, 03:26, "Rui Fan" <1996fan...@gmail.com  1996fan...@gmail.com>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> > I was thinking about using the existing numRecordsInPerSecond metric
>
>
> numRecordsInPerSecond sounds make sense to me, and I think
> it's necessary to mention it in the FLIP wiki. It will let other developers
> to easily understand. WDYT?
>
>
> BTW, that's why I ask whether the data skew score means total
> receive records.
>
>
> > this would always give you a score higher than 1, with no way to cap the
> score.
>
>
> Yeah, you are right. max/mean is not a score, it's the data skew multiple.
> And I guess max/mean is easier to understand than
> Average_absolute_deviation.
>
>
> > I'm more used to working with percentages. The problem with the max/mean
> metric is I wouldn't immediately know whether a score of 300 is bad for
> instance.
> > Whereas if users saw above 50% as suggested in the FLIP for instance,
> they would consider taking action. I'm tempted to push back on this
> suggestion. Happy to discuss further, there is a chance I'm not seeing the
> downside of the proposed percentage based metric yet. Please let me know.
>
>
> After I detailed read the FLIP and Average_absolute_deviation, we know
> 0% is the best, 100% is worst.
>
>
> I guess it is difficult for users who have not read the documentation to
> know the meaning of 50%. We hope that the designed Data skew will
> be easy for users to understand without reading or learning a series
> of backgrounds.
>
>
> For example, as you mentioned before, flink has a metric:
> numRecordsInPerSecond.
> I believe users know what numRecordsInPerSecond means even if they
> didn't read any documentation.
>
>
> Of course, I'm opening for it. I may have missed something. I'd like to
> hear
> more feedback from 

[jira] [Created] (FLINK-34389) JdbcAutoscalerStateStore explicitly writes update_time

[Rui Fan (Jira)]

Rui Fan created FLINK-34389:
---

 Summary: JdbcAutoscalerStateStore explicitly writes update_time
 Key: FLINK-34389
 URL: https://issues.apache.org/jira/browse/FLINK-34389
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler
Affects Versions: kubernetes-operator-1.8.0
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.8.0


JdbcAutoscalerStateStore explicitly writes update_time instead of relying on 
the database to update.

Some databases doesn't support update the timestamp column automatically. As 
the common source service, in order to support all databases well, it's 
better to handle it inside of the service.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Kubernetes Operator 1.8.0 release planning

[Rui Fan]

Thanks Gyula for driving this release!

Release 1.8.0 sounds make sense to me.

As you said, I'm developing the JDBC event handler.
Since I'm going on vacation starting this Friday, and I have some
other work before I go on vacation. After evaluating my time today,
I found that I cannot complete the development, testing, and merging
of the JDBC event handler this week. So I tend to put the JDBC
event handler in the next version.

Best,
Rui

On Mon, Feb 5, 2024 at 11:42 PM Gyula Fóra  wrote:

> Hi all!
>
> I would like to kick off the release planning for the operator 1.8.0
> release. The last operator release was November 22 last year. Since then we
> have added a number of fixes and improvements to both the operator and the
> autoscaler logic.
>
> There are a few outstanding PRs currently, including some larger features
> for the Autoscaler (JDBC event handler, Heap tuning), we have to make a
> decision regarding those as well whether to include in the release or not. 
> @Maximilian
> Michels  , @Rui Fan <1996fan...@gmail.com> what's your
> take regarding those PRs? I generally like to be a bit more conservative
> with large new features to avoid introducing last minute instabilities.
>
> My proposal would be to aim for the end of this week as the freeze date
> (Feb 9) and then we can prepare RC1 on monday.
>
> I am happy to volunteer as a release manager but I am of course open to
> working together with someone on this.
>
> What do you think?
>
> Cheers,
> Gyula
>