Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

[Yuan Mei]

+1（binding）

Best
Yuan

On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1 (binding)
>
> Best,
> Rui
>
> On Mon, May 6, 2024 at 11:01 AM Yanfei Lei  wrote:
>
> > +1 (binding)
> >
> > Best,
> > Yanfei
> >
> > Zakelly Lan  于2024年5月6日周一 11:00写道：
> > >
> > > +1 (binding)
> > >
> > > Thanks for driving this!
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Mon, May 6, 2024 at 10:54 AM yue ma  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Thanks for all the feedback, I'd like to start a vote on the
> FLIP-447:
> > > > Upgrade FRocksDB from 6.20.3 to 8.10.0 [1]. The discussion thread is
> > here
> > > > [2].
> > > >
> > > > The vote will be open for at least 72 hours unless there is an
> > objection or
> > > > insufficient votes.
> > > >
> > > > [1]
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0
> > > > [2] https://lists.apache.org/thread/lrxjfpjjwlq4sjzm1oolx58n1n8r48hw
> > > >
> > > > --
> > > > Best,
> > > > Yue
> > > >
> >
>

Re: [DISCUSS] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

[Yuan Mei]

Hey Yue,

Thanks for all the great efforts significantly improving rescaling and
upgrading rocksdb.

+1 for this.

Best
Yuan

On Wed, Apr 24, 2024 at 10:46 AM Zakelly Lan  wrote:

> Hi Yue,
>
> Thanks for this proposal!
>
> Given the great improvement we could have, the slight regression in write
> performance is a worthwhile trade-off, particularly as the mem-table
> operations contribute only a minor part to the overall overhead. So +1 for
> this.
>
>
> Best,
> Zakelly
>
> On Tue, Apr 23, 2024 at 12:53 PM Yun Tang  wrote:
>
> > Hi Yue,
> >
> > Thanks for driving this work.
> >
> > It has been three years since last major upgrade of FRocksDB. And it
> would
> > be great improvement of Flink's state-backend with this upgrade. +1 for
> > this work.
> >
> >
> > Best
> > Yun Tang
> > 
> > From: Yanfei Lei 
> > Sent: Tuesday, April 23, 2024 12:50
> > To: dev@flink.apache.org 
> > Subject: Re: [DISCUSS] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0
> >
> > Hi Yue & Roman,
> >
> > Thanks for initiating this FLIP and all the efforts for the upgrade.
> >
> > 8.10.0 introduces some new features, making it possible for Flink to
> > implement some new exciting features, and the upgrade also makes
> > FRocksDB easier to maintain, +1 for upgrading.
> >
> > I read the FLIP and have a minor comment, it would be better to add
> > some description about the environment/configuration of the nexmark's
> > result.
> >
> > Roman Khachatryan  于2024年4月23日周二 12:07写道：
> >
> > >
> > > Hi,
> > >
> > > Thanks for writing the proposal and preparing the upgrade.
> > >
> > > FRocksDB  definitely needs to be kept in sync with the upstream and the
> > new
> > > APIs are necessary for faster rescaling.
> > > We're already using a similar version internally.
> > >
> > > I reviewed the FLIP and it looks good to me (disclaimer: I took part in
> > > some steps of this effort).
> > >
> > >
> > > Regards,
> > > Roman
> > >
> > > On Mon, Apr 22, 2024, 08:11 yue ma  wrote:
> > >
> > > > Hi Flink devs,
> > > >
> > > > I would like to start a discussion on FLIP-447: Upgrade FRocksDB from
> > > > 6.20.3 to 8.10.0
> > > >
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0
> > > >
> > > > This FLIP proposes upgrading the version of FRocksDB in the Flink
> > Project
> > > > from 6.20.3 to 8.10.0.
> > > > The FLIP mainly introduces the main benefits of upgrading FRocksDB,
> > > > including the use of IngestDB which can improve Rescaling performance
> > by
> > > > more than 10 times in certain scenarios, as well as other potential
> > > > optimization points such as async_io, blob db, and tiered storage.The
> > > > FLIP also presented test results based on RocksDB 8.10, including
> > > > StateBenchmark and Nexmark tests.
> > > > Overall, upgrading FRocksDB may result in a small regression of write
> > > > performance( which is a very small part of the overall overhead), but
> > it
> > > > can bring many important performance benefits.
> > > > So we hope to upgrade the version of FRocksDB through this FLIP.
> > > >
> > > > Looking forward to everyone's feedback and suggestions. Thank you!
> > > > --
> > > > Best regards,
> > > > Yue
> > > >
> >
> >
> >
> > --
> > Best,
> > Yanfei
> >
>

[ANNOUNCE] New Apache Flink Committer - Zakelly Lan

[Yuan Mei]

Hi everyone,

On behalf of the PMC, I'm happy to let you know that Zakelly Lan has become
a new Flink Committer!

Zakelly has been continuously contributing to the Flink project since 2020,
with a focus area on Checkpointing, State as well as frocksdb (the default
on-disk state db).

He leads several FLIPs to improve checkpoints and state APIs, including
File Merging for Checkpoints and configuration/API reorganizations. He is
also one of the main contributors to the recent efforts of "disaggregated
state management for Flink 2.0" and drives the entire discussion in the
mailing thread, demonstrating outstanding technical depth and breadth of
knowledge.

Beyond his technical contributions, Zakelly is passionate about helping the
community in numerous ways. He spent quite some time setting up the Flink
Speed Center and rebuilding the benchmark pipeline after the original one
was out of lease. He helps build frocksdb and tests for the upcoming
frocksdb release (bump rocksdb from 6.20.3->8.10).

Please join me in congratulating Zakelly for becoming an Apache Flink
committer!

Best,
Yuan (on behalf of the Flink PMC)

[jira] [Created] (FLINK-34984) Disaggregated State Storage and Management (Umbrella FLIP)

[Yuan Mei (Jira)]

Yuan Mei created FLINK-34984:


 Summary: Disaggregated State Storage and Management (Umbrella FLIP)
 Key: FLINK-34984
 URL: https://issues.apache.org/jira/browse/FLINK-34984
 Project: Flink
  Issue Type: New Feature
  Components: API / Core, API / DataStream, Runtime / Checkpointing, 
Runtime / State Backends
Reporter: Yuan Mei






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)

[Yuan Mei]

Hey dev,

I'm happy to announce that FLIP-423: Disaggregated State Storage and
Management (Umbrella FLIP) [1] has been accepted with 7 approving votes (5
binding) [2]

Piotrek Nowojski (binding)
Feifan Wang (non-binding)
Jing Ge (binding)
Rui Fan (binding)
Xintong (binding)
Yue (non-binding)
Yuan Mei (binding)

Best
Yuan

[1] https://cwiki.apache.org/confluence/x/R4p3EQ
[2] https://www.mail-archive.com/dev@flink.apache.org/msg74884.html

Re: [VOTE] FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)

[Yuan Mei]

+1 vote myself & Thanks all for the voting.

I'm closing the vote and the result will be posted in a separate mail.

Best

Yuan

On Fri, Mar 29, 2024 at 3:07 PM yue ma  wrote:

> +1 (non-binding)
>
>
> Best,
> Yue
>

Re: [VOTE] FLIP-427: Disaggregated state Store

[Yuan Mei]

+1 (binding)

Best
Yuan




On Wed, Mar 27, 2024 at 6:37 PM Hangxiang Yu  wrote:

> Hi devs,
>
> Thanks all for your valuable feedback about FLIP-427: Disaggregated state
> Store [1].
> I'd like to start a vote on it.  The discussion thread is here [2].
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
> [1] https://cwiki.apache.org/confluence/x/T4p3EQ
> [2] https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft
>
>
> Best,
> Hangxiang
>

Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

[Yuan Mei]

Congratulations

Best
Yuan

On Thu, Mar 28, 2024 at 5:35 PM Samrat Deb  wrote:

> Congratulations !
> Great news
>
> Bests,
> Samrat
>
> On Thu, 28 Mar 2024 at 3:00 PM, Ahmed Hamdy  wrote:
>
> > Congratulations!
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Thu, 28 Mar 2024 at 08:16, Paul Lam  wrote:
> >
> > > Congrats!
> > >
> > > Best,
> > > Paul Lam
> > >
> > > > 2024年3月28日 16:03，Rui Fan <1996fan...@gmail.com> 写道：
> > > >
> > > > Congratulations~
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Thu, Mar 28, 2024 at 3:55 PM Yu Li  wrote:
> > > >
> > > >> CC the Flink user and dev mailing list.
> > > >>
> > > >> Paimon originated within the Flink community, initially known as
> Flink
> > > >> Table Store, and all our incubating mentors are members of the Flink
> > > >> Project Management Committee. I am confident that the bonds of
> > > >> enduring friendship and close collaboration will continue to unite
> the
> > > >> two communities.
> > > >>
> > > >> And congratulations all!
> > > >>
> > > >> Best Regards,
> > > >> Yu
> > > >>
> > > >> On Wed, 27 Mar 2024 at 20:35, Guojun Li 
> > > wrote:
> > > >>>
> > > >>> Congratulations!
> > > >>>
> > > >>> Best,
> > > >>> Guojun
> > > >>>
> > > >>> On Wed, Mar 27, 2024 at 5:24 PM wulin  wrote:
> > > >>>
> > >  Congratulations~
> > > 
> > > > 2024年3月27日 15:54，王刚  写道：
> > > >
> > > > Congratulations~
> > > >
> > > >> 2024年3月26日 10:25，Jingsong Li  写道：
> > > >>
> > > >> Hi Paimon community,
> > > >>
> > > >> I’m glad to announce that the ASF board has approved a
> resolution
> > to
> > > >> graduate Paimon into a full Top Level Project. Thanks to
> everyone
> > > >> for
> > > >> your help to get to this point.
> > > >>
> > > >> I just created an issue to track the things we need to modify
> [2],
> > > >> please comment on it if you feel that something is missing. You
> > can
> > > >> refer to apache documentation [1] too.
> > > >>
> > > >> And, we already completed the GitHub repo migration [3], please
> > > >> update
> > > >> your local git repo to track the new repo [4].
> > > >>
> > > >> You can run the following command to complete the remote repo
> > > >> tracking
> > > >> migration.
> > > >>
> > > >> git remote set-url origin https://github.com/apache/paimon.git
> > > >>
> > > >> If you have a different name, please change the 'origin' to your
> > > >> remote
> > >  name.
> > > >>
> > > >> Please join me in celebrating!
> > > >>
> > > >> [1]
> > > 
> > > >>
> > >
> >
> https://incubator.apache.org/guides/transferring.html#life_after_graduation
> > > >> [2] https://github.com/apache/paimon/issues/3091
> > > >> [3] https://issues.apache.org/jira/browse/INFRA-25630
> > > >> [4] https://github.com/apache/paimon
> > > >>
> > > >> Best,
> > > >> Jingsong Lee
> > > 
> > > 
> > > >>
> > >
> > >
> >
>

Re: [VOTE] FLIP-433: State Access on DataStream API V2

[Yuan Mei]

+1 (binding)

Best,
Yuan

On Tue, Mar 26, 2024 at 8:59 AM Yunfeng Zhou 
wrote:

> +1 (non-binding)
>
> Best,
> Yunfeng
>
> On Wed, Mar 20, 2024 at 8:29 PM weijie guo 
> wrote:
> >
> > Hi everyone,
> >
> >
> > Thanks for all the feedback about the FLIP-433: State Access on
> > DataStream API V2 [1]. The discussion thread is here [2].
> >
> >
> > The vote will be open for at least 72 hours unless there is an
> > objection or insufficient votes.
> >
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-433%3A+State+Access+on+DataStream+API+V2
> >
> > [2] https://lists.apache.org/thread/8ghzqkvt99p4k7hjqxzwhqny7zb7xrwo
>

Re: [VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State

[Yuan Mei]

+1 (binding)

Best,
Yuan

On Wed, Mar 27, 2024 at 7:31 PM Jinzhong Li 
wrote:

> Hi devs,
>
>
> I'd like to start a vote on the FLIP-428: Fault Tolerance/Rescale
> Integration for Disaggregated State [1]. The discussion thread is here [2].
>
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
> [1] https://cwiki.apache.org/confluence/x/UYp3EQ
>
> [2] https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b
>
> Best,
>
> Jinzhong
>

Re: [VOTE] FLIP-425: Asynchronous Execution Model

[Yuan Mei]

+1 (binding)

Best,
Yuan

On Thu, Mar 28, 2024 at 4:33 PM Xuannan Su  wrote:

> +1 (non-binding)
>
> Best regards,
> Xuannan
>
> On Wed, Mar 27, 2024 at 6:28 PM Yanfei Lei  wrote:
> >
> > Hi everyone,
> >
> > Thanks for all the feedback about the FLIP-425: Asynchronous Execution
> > Model [1]. The discussion thread is here [2].
> >
> > The vote will be open for at least 72 hours unless there is an
> > objection or insufficient votes.
> >
> > [1] https://cwiki.apache.org/confluence/x/S4p3EQ
> > [2] https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h
> >
> > Best regards,
> > Yanfei
>

Re: [VOTE] FLIP-424: Asynchronous State APIs

[Yuan Mei]

+1 (binding)

Best,
Yuan

On Thu, Mar 28, 2024 at 4:30 PM Xuannan Su  wrote:

> +1 (non-binding)
>
> Best,
> Xuannan
>
>
> On Wed, Mar 27, 2024 at 6:23 PM Zakelly Lan  wrote:
> >
> > Hi devs,
> >
> > I'd like to start a vote on the FLIP-424: Asynchronous State APIs [1].
> The
> > discussion thread is here [2].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> or
> > insufficient votes.
> >
> > [1] https://cwiki.apache.org/confluence/x/SYp3EQ
> > [2] https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864
> >
> >
> > Best,
> > Zakelly
>

Re: [VOTE] FLIP-426: Grouping Remote State Access

[Yuan Mei]

+1 (binding)

Best,
Yuan

On Wed, Mar 27, 2024 at 6:56 PM Jinzhong Li 
wrote:

> Hi devs,
>
> I'd like to start a vote on the FLIP-426: Grouping Remote State Access [1].
> The discussion thread is here [2].
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
>
> [1] https://cwiki.apache.org/confluence/x/TYp3EQ
>
> [2] https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf
>
>
> Best,
>
> Jinzhong
>

[VOTE] FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)

[Yuan Mei]

Hi devs,

I'd like to start a vote on the FLIP-423: Disaggregated State Storage and
Management (Umbrella FLIP) [1]. The discussion thread is here [2].

The vote will be open for at least 72 hours unless there is an objection or
insufficient votes.

[1] https://cwiki.apache.org/confluence/x/R4p3EQ
[2] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0


Best,
Yuan

Re: [DISCUSS] FLIP-423 ～FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

[Yuan Mei]

 > means always use Record-ordered for custom operators /
> > UDFs,
> > > >>> and
> > > >>> > >> keep
> > > >>> > >> > > > State-ordered for internal usages (built-in operators)
> > only.
> > > >>> > >> > > >
> > > >>> > >> > >
> > > >>> > >> > > Indeed, users may not be able to choose the mode
> properly. I
> > > >>> agree
> > > >>> > to
> > > >>> > >> keep
> > > >>> > >> > > such options for internal use.
> > > >>> > >> > >
> > > >>> > >> > >
> > > >>> > >> > > 2. Regarding Strictly-ordered and Out-of-order of
> > Watermarks.
> > > >>> > >> > > >
> > > >>> > >> > > > I'm not entirely sure about Strictly-ordered being the
> > > >>> default, or
> > > >>> > >> even
> > > >>> > >> > > > being supported. From my understanding, a Watermark(T)
> > only
> > > >>> > >> suggests that
> > > >>> > >> > > > all records with event time before T has arrived, and it
> > has
> > > >>> > >> nothing to
> > > >>> > >> > > do
> > > >>> > >> > > > with whether records with event time after T has arrived
> > or
> > > >>> not.
> > > >>> > >> From
> > > >>> > >> > > that
> > > >>> > >> > > > perspective, preventing certain records from arriving
> > > before a
> > > >>> > >> Watermark
> > > >>> > >> > > is
> > > >>> > >> > > > never supported. I also cannot come up with any use case
> > > where
> > > >>> > >> > > > Strictly-ordered is necessary. This implies the same
> issue
> > > as
> > > >>> 1):
> > > >>> > >> how
> > > >>> > >> > > does
> > > >>> > >> > > > the user choose between the two modes?
> > > >>> > >> > > >
> > > >>> > >> > > > I'd suggest not expose the knob to users and only
> support
> > > >>> > >> Out-of-order,
> > > >>> > >> > > > until we see a concrete use case that Strictly-ordered
> is
> > > >>> needed.
> > > >>> > >> > > >
> > > >>> > >> > >
> > > >>> > >> > > The semantics of watermarks do not define the sequence
> > > between a
> > > >>> > >> watermark
> > > >>> > >> > > and subsequent records. For the most part, this is
> > > >>> inconsequential,
> > > >>> > >> except
> > > >>> > >> > > it may affect some current users who have previously
> relied
> > on
> > > >>> the
> > > >>> > >> implicit
> > > >>> > >> > > assumption of an ordered execution. I'd be fine with
> > initially
> > > >>> > >> supporting
> > > >>> > >> > > only out-of-order processing. We may consider exposing the
> > > >>> > >> > > 'Strictly-ordered' mode once we encounter a concrete use
> > case
> > > >>> that
> > > >>> > >> > > necessitates it.
> > > >>> > >> > >
> > > >>> > >> > >
> > > >>> > >> > > My philosophies behind not exposing the two config options
> > > are:
> > > >>> > >> > > > - There are already too many options in Flink that
> barely
> > > >>> know how
> > > >>> > >> to use
> > > >>> > >> > > > them. I think Flink should try as much as possible to
> > decide
> > > >>> its
> > > >>> > own
> > > >>> > >> > > > behavior, rather than throwing all the decisions to the
> > >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

[Yuan Mei]

Thanks for driving these efforts!

Congratulations

Best
Yuan

On Thu, Mar 21, 2024 at 4:35 PM Yu Li  wrote:

> Congratulations and look forward to its further development!
>
> Best Regards,
> Yu
>
> On Thu, 21 Mar 2024 at 15:54, ConradJam  wrote:
> >
> > Congrattulations!
> >
> > Leonard Xu  于2024年3月20日周三 21:36写道：
> >
> > > Hi devs and users,
> > >
> > > We are thrilled to announce that the donation of Flink CDC as a
> > > sub-project of Apache Flink has completed. We invite you to explore
> the new
> > > resources available:
> > >
> > > - GitHub Repository: https://github.com/apache/flink-cdc
> > > - Flink CDC Documentation:
> > > https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > >
> > > After Flink community accepted this donation[1], we have completed
> > > software copyright signing, code repo migration, code cleanup, website
> > > migration, CI migration and github issues migration etc.
> > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong,
> Qingsheng
> > > Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors for
> their
> > > contributions and help during this process！
> > >
> > >
> > > For all previous contributors: The contribution process has slightly
> > > changed to align with the main Flink project. To report bugs or
> suggest new
> > > features, please open tickets
> > > Apache Jira (https://issues.apache.org/jira).  Note that we will no
> > > longer accept GitHub issues for these purposes.
> > >
> > >
> > > Welcome to explore the new repository and documentation. Your feedback
> and
> > > contributions are invaluable as we continue to improve Flink CDC.
> > >
> > > Thanks everyone for your support and happy exploring Flink CDC!
> > >
> > > Best，
> > > Leonard
> > > [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > >
> > >
> >
> > --
> > Best
> >
> > ConradJam
>

Re: [DISCUSS] FLIP-423 ～FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

[Yuan Mei]

ease note that it is not a replacement of the
> > > original local
> > > > >> >> > state with synchronous execution. Instead this is a solution
> > > embracing the
> > > > >> >> > cloud-native era, providing much more scalability and
> resource
> > > efficiency
> > > > >> >> > when handling a *huge state*.
> > > > >> >> >
> > > > >> >> > What also worries me a lot in this fine grained model is the
> > > effect on the
> > > > >> >> > > checkpointing times.
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Your concerns are very reasonable. Faster checkpointing is
> > > always a core
> > > > >> >> > advantage of disaggregated state, but only for the async
> phase.
> > > There will
> > > > >> >> > be some complexity introduced by in-flight requests, but I'd
> > > suggest a
> > > > >> >> > checkpoint containing those in-flight state requests as part
> of
> > > the state,
> > > > >> >> > to accelerate the sync phase by skipping the buffer draining.
> > > This makes
> > > > >> >> > the buffer size have little impact on checkpoint time. And
> all
> > > the changes
> > > > >> >> > keep within the execution model we proposed while the
> > checkpoint
> > > barrier
> > > > >> >> > alignment or handling will not be touched in our proposal,
> so I
> > > guess
> > > > >> >> > the complexity is relatively controllable. I have faith in
> that
> > > :)
> > > > >> >> >
> > > > >> >> > Also regarding the overheads, it would be great if you could
> > > provide
> > > > >> >> > > profiling results of the benchmarks that you conducted to
> > > verify the
> > > > >> >> > > results. Or maybe if you could describe the steps to
> > reproduce
> > > the
> > > > >> >> > results?
> > > > >> >> > > Especially "Hashmap (sync)" vs "Hashmap with async API".
> > > > >> >> > >
> > > > >> >> >
> > > > >> >> > Yes we could profile the benchmarks. And for the comparison
> of
> > > "Hashmap
> > > > >> >> > (sync)" vs "Hashmap with async API", we conduct a Wordcount
> job
> > > written
> > > > >> >> > with async APIs but disabling the async execution by directly
> > > completing
> > > > >> >> > the future using sync state access. This evaluates the
> overhead
> > > of newly
> > > > >> >> > introduced modules like 'AEC' in sync execution (even though
> > > they are not
> > > > >> >> > designed for it). The code will be provided later. For other
> > > results of our
> > > > >> >> > PoC[1], you can follow the instructions here[2] to reproduce.
> > > Since the
> > > > >> >> > compilation may take some effort, we will directly provide
> the
> > > jar for
> > > > >> >> > testing next week.
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > And @Yunfeng Zhou, I have noticed your mail but it is a bit
> > late
> > > in my
> > > > >> >> > local time and the next few days are weekends. So I will
> reply
> > > to you
> > > > >> >> > later. Thanks for your response!
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > [1]
> > > > >> >> >
> > > > >> >> >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-PoCResults
> > > > >> >> > [2]
> > > > >> >> >
> > > > >> >> >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855#FLIP423:DisaggregatedStateStorageandManagement(UmbrellaFLIP)-Appendix:HowtorunthePoC
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Best,
> > > > >> >> > Zakelly
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Fri, Mar 1, 2024 at 6:38 PM Yunfeng Zhou <
> > > flink.zhouyunf...@gmail.com>
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> > > Hi,
> > > > >> >> > >
> > > > >> >> > > Thanks for proposing this design! I just read FLIP-424 and
> > > FLIP-425
> > > > >> >> > > and have some questions about the proposed changes.
> > > > >> >> > >
> > > > >> >> > > For Async API (FLIP-424)
> > > > >> >> > >
> > > > >> >> > > 1. Why do we need a close() method on StateIterator? This
> > > method seems
> > > > >> >> > > unused in the usage example codes.
> > > > >> >> > >
> > > > >> >> > > 2. In FutureUtils.combineAll()'s JavaDoc, it is stated that
> > > "No null
> > > > >> >> > > entries are allowed". It might be better to further explain
> > > what will
> > > > >> >> > > happen if a null value is passed, ignoring the value in the
> > > returned
> > > > >> >> > > Collection or throwing exceptions. Given that
> > > > >> >> > > FutureUtils.emptyFuture() can be returned in the example
> > code,
> > > I
> > > > >> >> > > suppose the former one might be correct.
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > For Async Execution (FLIP-425)
> > > > >> >> > >
> > > > >> >> > > 1. According to Fig 2 of this FLIP, if a recordB has its
> key
> > > collide
> > > > >> >> > > with an ongoing recordA, its processElement() method can
> > still
> > > be
> > > > >> >> > > triggered immediately, and then it might be moved to the
> > > blocking
> > > > >> >> > > buffer in AEC if it involves state operations. This means
> > that
> > > > >> >> > > recordB's output will precede recordA's output in
> downstream
> > > > >> >> > > operators, if recordA involves state operations while
> recordB
> > > does
> > > > >> >> > > not. This will harm the correctness of Flink jobs in some
> use
> > > cases.
> > > > >> >> > > For example, in dim table join cases, recordA could be a
> > delete
> > > > >> >> > > operation that involves state access, while recordB could
> be
> > > an insert
> > > > >> >> > > operation that needs to visit external storage without
> state
> > > access.
> > > > >> >> > > If recordB's output precedes recordA's, then an entry that
> is
> > > supposed
> > > > >> >> > > to finally exist with recordB's value in the sink table
> might
> > > actually
> > > > >> >> > > be deleted according to recordA's command. This situation
> > > should be
> > > > >> >> > > avoided and the order of same-key records should be
> strictly
> > > > >> >> > > preserved.
> > > > >> >> > >
> > > > >> >> > > 2. The FLIP says that StateRequests submitted by Callbacks
> > > will not
> > > > >> >> > > invoke further yield() methods. Given that yield() is used
> > > when there
> > > > >> >> > > is "too much" in-flight data, does it mean StateRequests
> > > submitted by
> > > > >> >> > > Callbacks will never be "too much"? What if the total
> number
> > of
> > > > >> >> > > StateRequests exceed the capacity of Flink operator's
> memory
> > > space?
> > > > >> >> > >
> > > > >> >> > > 3. In the "Watermark" section, this FLIP provided an
> > > out-of-order
> > > > >> >> > > execution mode apart from the default strictly-ordered
> mode,
> > > which can
> > > > >> >> > > optimize performance by allowing more concurrent
> executions.
> > > > >> >> > >
> > > > >> >> > > 3.1 I'm concerned that the out-of-order execution mode,
> along
> > > with the
> > > > >> >> > > epoch mechanism, would bring more complexity to the
> execution
> > > model
> > > > >> >> > > than the performance improvement it promises. Could we add
> > some
> > > > >> >> > > benchmark results proving the benefit of this mode?
> > > > >> >> > >
> > > > >> >> > > 3.2 The FLIP might need to add a public API section
> > describing
> > > how
> > > > >> >> > > users or developers can switch between these two execution
> > > modes.
> > > > >> >> > >
> > > > >> >> > > 3.3 Apart from the watermark and checkpoint mentioned in
> this
> > > FLIP,
> > > > >> >> > > there are also more other events that might appear in the
> > > stream of
> > > > >> >> > > data records. It might be better to generalize the
> execution
> > > mode
> > > > >> >> > > mechanism to handle all possible events.
> > > > >> >> > >
> > > > >> >> > > 4. It might be better to treat callback-handling as a
> > > > >> >> > > MailboxDefaultAction, instead of Mails, to avoid the
> overhead
> > > of
> > > > >> >> > > repeatedly creating Mail objects.
> > > > >> >> > >
> > > > >> >> > > 5. Could this FLIP provide the current default values for
> > > things like
> > > > >> >> > > active buffer size thresholds and timeouts? These could
> help
> > > with
> > > > >> >> > > memory consumption and latency analysis.
> > > > >> >> > >
> > > > >> >> > > 6. Why do we need to record the hashcode of a record in its
> > > > >> >> > > RecordContext? It seems not used.
> > > > >> >> > >
> > > > >> >> > > 7. In "timers can be stored on the JVM heap or RocksDB",
> the
> > > link
> > > > >> >> > > points to a document in flink-1.15. It might be better to
> > > verify the
> > > > >> >> > > referenced content is still valid in the latest Flink and
> > > update the
> > > > >> >> > > link accordingly. Same for other references if any.
> > > > >> >> > >
> > > > >> >> > > Best,
> > > > >> >> > > Yunfeng Zhou
> > > > >> >> > >
> > > > >> >> > > On Thu, Feb 29, 2024 at 2:17 PM Yuan Mei <
> > > yuanmei.w...@gmail.com> wrote:
> > > > >> >> > > >
> > > > >> >> > > > Hi Devs,
> > > > >> >> > > >
> > > > >> >> > > > This is a joint work of Yuan Mei, Zakelly Lan, Jinzhong
> Li,
> > > Hangxiang
> > > > >> >> > Yu,
> > > > >> >> > > > Yanfei Lei and Feng Wang. We'd like to start a discussion
> > > about
> > > > >> >> > > introducing
> > > > >> >> > > > Disaggregated State Storage and Management in Flink 2.0.
> > > > >> >> > > >
> > > > >> >> > > > The past decade has witnessed a dramatic shift in Flink's
> > > deployment
> > > > >> >> > > mode,
> > > > >> >> > > > workload patterns, and hardware improvements. We've moved
> > > from the
> > > > >> >> > > > map-reduce era where workers are computation-storage
> > tightly
> > > coupled
> > > > >> >> > > nodes
> > > > >> >> > > > to a cloud-native world where containerized deployments
> on
> > > Kubernetes
> > > > >> >> > > > become standard. To enable Flink's Cloud-Native future,
> we
> > > introduce
> > > > >> >> > > > Disaggregated State Storage and Management that uses DFS
> as
> > > primary
> > > > >> >> > > storage
> > > > >> >> > > > in Flink 2.0, as promised in the Flink 2.0 Roadmap.
> > > > >> >> > > >
> > > > >> >> > > > Design Details can be found in FLIP-423[1].
> > > > >> >> > > >
> > > > >> >> > > > This new architecture is aimed to solve the following
> > > challenges
> > > > >> >> > brought
> > > > >> >> > > in
> > > > >> >> > > > the cloud-native era for Flink.
> > > > >> >> > > > 1. Local Disk Constraints in containerization
> > > > >> >> > > > 2. Spiky Resource Usage caused by compaction in the
> current
> > > state model
> > > > >> >> > > > 3. Fast Rescaling for jobs with large states (hundreds of
> > > Terabytes)
> > > > >> >> > > > 4. Light and Fast Checkpoint in a native way
> > > > >> >> > > >
> > > > >> >> > > > More specifically, we want to reach a consensus on the
> > > following issues
> > > > >> >> > > in
> > > > >> >> > > > this discussion:
> > > > >> >> > > >
> > > > >> >> > > > 1. Overall design
> > > > >> >> > > > 2. Proposed Changes
> > > > >> >> > > > 3. Design details to achieve Milestone1
> > > > >> >> > > >
> > > > >> >> > > > In M1, we aim to achieve an end-to-end baseline version
> > > using DFS as
> > > > >> >> > > > primary storage and complete core functionalities,
> > including:
> > > > >> >> > > >
> > > > >> >> > > > - Asynchronous State APIs (FLIP-424)[2]: Introduce new
> APIs
> > > for
> > > > >> >> > > > asynchronous state access.
> > > > >> >> > > > - Asynchronous Execution Model (FLIP-425)[3]: Implement a
> > > non-blocking
> > > > >> >> > > > execution model leveraging the asynchronous APIs
> introduced
> > > in
> > > > >> >> > FLIP-424.
> > > > >> >> > > > - Grouping Remote State Access (FLIP-426)[4]: Enable
> > > retrieval of
> > > > >> >> > remote
> > > > >> >> > > > state data in batches to avoid unnecessary round-trip
> costs
> > > for remote
> > > > >> >> > > > access
> > > > >> >> > > > - Disaggregated State Store (FLIP-427)[5]: Introduce the
> > > initial
> > > > >> >> > version
> > > > >> >> > > of
> > > > >> >> > > > the ForSt disaggregated state store.
> > > > >> >> > > > - Fault Tolerance/Rescale Integration (FLIP-428)[6]:
> > > Integrate
> > > > >> >> > > > checkpointing mechanisms with the disaggregated state
> store
> > > for fault
> > > > >> >> > > > tolerance and fast rescaling.
> > > > >> >> > > >
> > > > >> >> > > > We will vote on each FLIP in separate threads to make
> sure
> > > each FLIP
> > > > >> >> > > > reaches a consensus. But we want to keep the discussion
> > > within a
> > > > >> >> > focused
> > > > >> >> > > > thread (this thread) for easier tracking of contexts to
> > avoid
> > > > >> >> > duplicated
> > > > >> >> > > > questions/discussions and also to think of the
> > > problem/solution in a
> > > > >> >> > full
> > > > >> >> > > > picture.
> > > > >> >> > > >
> > > > >> >> > > > Looking forward to your feedback
> > > > >> >> > > >
> > > > >> >> > > > Best,
> > > > >> >> > > > Yuan, Zakelly, Jinzhong, Hangxiang, Yanfei and Feng
> > > > >> >> > > >
> > > > >> >> > > > [1] https://cwiki.apache.org/confluence/x/R4p3EQ
> > > > >> >> > > > [2] https://cwiki.apache.org/confluence/x/SYp3EQ
> > > > >> >> > > > [3] https://cwiki.apache.org/confluence/x/S4p3EQ
> > > > >> >> > > > [4] https://cwiki.apache.org/confluence/x/TYp3EQ
> > > > >> >> > > > [5] https://cwiki.apache.org/confluence/x/T4p3EQ
> > > > >> >> > > > [6] https://cwiki.apache.org/confluence/x/UYp3EQ
> > > > >> >> > >
> > > > >> >> >
> > >
> >
>

[DISCUSS] FLIP-423 ～FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

[Yuan Mei]

Hi Devs,

This is a joint work of Yuan Mei, Zakelly Lan, Jinzhong Li, Hangxiang Yu,
Yanfei Lei and Feng Wang. We'd like to start a discussion about introducing
Disaggregated State Storage and Management in Flink 2.0.

The past decade has witnessed a dramatic shift in Flink's deployment mode,
workload patterns, and hardware improvements. We've moved from the
map-reduce era where workers are computation-storage tightly coupled nodes
to a cloud-native world where containerized deployments on Kubernetes
become standard. To enable Flink's Cloud-Native future, we introduce
Disaggregated State Storage and Management that uses DFS as primary storage
in Flink 2.0, as promised in the Flink 2.0 Roadmap.

Design Details can be found in FLIP-423[1].

This new architecture is aimed to solve the following challenges brought in
the cloud-native era for Flink.
1. Local Disk Constraints in containerization
2. Spiky Resource Usage caused by compaction in the current state model
3. Fast Rescaling for jobs with large states (hundreds of Terabytes)
4. Light and Fast Checkpoint in a native way

More specifically, we want to reach a consensus on the following issues in
this discussion:

1. Overall design
2. Proposed Changes
3. Design details to achieve Milestone1

In M1, we aim to achieve an end-to-end baseline version using DFS as
primary storage and complete core functionalities, including:

- Asynchronous State APIs (FLIP-424)[2]: Introduce new APIs for
asynchronous state access.
- Asynchronous Execution Model (FLIP-425)[3]: Implement a non-blocking
execution model leveraging the asynchronous APIs introduced in FLIP-424.
- Grouping Remote State Access (FLIP-426)[4]: Enable retrieval of remote
state data in batches to avoid unnecessary round-trip costs for remote
access
- Disaggregated State Store (FLIP-427)[5]: Introduce the initial version of
the ForSt disaggregated state store.
- Fault Tolerance/Rescale Integration (FLIP-428)[6]: Integrate
checkpointing mechanisms with the disaggregated state store for fault
tolerance and fast rescaling.

We will vote on each FLIP in separate threads to make sure each FLIP
reaches a consensus. But we want to keep the discussion within a focused
thread (this thread) for easier tracking of contexts to avoid duplicated
questions/discussions and also to think of the problem/solution in a full
picture.

Looking forward to your feedback

Best,
Yuan, Zakelly, Jinzhong, Hangxiang, Yanfei and Feng

[1] https://cwiki.apache.org/confluence/x/R4p3EQ
[2] https://cwiki.apache.org/confluence/x/SYp3EQ
[3] https://cwiki.apache.org/confluence/x/S4p3EQ
[4] https://cwiki.apache.org/confluence/x/TYp3EQ
[5] https://cwiki.apache.org/confluence/x/T4p3EQ
[6] https://cwiki.apache.org/confluence/x/UYp3EQ

Re: [VOTE] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

[Yuan Mei]

+1

On Thu, Jan 25, 2024 at 10:57 AM Xuannan Su  wrote:

> +1 (non-binding)
>
> Best,
> Xuannan
>
> On Thu, Jan 25, 2024 at 10:15 AM Lijie Wang 
> wrote:
> >
> > +1 (binding)
> >
> > Best,
> > Lijie
> >
> > Yanfei Lei  于2024年1月25日周四 10:06写道：
> >
> > > +1 (binding)
> > >
> > > Hangxiang Yu  于2024年1月25日周四 10:00写道：
> > > >
> > > > +1 (binding)
> > > >
> > > > On Thu, Jan 25, 2024 at 8:49 AM Rui Fan <1996fan...@gmail.com>
> wrote:
> > > >
> > > > > +1(binding)
> > > > >
> > > > > Best,
> > > > > Rui
> > > > >
> > > > > On Wed, 24 Jan 2024 at 21:50, Zakelly Lan 
> > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I'd like to start a vote on the FLIP-406: Reorganize State &
> > > > > Checkpointing
> > > > > > & Recovery Configuration [1]. The discussion thread is here [2].
> > > > > >
> > > > > > The vote will be open for at least 72 hours unless there is an
> > > objection
> > > > > or
> > > > > > insufficient votes.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > > > > [2]
> https://lists.apache.org/thread/0oc10cr2q2ms855dbo29s7v08xs3bvqg
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Hangxiang.
> > >
> > >
> > >
> > > --
> > > Best,
> > > Yanfei
> > >
>

Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

[Yuan Mei]

+1 binding

Best
Yuan

On Fri, Jan 19, 2024 at 12:09 PM Zakelly Lan  wrote:

> Hi everyone,
>
> I'd like to start a vote on the FLIP-416: Deprecate and remove the
> RestoreMode#LEGACY [1]. The discussion thread is here [2].
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
> [1] https://cwiki.apache.org/confluence/x/ookkEQ
> [2] https://lists.apache.org/thread/ho77fx13lw4ds52t0fs1xqz2vtn50n2o
>
>
> Best,
> Zakelly
>

Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

[Yuan Mei]

+1

Thanks for driving this Zakelly!

Best
Yuan

On Mon, Jan 15, 2024 at 10:47 PM Piotr Nowojski 
wrote:

> +1 good idea!
>
> pon., 15 sty 2024 o 05:11 Jinzhong Li 
> napisał(a):
>
> > Hi Zakelly,
> >
> > Thanks for driving the discussion. It makes sense to remove  LEGACY mode
> in
> > Flink 2.0.
> >
> > Best,
> > Jinzhong Li
> >
> > On Mon, Jan 15, 2024 at 10:34 AM Xuannan Su 
> wrote:
> >
> > > Hi Zakelly,
> > >
> > > Thanks for driving this. +1 to removing the LEGACY mode.
> > >
> > > Best regards,
> > > Xuannan
> > >
> > > On Mon, Jan 15, 2024 at 3:22 AM Danny Cranmer  >
> > > wrote:
> > > >
> > > > +1 to removing LEGACY mode in Flink 2.0. Thanks for driving.
> > > >
> > > > Danny,
> > > >
> > > > On Sat, 13 Jan 2024, 08:20 Yanfei Lei,  wrote:
> > > >
> > > > > Thanks Zakelly for starting this discussion.
> > > > >
> > > > > Regardless of whether it is for users or developers, deprecating
> > > > > RestoreMode#LEGACY makes the semantics clearer and lower
> maintenance
> > > > > costs, and Flink 2.0 is a good time point to do this.
> > > > > So +1 for the overall idea.
> > > > >
> > > > > Best,
> > > > > Yanfei
> > > > >
> > > > > Zakelly Lan  于2024年1月11日周四 14:57写道：
> > > > >
> > > > > >
> > > > > > Hi devs,
> > > > > >
> > > > > > I'd like to start a discussion on FLIP-416: Deprecate and remove
> > the
> > > > > > RestoreMode#LEGACY[1].
> > > > > >
> > > > > > The FLIP-193[2] introduced two modes of state file ownership
> during
> > > > > > checkpoint restoration: RestoreMode#CLAIM and
> RestoreMode#NO_CLAIM.
> > > The
> > > > > > LEGACY mode, which was how Flink worked until 1.15, has been
> > > superseded
> > > > > by
> > > > > > NO_CLAIM as the default mode. The main drawback of LEGACY mode is
> > > that
> > > > > the
> > > > > > new job relies on artifacts from the old job without cleaning
> them
> > > up,
> > > > > > leaving users uncertain about when it is safe to delete the old
> > > > > checkpoint
> > > > > > directories. This leads to the accumulation of unnecessary
> > checkpoint
> > > > > files
> > > > > > that are never cleaned up. Considering cluster availability and
> job
> > > > > > maintenance, it is not recommended to use LEGACY mode. Users
> could
> > > choose
> > > > > > the other two modes to get a clear semantic for the state file
> > > ownership.
> > > > > >
> > > > > > This FLIP proposes to deprecate the LEGACY mode and remove it
> > > completely
> > > > > in
> > > > > > the upcoming Flink 2.0. This will make the semantic clear as well
> > as
> > > > > > eliminate many bugs caused by mode transitions involving LEGACY
> > mode
> > > > > (e.g.
> > > > > > FLINK-27114 [3]) and enhance code maintainability.
> > > > > >
> > > > > > Looking forward to hearing from you!
> > > > > >
> > > > > > [1] https://cwiki.apache.org/confluence/x/ookkEQ
> > > > > > [2] https://cwiki.apache.org/confluence/x/bIyqCw
> > > > > > [3] https://issues.apache.org/jira/browse/FLINK-27114
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > >
> > >
> >
>

Re: [VOTE] Accept Flink CDC into Apache Flink

[Yuan Mei]

+1 binding



On Tue, Jan 9, 2024 at 3:21 PM Yuan Mei  wrote:

> +1
>
> Best,
> Yuan
>
> On Tue, Jan 9, 2024 at 3:06 PM tison  wrote:
>
>> +1 non-binding
>>
>> Best,
>> tison.
>>
>> Leonard Xu  于2024年1月9日周二 15:05写道：
>> >
>> > Hello all,
>> >
>> > This is the official vote whether to accept the Flink CDC code
>> contribution
>> >  to Apache Flink.
>> >
>> > The current Flink CDC code, documentation, and website can be
>> > found here:
>> > code: https://github.com/ververica/flink-cdc-connectors <
>> https://github.com/ververica/flink-cdc-connectors>
>> > docs: https://ververica.github.io/flink-cdc-connectors/ <
>> https://ververica.github.io/flink-cdc-connectors/>
>> >
>> > This vote should capture whether the Apache Flink community is
>> interested
>> > in accepting, maintaining, and evolving Flink CDC.
>> >
>> > Regarding my original proposal[1] in the dev mailing list, I firmly
>> believe
>> > that this initiative aligns perfectly with Flink. For the Flink
>> community,
>> > it represents an opportunity to bolster Flink's competitive edge in
>> streaming
>> > data integration, fostering the robust growth and prosperity of the
>> Apache Flink
>> > ecosystem. For the Flink CDC project, becoming a sub-project of Apache
>> Flink
>> > means becoming an integral part of a neutral open-source community,
>> capable of
>> > attracting a more diverse pool of contributors.
>> >
>> > All Flink CDC maintainers are dedicated to continuously contributing to
>> achieve
>> > seamless integration with Flink. Additionally, PMC members like Jark,
>> Qingsheng,
>> > and I are willing to infacilitate the expansion of contributors and
>> committers to
>> > effectively maintain this new sub-project.
>> >
>> > This is a "Adoption of a new Codebase" vote as per the Flink bylaws [2].
>> > Only PMC votes are binding. The vote will be open at least 7 days
>> > (excluding weekends), meaning until Thursday January 18 12:00 UTC, or
>> until we
>> > achieve the 2/3rd majority. We will follow the instructions in the
>> Flink Bylaws
>> > in the case of insufficient active binding voters:
>> >
>> > > 1. Wait until the minimum length of the voting passes.
>> > > 2. Publicly reach out via personal email to the remaining binding
>> voters in the
>> > voting mail thread for at least 2 attempts with at least 7 days between
>> two attempts.
>> > > 3. If the binding voter being contacted still failed to respond after
>> all the attempts,
>> > the binding voter will be considered as inactive for the purpose of
>> this particular voting.
>> >
>> > Welcome voting !
>> >
>> > Best,
>> > Leonard
>> > [1] https://lists.apache.org/thread/o7klnbsotmmql999bnwmdgo56b6kxx9l
>> > [2]
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
>>
>

Re: [VOTE] Accept Flink CDC into Apache Flink

[Yuan Mei]

+1

Best,
Yuan

On Tue, Jan 9, 2024 at 3:06 PM tison  wrote:

> +1 non-binding
>
> Best,
> tison.
>
> Leonard Xu  于2024年1月9日周二 15:05写道：
> >
> > Hello all,
> >
> > This is the official vote whether to accept the Flink CDC code
> contribution
> >  to Apache Flink.
> >
> > The current Flink CDC code, documentation, and website can be
> > found here:
> > code: https://github.com/ververica/flink-cdc-connectors <
> https://github.com/ververica/flink-cdc-connectors>
> > docs: https://ververica.github.io/flink-cdc-connectors/ <
> https://ververica.github.io/flink-cdc-connectors/>
> >
> > This vote should capture whether the Apache Flink community is interested
> > in accepting, maintaining, and evolving Flink CDC.
> >
> > Regarding my original proposal[1] in the dev mailing list, I firmly
> believe
> > that this initiative aligns perfectly with Flink. For the Flink
> community,
> > it represents an opportunity to bolster Flink's competitive edge in
> streaming
> > data integration, fostering the robust growth and prosperity of the
> Apache Flink
> > ecosystem. For the Flink CDC project, becoming a sub-project of Apache
> Flink
> > means becoming an integral part of a neutral open-source community,
> capable of
> > attracting a more diverse pool of contributors.
> >
> > All Flink CDC maintainers are dedicated to continuously contributing to
> achieve
> > seamless integration with Flink. Additionally, PMC members like Jark,
> Qingsheng,
> > and I are willing to infacilitate the expansion of contributors and
> committers to
> > effectively maintain this new sub-project.
> >
> > This is a "Adoption of a new Codebase" vote as per the Flink bylaws [2].
> > Only PMC votes are binding. The vote will be open at least 7 days
> > (excluding weekends), meaning until Thursday January 18 12:00 UTC, or
> until we
> > achieve the 2/3rd majority. We will follow the instructions in the Flink
> Bylaws
> > in the case of insufficient active binding voters:
> >
> > > 1. Wait until the minimum length of the voting passes.
> > > 2. Publicly reach out via personal email to the remaining binding
> voters in the
> > voting mail thread for at least 2 attempts with at least 7 days between
> two attempts.
> > > 3. If the binding voter being contacted still failed to respond after
> all the attempts,
> > the binding voter will be considered as inactive for the purpose of this
> particular voting.
> >
> > Welcome voting !
> >
> > Best,
> > Leonard
> > [1] https://lists.apache.org/thread/o7klnbsotmmql999bnwmdgo56b6kxx9l
> > [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
>

Re: [ANNOUNCE] The Flink Speed Center and benchmark daily run are back online

[Yuan Mei]

Thank you for your great efforts!

Best
Yuan

On Fri, Oct 20, 2023 at 4:08 PM Sergey Nuyanzin  wrote:

> Thanks a lot for working on this!
>
> On Fri, Oct 20, 2023 at 9:27 AM Yangze Guo  wrote:
>
> > Thanks for the effort, Zhaoqian!
> >
> > Best,
> > Yangze Guo
> >
> > On Fri, Oct 20, 2023 at 2:55 PM Leonard Xu  wrote:
> > >
> > > Thanks Zakelly for the great work.
> > >
> > > Best,
> > > Leonard
> > >
> > > > 2023年10月19日 下午7:39，Jing Ge  写道：
> > > >
> > > > Hi Zakelly,
> > > >
> > > > Thank you for your effort! Really appreciate it!
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Thu, Oct 19, 2023 at 12:02 PM Yun Tang  wrote:
> > > >
> > > >> Thanks for Zakelly's great work!
> > > >>
> > > >>
> > > >> Best
> > > >> Yun Tang
> > > >> 
> > > >> From: Piotr Nowojski 
> > > >> Sent: Thursday, October 19, 2023 17:56
> > > >> To: dev@flink.apache.org 
> > > >> Subject: Re: [ANNOUNCE] The Flink Speed Center and benchmark daily
> > run are
> > > >> back online
> > > >>
> > > >> Thank you!
> > > >>
> > > >> czw., 19 paź 2023 o 11:31 Konstantin Knauf 
> > napisał(a):
> > > >>
> > > >>> Thanks a lot for working on this!
> > > >>>
> > > >>> Am Do., 19. Okt. 2023 um 10:24 Uhr schrieb Zakelly Lan <
> > > >>> zakelly@gmail.com>:
> > > >>>
> > >  Hi everyone,
> > > 
> > >  Flink benchmarks [1] generate daily performance reports in the
> > Apache
> > >  Flink slack channel (#flink-dev-benchmarks) to detect performance
> > >  regression [2]. Those benchmarks previously were running on
> several
> > >  machines donated and maintained by Ververica. Unfortunately, those
> > >  machines were gone due to account issues [3] and the benchmarks
> > daily
> > >  run stopped since August 24th delaying the release of Flink 1.18 a
> > >  bit. [4].
> > > 
> > >  Ververica donated several new machines! After several weeks of
> > work, I
> > >  have successfully re-established the codespeed panel and benchmark
> > >  daily run pipelines on them. At this time, we are pleased to
> > announce
> > >  that the Flink Speed Center and benchmark pipelines are back
> online.
> > >  These new machines have a more formal management to ensure that
> > >  previous accidents will not occur in the future.
> > > 
> > >  What's more, I successfully recovered historical data backed up by
> > >  Yanfei Lei [5]. So with the old domain [6] redirected to the new
> > >  machines, the old links that existed in previous records will
> still
> > be
> > >  valid. Besides the benchmarks with Java8 and Java11, I also added
> a
> > >  pipeline for Java17 running daily.
> > > 
> > >  How to use it:
> > >  We also registered a new domain name 'flink-speed.xyz' for the
> > Flink
> > >  Speed Center [7]. It is recommended to use the new domain in the
> > >  future. Currently, the self-service method of triggering
> benchmarks
> > is
> > >  unavailable considering the lack of resources and potential
> > >  vulnerabilities of Jenkins. Please contact one of Apache Flink
> PMCs
> > to
> > >  submit a benchmark. More info is updated on the wiki[8].
> > > 
> > >  Daily Monitoring:
> > >  The performance daily monitoring on the Apache Flink slack channel
> > [2]
> > >  is still unavailable as the benchmark results need more time to
> > >  stabilize in the new environment. Once the baseline results become
> > >  available for regression detection, I will enable the daily
> > >  monitoring.
> > > 
> > >  Please feel free to reach out to me if you have any suggestions or
> > >  questions. Thanks Ververica again for denoting machines!
> > > 
> > > 
> > >  Best,
> > >  Zakelly
> > > 
> > >  [1] https://github.com/apache/flink-benchmarks
> > >  [2]
> > https://lists.apache.org/thread/zok62sx4m50c79htfp18ymq5vmtgbgxj
> > >  [3] https://issues.apache.org/jira/browse/FLINK-33052
> > >  [4]
> > https://lists.apache.org//thread/5x28rp3zct4p603hm4zdwx6kfr101w38
> > >  [5] https://issues.apache.org/jira/browse/FLINK-30890
> > >  [6] http://codespeed.dak8s.net:8000
> > >  [7] http://flink-speed.xyz
> > >  [8]
> > > 
> > > >>>
> > > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847
> > > 
> > > >>>
> > > >>>
> > > >>> --
> > > >>> https://twitter.com/snntrable
> > > >>> https://github.com/knaufk
> > > >>>
> > > >>
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

Re: FW: RE: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

[Yuan Mei]

+1 for the proposal

But "Since the signature of the public state API has been changed", I was
wondering whether this would be more fittable in Flink 2.0, instead of 1.19?

WDYT?

Best
Yuan

On Wed, Oct 11, 2023 at 4:34 PM David Radley 
wrote:

> Hi Zakelly,
> Thanks for making this clear for me.  We should document the impact on the
> user in the release notes, which will be a minimal rewrite and recompile of
> any java using the old APIs.
> I think it is a good point you make about if there are future
> implementations that are
> worth retrying (such as network access) – then there could be retries. I
> agree we should not be trying to create code now for an implementation
> consideration that is not there yet,
>
> +1 from me ,
>  Kind regards, David.
>
> From: Zakelly Lan 
> Date: Wednesday, 11 October 2023 at 04:25
> To: dev@flink.apache.org 
> Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-368 Reorganize the
> exceptions thrown in state interfaces
> Hi David,
>
> Thanks for your response.
>
> The exceptions thrown by state interfaces are NOT retriable. For
> example, there may be some elements sent to the wrong subtask due to a
> non-deterministic hashCode() algorithm and the key group is not
> matching. Or the rocksdb may fail to read a file if it has been
> deleted by the user. If there are future implementations that are
> worth retrying (such as network access), it would be better to let the
> implementation itself handle the retries and provide a configuration
> for this, rather than requiring users to catch these exceptions.
>
> Regarding the release and documentation, I have mentioned that this
> change is targeted for version 1.19 with proper documentation. You may
> have noticed that state interfaces are annotated with @PublicEvolving,
> which means these interfaces may change across versions. The changes
> are suitable for a minor release (1.18.0 currently to 1.19.0 in the
> future) as defined by the API compatibility guarantees of Flink[1].
>
>
>
> Best,
> Zakelly
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#api-compatibility-guarantees
>
> On Tue, Oct 10, 2023 at 6:19 PM David Radley 
> wrote:
> >
> > Hi,
> > I notice
> https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/state/ValueState.html
> is an external API. I am concerned that this change will break existing
> applications using the old interface, they are likely to have catches /
> throws around the existing checked Exceptions.
> >
> > If we go with RunTimeException, I would suggest that this sort of
> breaking change should be done on a Flink version change, where it is
> appropriate to make breaking changes to the API with associated
> documentation.
> >
> > If we want this change on a minor release,  we could create a new class
> ValueState2– that is used internally with the cleaned up Exceptions, but
> still expose the old class and Exceptions for existing external
> applications. I guess new applications could use the new ValueState2 .
> >
> > What do you think?
> > Kind regards, David.
> >
> >
> > From: David Radley 
> > Date: Tuesday, 10 October 2023 at 09:49
> > To: dev@flink.apache.org 
> > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-368 Reorganize the exceptions
> thrown in state interfaces
> > Hi ,
> > The argument seems to be that the errors cannot be acted on so should be
> runtime exceptions. I want to confirm that none of these errors could /
> should be retriable. If there is a possibility that the state is available
> at some time later then I assume a checked retriable Exception would be
> appropriate for those cases; and be part of the contract with the caller.
> Can we be sure that there is no possibility that the state will become
> available; if so then I agree that a runtime Exception is appropriate. What
> do you think?
> >
> >
> >
> > Kind regards, David.
> >
> >
> > From: Zakelly Lan 
> > Date: Monday, 9 October 2023 at 18:12
> > To: dev@flink.apache.org 
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-368 Reorganize the exceptions
> thrown in state interfaces
> > Hi everyone,
> >
> > It seems we're gradually reaching a consensus. So I would like to
> > start a vote after 72 hours if there are no further discussions.
> >
> > Please let me know if you have any concerns, thanks!
> >
> >
> > Best,
> > Zakelly
> >
> >
> > On Sat, Oct 7, 2023 at 4:07 PM Zakelly Lan 
> wrote:
> > >
> > > Hi Jing,
> > >
> > > Sorry for the late reply! I agree with you that we do not expect users
> > > to do anything with Flink and we won't "bother" them with those
> > > exceptions. However, users can still catch the `Throwable` and perform
> > > any necessary logging activities, similar to how they use Java
> > > Collection interfaces.
> > >
> > >
> > > Thanks for your insights!
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Thu, Sep 21, 2023 at 8:43 PM Jing Ge 
> wrote:
> > > >
> > > > Fair enough! Thanks Zakelly for the information. Afaic, even users
> 

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

[Yuan Mei]

Hey Zakelly,

Thanks very much for the efforts to re-build the entire benchmark
environment.

As long as we have
1) the pipeline set up and ready (no need for the entire portal ready),
2) get benchmark comparison numbers (comparing with the commit just before
the benchmark pipeline is down) and
3) confirmed no-regression, it should be good enough.

Thanks again!

Best
Yuan

On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan  wrote:

> Hi everyone,
>
> I am working on rebuilding the benchmark pipeline and it's almost
> done. However, due to the change in machines for benchmarking, I will
> need a few more days to run tests and gather the baseline scores for
> further comparison. Once the pipeline is fully ready, we will proceed
> with the performance test for release 1.18.0.
>
> Please let me know if you have any concerns. Thank you all for your
> patience.
>
> Best,
> Zakelly
>
> On Mon, Sep 18, 2023 at 6:57 PM Jing Ge 
> wrote:
> >
> > Hi everyone,
> >
> > The RC0 for Apache Flink 1.18.0 has been created. This RC is currently
> for
> > preview only to facilitate the integrated testing since the benchmark
> tests
> > are not available yet[1] and the release announcement is still under
> > review. The RC1 will be released after all benchmarks tests are passed.
> The
> > related voting process will be triggered once the announcement is ready.
> > The RC0 has all the artifacts that we would typically have for a release,
> > except for the release note and the website pull request for the release
> > announcement.
> >
> > The following contents are available for your review:
> >
> > - The preview source release and binary convenience releases [2], which
> > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3].
> > - all artifacts that would normally be deployed to the Maven
> > Central Repository [4].
> > - source code tag "release-1.18.0-rc0" [5]
> >
> > Your help testing the release will be greatly appreciated! And we'll
> > create the rc1 release and the voting thread as soon as all the efforts
> are
> > finished.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-33052
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1656/
> > [5] https://github.com/apache/flink/releases/tag/release-1.18.0-rc0
> >
> > Best regards,
> > Qingsheng, Sergei, Konstantin and Jing
>

Re: [DISCUSS] Update Flink Roadmap

[Yuan Mei]

Sorry for taking so long

I've added a section about Flink Disaggregated State Management Evolution
in the attached doc.

I found some of the contents might be overlapped with the "large-scale
streaming jobs". So that part might need some changes as well.

Please let me know what you think.

Best
Yuan

On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei  wrote:

> Sorry have missed this email and respond a bit late.
>
> I will put a draft for the long-term vision for the state as well as
> large-scale state support into the roadmap.
>
> Best
> Yuan
>
> On Mon, Jul 17, 2023 at 10:34 AM Jark Wu  wrote:
>
>> Hi Jiabao,
>>
>> Thank you for your suggestions. I have added them to the "Going Beyond a
>> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs" sections.
>>
>> Best,
>> Jark
>>
>> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun > .invalid>
>> wrote:
>>
>> > Thanks Jark and Martijn for driving this.
>> >
>> > There are two suggestions about the Table API:
>> >
>> > - Add the JSON type to adapt to the no sql database type.
>> > - Remove changelog normalize operator for upsert stream.
>> >
>> >
>> > Best,
>> > Jiabao
>> >
>> >
>> > > 2023年7月13日 下午3:49，Jark Wu  写道：
>> > >
>> > > Hi all,
>> > >
>> > > Sorry for taking so long back here.
>> > >
>> > > Martijn and I have drafted the first version of the updated roadmap,
>> > > including the updated feature radar reflecting the current state of
>> > > different components.
>> > >
>> >
>> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
>> > >
>> > > Feel free to leave comments in the thread or the document.
>> > > We may miss mentioning something important, so your help in enriching
>> > > the content is greatly appreciated.
>> > >
>> > > Best,
>> > > Jark & Martijn
>> > >
>> > >
>> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge 
>> wrote:
>> > >
>> > >> Hi Jark,
>> > >>
>> > >> Fair enough. Let's do it like you suggested. Thanks!
>> > >>
>> > >> Best regards,
>> > >> Jing
>> > >>
>> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu  wrote:
>> > >>
>> > >>> Hi Jing,
>> > >>>
>> > >>> This thread is for discussing the roadmap for versions 1.18, 2.0,
>> and
>> > >> even
>> > >>> more.
>> > >>> One of the outcomes of this discussion will be an updated version of
>> > the
>> > >>> current roadmap.
>> > >>> Let's work together on refining the roadmap in this thread.
>> > >>>
>> > >>> Best,
>> > >>> Jark
>> > >>>
>> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge 
>> > wrote:
>> > >>>
>> > >>>> Hi Jark,
>> > >>>>
>> > >>>> Thanks for driving it! For point 2, since we are developing 1.18
>> now,
>> > >>>> does it make sense to update the roadmap this time while we are
>> > >> releasing
>> > >>>> 1.18? This discussion thread will be focusing on the Flink 2.0
>> > roadmap,
>> > >>> as
>> > >>>> you mentioned previously. WDYT?
>> > >>>>
>> > >>>> Best regards,
>> > >>>> Jing
>> > >>>>
>> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu  wrote:
>> > >>>>
>> > >>>>> Hi all,
>> > >>>>>
>> > >>>>> Martijn and I would like to initiate a discussion on the Flink
>> > >> roadmap,
>> > >>>>> which should cover the project's long-term roadmap and the regular
>> > >>> update
>> > >>>>> mechanism.
>> > >>>>>
>> > >>>>> Xintong has already started a discussion about Flink 2.0 planning.
>> > >> One
>> > >>> of
>> > >>>>> the points raised in that discussion is that we should have a
>> > >>> high-level
>> > >>>>> discussion of the roadmap to present where the proje

[ANNOUNCE] New Apache Flink Committer - Hangxiang Yu

[Yuan Mei]

On behalf of the PMC, I'm happy to announce Hangxiang Yu as a new Flink
Committer.

Hangxiang has been active in the Flink community for more than 1.5 years
and has played an important role in developing and maintaining State and
Checkpoint related features/components, including Generic Incremental
Checkpoints (take great efforts to make the feature prod-ready). Hangxiang
is also the main driver of the FLIP-263: Resolving schema compatibility.

Hangxiang is passionate about the Flink community. Besides the technical
contribution above, he is also actively promoting Flink: talks about Generic
Incremental Checkpoints in Flink Forward and Meet-up. Hangxiang also spent
a good amount of time supporting users, participating in Jira/mailing list
discussions, and reviewing code.

Please join me in congratulating Hangxiang for becoming a Flink Committer!

Thanks,
Yuan Mei (on behalf of the Flink PMC)

[ANNOUNCE] New Apache Flink Committer - Yanfei Lei

[Yuan Mei]

On behalf of the PMC, I'm happy to announce Yanfei Lei as a new Flink
Committer.

Yanfei has been active in the Flink community for almost two years and has
played an important role in developing and maintaining State and Checkpoint
related features/components, including RocksDB Rescaling Performance
Improvement and Generic Incremental Checkpoints.

Yanfei also helps improve community infrastructure in many ways, including
migrating the Flink Daily performance benchmark to the Apache Flink slack
channel. She is the maintainer of the benchmark and has improved its
detection stability significantly. She is also one of the major maintainers
of the FrocksDB Repo and released FRocksDB 6.20.3 (part of Flink 1.17
release). Yanfei is a very active community member, supporting users and
participating
in tons of discussions on the mailing lists.

Please join me in congratulating Yanfei for becoming a Flink Committer!

Thanks,
Yuan Mei (on behalf of the Flink PMC)

Re: [DISCUSS] Update Flink Roadmap

[Yuan Mei]

Sorry have missed this email and respond a bit late.

I will put a draft for the long-term vision for the state as well as
large-scale state support into the roadmap.

Best
Yuan

On Mon, Jul 17, 2023 at 10:34 AM Jark Wu  wrote:

> Hi Jiabao,
>
> Thank you for your suggestions. I have added them to the "Going Beyond a
> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs" sections.
>
> Best,
> Jark
>
> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun 
> wrote:
>
> > Thanks Jark and Martijn for driving this.
> >
> > There are two suggestions about the Table API:
> >
> > - Add the JSON type to adapt to the no sql database type.
> > - Remove changelog normalize operator for upsert stream.
> >
> >
> > Best,
> > Jiabao
> >
> >
> > > 2023年7月13日 下午3:49，Jark Wu  写道：
> > >
> > > Hi all,
> > >
> > > Sorry for taking so long back here.
> > >
> > > Martijn and I have drafted the first version of the updated roadmap,
> > > including the updated feature radar reflecting the current state of
> > > different components.
> > >
> >
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> > >
> > > Feel free to leave comments in the thread or the document.
> > > We may miss mentioning something important, so your help in enriching
> > > the content is greatly appreciated.
> > >
> > > Best,
> > > Jark & Martijn
> > >
> > >
> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge 
> wrote:
> > >
> > >> Hi Jark,
> > >>
> > >> Fair enough. Let's do it like you suggested. Thanks!
> > >>
> > >> Best regards,
> > >> Jing
> > >>
> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu  wrote:
> > >>
> > >>> Hi Jing,
> > >>>
> > >>> This thread is for discussing the roadmap for versions 1.18, 2.0, and
> > >> even
> > >>> more.
> > >>> One of the outcomes of this discussion will be an updated version of
> > the
> > >>> current roadmap.
> > >>> Let's work together on refining the roadmap in this thread.
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge 
> > wrote:
> > >>>
> >  Hi Jark,
> > 
> >  Thanks for driving it! For point 2, since we are developing 1.18
> now,
> >  does it make sense to update the roadmap this time while we are
> > >> releasing
> >  1.18? This discussion thread will be focusing on the Flink 2.0
> > roadmap,
> > >>> as
> >  you mentioned previously. WDYT?
> > 
> >  Best regards,
> >  Jing
> > 
> >  On Thu, Jun 1, 2023 at 3:31 PM Jark Wu  wrote:
> > 
> > > Hi all,
> > >
> > > Martijn and I would like to initiate a discussion on the Flink
> > >> roadmap,
> > > which should cover the project's long-term roadmap and the regular
> > >>> update
> > > mechanism.
> > >
> > > Xintong has already started a discussion about Flink 2.0 planning.
> > >> One
> > >>> of
> > > the points raised in that discussion is that we should have a
> > >>> high-level
> > > discussion of the roadmap to present where the project is heading
> > >>> (which
> > > doesn't necessarily need to block the Flink 2.0 planning).
> Moreover,
> > >>> the
> > > roadmap on the Flink website [1] hasn't been updated for half a
> year,
> > >>> and
> > > the last update was for the feature radar for the 1.15 release. It
> > >> has
> >  been
> > > 2 years since the community discussed Flink's overall roadmap.
> > >
> > > I would like to raise two topics for discussion:
> > >
> > > 1. The new roadmap. This should be an updated version of the
> current
> > > roadmap[1].
> > > 2. A mechanism to regularly discuss and update the roadmap.
> > >
> > > To make the first topic discussion more efficient, Martijn and I
> >  volunteer
> > > to summarize the ongoing big things of different components and
> > >>> present a
> > > roadmap draft to the community in the next few weeks. This should
> be
> > >> a
> >  good
> > > starting point for a more detailed discussion.
> > >
> > > Regarding the regular update mechanism, there was a proposal in a
> > >>> thread
> > > [2] three years ago to make the release manager responsible for
> > >>> updating
> > > the roadmap. However, it appears that this was not documented as a
> >  release
> > > management task [3], and the roadmap update wasn't performed for
> > >>> releases
> > > 1.16 and 1.17.
> > >
> > > In my opinion, making release managers responsible for keeping the
> >  roadmap
> > > up to date is a good idea. Specifically, release managers of
> release
> > >> X
> >  can
> > > kick off the roadmap update at the beginning of release X, which
> can
> > >>> be a
> > > joint task with collecting a feature list [4]. Additionally,
> release
> > > managers of release X-1 can help verify and remove the accomplished
> > >>> items
> > > from the roadmap and update the feature radar.
> > >
> > > What do you think? Do you have other ideas?
> > >
> > > Best,
> 

Re: [VOTE] Release 2.0 must-have work items - Round 2

[Yuan Mei]

+1 (binding)

Thanks for driving the discussion through and for all the efforts in
resolving the complexities :-)

Best
Yuan

On Thu, Jul 20, 2023 at 5:23 PM Xintong Song  wrote:

> Hi all,
>
> I'd like to start another round of VOTE for the must-have work items for
> release 2.0 [1]. The corresponding discussion thread is [2], and the
> previous voting thread is [3]. All comments from the previous voting thread
> have been addressed.
>
> Please note that once the vote is approved, any changes to the must-have
> items (adding / removing must-have items, changing the priority) requires
> another vote. Assigning contributors / reviewers, updating descriptions /
> progress, changes to nice-to-have items do not require another vote.
>
> The vote will be open until at least July 25, following the consensus
> voting process. Votes of PMC members are binding.
>
> Best,
>
> Xintong
>
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
>
> [2] https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
>
> [3] https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m
>

[jira] [Created] (FLINK-32651) Benchmark Support for Changelog Statebackend

[Yuan Mei (Jira)]

Yuan Mei created FLINK-32651:


 Summary: Benchmark Support for Changelog Statebackend
 Key: FLINK-32651
 URL: https://issues.apache.org/jira/browse/FLINK-32651
 Project: Flink
  Issue Type: Improvement
Reporter: Yuan Mei






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release 2.0 must-have work items

[Yuan Mei]

DataSet, how users can implement
> > >> Sort/PartitionBy,
> > >> > etc
> > >> > > > as they did with DataSet?
> > >> > > > > Do we will also provide similar api in datastream or some
> other
> > >> thing
> > >> > > > before we remove DataSet?
> > >> > > > > Btw, as far as I see, with regarding to replcaing DataSet with
> > >> > > > Datastream, Datastream are missing many API. I think it may well
> > >> take
> > >> > > much
> > >> > > > effort to fully cover the missing api.
> > >> > > > >
> > >> > > > > [1]
> > >> https://lists.apache.org/thread/syjmt8f74gh8ok3z4lhgt95zl4dzn168
> > >> > > > >
> > >> > > > > Best regards,
> > >> > > > > Yuxia
> > >> > > > >
> > >> > > > > - 原始邮件 -
> > >> > > > > 发件人: "Jing Ge" 
> > >> > > > > 收件人: "dev" 
> > >> > > > > 发送时间: 星期三, 2023年 7 月 12日 上午 1:23:40
> > >> > > > > 主题: Re: [VOTE] Release 2.0 must-have work items
> > >> > > > >
> > >> > > > > agree with what Leonard said. There are actually more issues
> wrt
> > >> the
> > >> > > new
> > >> > > > > Source and SinkV2[1]
> > >> > > > >
> > >> > > > > Speaking of must-have vs nice-to-have, I think it depends on
> the
> > >> > > > priority.
> > >> > > > > If removing them has higher priority, we should keep related
> > >> tasks as
> > >> > > > > must-have and make sure enough effort will be put to solve
> those
> > >> > issues
> > >> > > > and
> > >> > > > > therefore be able to remove those APIs.
> > >> > > > >
> > >> > > > > Best regards,
> > >> > > > > Jing
> > >> > > > >
> > >> > > > > [1]
> > >> https://lists.apache.org/thread/90qc9nrlzf0vbvg92klzp9ftxxc43nbk
> > >> > > > >
> > >> > > > > On Tue, Jul 11, 2023 at 10:26 AM Leonard Xu <
> xbjt...@gmail.com>
> > >> > wrote:
> > >> > > > >
> > >> > > > > > Thanks Xintong for driving this great work! But I’ve to give
> > my
> > >> > > > > > -1(binding) here：
> > >> > > > > >
> > >> > > > > > -1 to mark "deprecat SourceFunction/SinkFunction/Sinkv1"
> item
> > as
> > >> > must
> > >> > > > to
> > >> > > > > > have for release 2.0.
> > >> > > > > >
> > >> > > > > > I do a lot of connector work in the community, and I have
> two
> > >> > > insights
> > >> > > > > > from past experience:
> > >> > > > > >
> > >> > > > > > 1. Many developers reported that it is very difficult to
> > migrate
> > >> > from
> > >> > > > > > SourceFunction to new Source [1]. The migration of existing
> > >> > > conenctors
> > >> > > > > > after deprecated SourceFunction is very difficult. Some
> > >> developers
> > >> > > > (Flavio
> > >> > > > > > Pompermaier) reported that they gave up the migration
> because
> > it
> > >> > was
> > >> > > > too
> > >> > > > > > complicated. I believe it's not a few cases. This means that
> > >> > > > deprecating
> > >> > > > > > SourceFunction related interfaces require community
> > >> contributors to
> > >> > > > reduce
> > >> > > > > > the migration cost before starting the migration work.
> > >> > > > > >
> > >> > > > > > 2. IIRC, the function of SinkV2 cannot currently cover
> > >> SinkFunction
> > >> > > as
> > >> > > > > > described in FLIP-287[2], it means the migration path after
> > >> > deprecate
> > >> > > > > > SinkFunction/Sinkv1 does not exist, thus we cannot mark the
> > >> related
> > >> > > > > > interfaces of sinkfunction/sinkv1  as deprecated in 1.18.
> > >> > > > > >
> > >> > > > > > Based on these two cognitions, I think we should not mark
> > these
> > >> > > > interfaces
> > >> > > > > > as must to have in 2.0. Maintaining the two sets of
> > source/sink
> > >> > > > interfaces
> > >> > > > > > is not a concern for me, users can choose the interface to
> > >> > implement
> > >> > > > > > according to their energy and needs.
> > >> > > > > >
> > >> > > > > > Btw, some work items in 2.0 are marked as must to have, but
> no
> > >> > > > contributor
> > >> > > > > > has claimed them yet. I think this is a risk and hope the
> > >> Release
> > >> > > > Managers
> > >> > > > > > could pay attention to it.
> > >> > > > > >
> > >> > > > > > Thank you all RMs for your work, sorry again for
> interrupting
> > >> the
> > >> > > vote
> > >> > > > > >
> > >> > > > > > Best,
> > >> > > > > > Leonard
> > >> > > > > >
> > >> > > > > > [1]
> > >> > https://lists.apache.org/thread/sqq26s9rorynr4vx4nhxz3fmmxpgtdqp
> > >> > > > > > [2]
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240880853
> > >> > > > > >
> > >> > > > > > > On Jul 11, 2023, at 4:11 PM, Yuan Mei <
> > yuanmei.w...@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > > > > >
> > >> > > > > > > As a second thought, I think "Eager State Declaration" is
> > >> > probably
> > >> > > > not a
> > >> > > > > > > must-have.
> > >> > > > > > >
> > >> > > > > > > I was originally thinking it is a prerequisite for "state
> > >> > querying
> > >> > > > for
> > >> > > > > > > disaggregated state management".
> > >> > > > > > >
> > >> > > > > > > Since disaggregated state management itself is not a
> > >> must-have,
> > >> > > > "Eager
> > >> > > > > > > State Declaration" is not as well. We can downgrade it to
> > >> "nice
> > >> > to
> > >> > > > have"
> > >> > > > > > if
> > >> > > > > > > no objection.
> > >> > > > > > >
> > >> > > > > > > Best
> > >> > > > > > >
> > >> > > > > > > Yuan
> > >> > > > > > >
> > >> > > > > > > On Mon, Jul 10, 2023 at 7:02 PM Jing Ge
> > >> >  > >> > > >
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > >> +1
> > >> > > > > > >>
> > >> > > > > > >> On Mon, Jul 10, 2023 at 12:52 PM Yu Li  >
> > >> > wrote:
> > >> > > > > > >>
> > >> > > > > > >>> +1 (binding)
> > >> > > > > > >>>
> > >> > > > > > >>> Thanks for driving this and great to see us moving
> > forward.
> > >> > > > > > >>>
> > >> > > > > > >>> Best Regards,
> > >> > > > > > >>> Yu
> > >> > > > > > >>>
> > >> > > > > > >>>
> > >> > > > > > >>> On Mon, 10 Jul 2023 at 11:59, Feng Wang <
> > >> wangfeng...@gmail.com
> > >> > >
> > >> > > > wrote:
> > >> > > > > > >>>
> > >> > > > > > >>>> +1
> > >> > > > > > >>>> Thanks for driving this, looking forward to the next
> > stage
> > >> of
> > >> > > > flink.
> > >> > > > > > >>>>
> > >> > > > > > >>>> On Fri, Jul 7, 2023 at 5:31 PM Xintong Song <
> > >> > > > tonysong...@gmail.com>
> > >> > > > > > >>> wrote:
> > >> > > > > > >>>>
> > >> > > > > > >>>>> Hi all,
> > >> > > > > > >>>>>
> > >> > > > > > >>>>> I'd like to start the VOTE for the must-have work
> items
> > >> for
> > >> > > > release
> > >> > > > > > >> 2.0
> > >> > > > > > >>>>> [1]. The corresponding discussion thread is [2].
> > >> > > > > > >>>>>
> > >> > > > > > >>>>> Please note that once the vote is approved, any
> changes
> > to
> > >> > the
> > >> > > > > > >>> must-have
> > >> > > > > > >>>>> items (adding / removing must-have items, changing the
> > >> > > priority)
> > >> > > > > > >>> requires
> > >> > > > > > >>>>> another vote. Assigning contributors / reviewers,
> > updating
> > >> > > > > > >>> descriptions /
> > >> > > > > > >>>>> progress, changes to nice-to-have items do not require
> > >> > another
> > >> > > > vote.
> > >> > > > > > >>>>>
> > >> > > > > > >>>>> The vote will be open until at least July 12,
> following
> > >> the
> > >> > > > consensus
> > >> > > > > > >>>>> voting process. Votes of PMC members are binding.
> > >> > > > > > >>>>>
> > >> > > > > > >>>>> Best,
> > >> > > > > > >>>>>
> > >> > > > > > >>>>> Xintong
> > >> > > > > > >>>>>
> > >> > > > > > >>>>>
> > >> > > > > > >>>>> [1]
> > >> > > > https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > >> > > > > > >>>>>
> > >> > > > > > >>>>> [2]
> > >> > > >
> https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
> > >> > > > > > >>>>>
> > >> > > > > > >>>>
> > >> > > > > > >>>
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [VOTE] Release 2.0 must-have work items

[Yuan Mei]

As a second thought, I think "Eager State Declaration" is probably not a
must-have.

I was originally thinking it is a prerequisite for "state querying for
disaggregated state management".

Since disaggregated state management itself is not a must-have, "Eager
State Declaration" is not as well. We can downgrade it to "nice to have" if
no objection.

Best

Yuan

On Mon, Jul 10, 2023 at 7:02 PM Jing Ge  wrote:

> +1
>
> On Mon, Jul 10, 2023 at 12:52 PM Yu Li  wrote:
>
> > +1 (binding)
> >
> > Thanks for driving this and great to see us moving forward.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Mon, 10 Jul 2023 at 11:59, Feng Wang  wrote:
> >
> > > +1
> > > Thanks for driving this, looking forward to the next stage of flink.
> > >
> > > On Fri, Jul 7, 2023 at 5:31 PM Xintong Song 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start the VOTE for the must-have work items for release
> 2.0
> > > > [1]. The corresponding discussion thread is [2].
> > > >
> > > > Please note that once the vote is approved, any changes to the
> > must-have
> > > > items (adding / removing must-have items, changing the priority)
> > requires
> > > > another vote. Assigning contributors / reviewers, updating
> > descriptions /
> > > > progress, changes to nice-to-have items do not require another vote.
> > > >
> > > > The vote will be open until at least July 12, following the consensus
> > > > voting process. Votes of PMC members are binding.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > > >
> > > > [2] https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
> > > >
> > >
> >
>

Re: [VOTE] Release 2.0 must-have work items

[Yuan Mei]

+1 (binding)

Thanks for driving this!

Best
Yuan

On Mon, Jul 10, 2023 at 10:26 AM Jark Wu  wrote:

> +1  (binding)
>
> Thanks for driving this. Looking forward to starting the 2.0 works.
>
> Best,
> Jark
>
> On Fri, 7 Jul 2023 at 17:31, Xintong Song  wrote:
>
> > Hi all,
> >
> > I'd like to start the VOTE for the must-have work items for release 2.0
> > [1]. The corresponding discussion thread is [2].
> >
> > Please note that once the vote is approved, any changes to the must-have
> > items (adding / removing must-have items, changing the priority) requires
> > another vote. Assigning contributors / reviewers, updating descriptions /
> > progress, changes to nice-to-have items do not require another vote.
> >
> > The vote will be open until at least July 12, following the consensus
> > voting process. Votes of PMC members are binding.
> >
> > Best,
> >
> > Xintong
> >
> >
> > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> >
> > [2] https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
> >
>

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Yuan Mei]

Congrats everyone :-)

Best
Yuan

On Fri, Jul 7, 2023 at 11:29 AM Hang Ruan  wrote:

> Hi, Leonard.
>
> I would like to help to add this page. Please assign this issue to me.
> Thanks.
>
> Best,
> Hang
>
> Leonard Xu  于2023年7月7日周五 11:26写道：
>
>> Congrats to all !
>>
>> It will be helpful to promote Apache Flink if we can add a page to our
>> website like others[2]. I’ve created an issue to improve this.
>>
>>
>> Best,
>> Leonard
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-32555
>> [2] https://spark.apache.org/news/sigmod-system-award.html
>>
>

Re: [DISCUSS] Release 2.0 Work Items

[Yuan Mei]

Thanks Xintong!

I am +1 on the change.

Best
Yuan

On Mon, Jul 3, 2023 at 6:20 PM Jing Ge  wrote:

> Hi Sergey,
>
> Thanks for the clarification! I will not hijack this thread to discuss
> Scala code strategy.
>
> Best regards,
> Jing
>
> On Mon, Jul 3, 2023 at 10:51 AM Sergey Nuyanzin 
> wrote:
>
> > Hi Jing,
> >
> > Maybe I was not clear enough, sorry.
> > However the main reason for this item about Calcite rules is not
> abandoning
> > Scala.
> > The main reason are changes in Calcite itself where there was introduced
> > code generator framework (immutables)
> > to generate config java classes for rules and old api (which is used in
> > Scala Calcirte rules) for that is marked as deprecated.
> > Since Immutables implies code generation while java compilation it seems
> > impossible to use for rules in Scala code.
> >
> > On Mon, Jul 3, 2023 at 10:44 AM Jing Ge 
> > wrote:
> >
> > > Hi,
> > >
> > > Speaking of "Move Calcite rules from Scala to Java", I was wondering if
> > > this thread is the right place to talk about it. Afaik, the Flink
> > community
> > > has decided to abandon Scala. That is the reason, I guess, we want to
> > move
> > > those Calcite rules from Scala to Java. On the other side, new Scala
> code
> > > will be added while developing new features[1]. Do we have any thoughts
> > > wrt the Scala code strategy?
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > >
> > > [1] https://lists.apache.org/thread/tnygl4n3q1fx75cl2vclc78j8mrxmz6y
> > >
> > > On Mon, Jul 3, 2023 at 10:31 AM Xintong Song 
> > > wrote:
> > >
> > > > Thanks all for the discussion.
> > > >
> > > >
> > > > IIUC, we need to make the following changes. Please correct me if I
> get
> > > it
> > > > wrong.
> > > >
> > > >
> > > > 1. Disaggregated State Management - Clarify that only the public API
> > > > related part is must-have for 2.0.
> > > >
> > > > 2. Java version support - Split it into 3 items: a) make java 17 the
> > > > default (must-have), b) drop java 8 (must-have), and c) drop java 11
> > > > (nice-to-have)
> > > >
> > > > 3. Add MetricGroup#getLogicalScope - Should be promoted to must-have
> > > >
> > > > 4. ProcessFunction API - Should be downgrade to nice-to-have
> > > >
> > > > 5. Configuration - Add an item "revisit all config option types and
> > > default
> > > > values", which IIUC should also be a must-have
> > > >
> > > >
> > > > There seems to be no changes needed for "Move Calcite rules from
> Scala
> > to
> > > > Java" as it's already nice-to-have.
> > > >
> > > >
> > > > If there's no objections, I'll update the wiki page accordingly, and
> > > start
> > > > a VOTE in the next couple of days.
> > > >
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Fri, Jun 30, 2023 at 12:53 AM Teoh, Hong
> >  > > >
> > > > wrote:
> > > >
> > > > > Thanks Xintong for driving the effort.
> > > > >
> > > > > I’d add a +1 to reworking configs, as suggested by @Jark and
> > @Chesnay,
> > > > > especially the types. We have various configs that encode Time /
> > > > MemorySize
> > > > > that are Long instead!
> > > > >
> > > > > Regards,
> > > > > Hong
> > > > >
> > > > >
> > > > >
> > > > > > On 29 Jun 2023, at 16:19, Yuan Mei 
> wrote:
> > > > > >
> > > > > > CAUTION: This email originated from outside of the organization.
> Do
> > > not
> > > > > click links or open attachments unless you can confirm the sender
> and
> > > > know
> > > > > the content is safe.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks for driving this effort, Xintong!
> > > > > >
> > > > > > To Chesnay
> > > > > >> I'm curious as to why the "Disaggregated State Management" item
> is
> > > > > >> marke

Re: [DISCUSS] Release 2.0 Work Items

[Yuan Mei]

Thanks for driving this effort, Xintong!

To Chesnay
> I'm curious as to why the "Disaggregated State Management" item is
> marked as a must-have; will it require changes that break something?
> What prevents it from being added in 2.1?

As to "Disaggregated State Management".

We plan to provide a new type of state backend to support DFS as primary
storage.
To achieve this, we at least need to include two parts of amends (not
entirely sure yet, since we are still in the designing and prototype phase)

1. Statebackend Change
2. State Access Change

Not all of the interfaces related are `@Internal`. Some of the interfaces
like `StateBackend` is `@PublicEvolving`
So, you are right in the sense that "Disaggregated State Management" itself
probably does not need to be a "Must Have"

But I was hoping changes that related to public APIs can be finalized and
merged in Flink 2.0 (I will fix the wiki accordingly).

I also agree with Jark that 2.0 is a good chance to rework the default
value of configurations.

Best
Yuan


On Thu, Jun 29, 2023 at 8:43 PM Chesnay Schepler  wrote:

> Something else configuration-related is that there are a bunch of
> options where the type isn't quite correct (e.g., a String where it
> could be an enum, a string where it should be an int or something).
> Could do a pass over those as well.
>
> On 29/06/2023 13:50, Jark Wu wrote:
> > Hi,
> >
> > I think one more thing we need to consider to do in 2.0 is changing the
> > default value of configuration to improve out-of-box user experience.
> >
> > Currently, in order to run a Flink job, users may need to set
> > a bunch of configurations, such as minibatch, checkpoint interval,
> > exactly-once,
> > incremental-checkpoint, etc. It's very verbose and hard to use for
> > beginners.
> > Most of them can have a universally applicable value.  Because changing
> the
> > default value is a breaking change. I think It's worth considering
> changing
> > them in 2.0.
> >
> > What do you think?
> >
> > Best,
> > Jark
> >
> >
> > On Wed, 28 Jun 2023 at 14:10, Sergey Nuyanzin 
> wrote:
> >
> >> Hi Chesnay
> >>
> >>> "Move Calcite rules from Scala to Java": I would hope that this would
> be
> >>> an entirely internal change, and could thus be an incremental process
> >>> independent of major releases.
> >>> What is the actual scale of this item; how much are we actually
> >> re-writing?
> >>
> >> Thanks for asking
> >> yes, you're right, that should be internal change.
> >> Yeah I was also thinking about incremental change (rule by rule or
> >> reasonable small group of rules).
> >> And yes, this could be an independent (on major release) activity
> >>
> >> The problem is actually for children of RelOptRule.
> >> Currently I see 60+ such rules (in Scala) using the mentioned deprecated
> >> api.
> >> There are also children of ConverterRule (50+) which do not have such
> >> issues.
> >> Maybe it could be considered as the next step to have all the rules in
> >> Java.
> >>
> >> On Tue, Jun 27, 2023 at 1:34 PM Xintong Song 
> >> wrote:
> >>
> >>> Hi Alex & Gyula,
> >>>
> >>> By compatibility discussion do you mean the "[DISCUSS] FLIP-321:
> >> Introduce
>  an API deprecation process" thread [1]?
> 
> >>> Yes, I meant the FLIP-321 discussion. I just noticed I pasted the wrong
> >> url
> >>> in my previous email. Sorry for the mistake.
> >>>
> >>> I am also curious to know if the rationale behind this new API has been
>  previously discussed on the mailing list. Do we have a list of
> >>> shortcomings
>  in the current DataStream API that it tries to resolve? How does the
>  current ProcessFunction functionality fit into the picture? Will it be
> >>> kept
>  as is or subsumed by new API?
> 
> >>> I don't think we should create a replacement for the DataStream API
> >> unless
>  we have a very good reason to do so and with a proper discussion about
> >>> this
>  as Alex said.
> >>>
> >>> The ProcessFunction API which is targeting to replace DataStream API is
> >>> still a proposal, not a decision. Sorry for the confusion, I should
> have
> >>> been more careful with my words, not giving the impression that this is
> >>> something we'll do anyway.
> >>>
> >>> There will be a FLIP describing the motivations and designs in detail,
> >> for
> >>> the community to discuss and vote on. We are still working on it. TBH,
> >> this
> >>> is not trivial and we would need more time on it.
> >>>
> >>> Just to quickly share some backgrounds:
> >>>
> >>> - We see quite some problems with the current DataStream APIs
> >>>- Users are working with concrete classes rather than
> interfaces,
> >>>which means
> >>>- Users can access methods that are designed to be used by
> internal
> >>>   classes, even though they are annotated with `@Internal`.
> E.g.,
> >>>   `DataStream#getTransformation`.
> >>>   - Changes to the non-API implementations (e.g.,
> >> `Transformation`)
> >>>   

Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints - Re-sent

[Yuan Mei]

Please ignore this duplicated voting email due to infra outage
INFRA-24572[1].

Voting is happening in the thread [2]

[1] https://issues.apache.org/jira/browse/INFRA-24572
[2] https://www.mail-archive.com/dev@flink.apache.org/msg66500.html

Best,
Yuan

On Wed, May 10, 2023 at 1:08 AM Zakelly Lan  wrote:

> Hi everyone,
>
> (I'm resending this vote since the last email was blocked for some reason.)
>
> Thanks for all the feedback for FLIP-306: Unified File Merging
> Mechanism for Checkpoints[1] on the discussion thread[2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until May 11th, 12:00PM GMT) unless there is an objection or an
> insufficient number of votes.
>
>
> Best,
> Zakelly
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints
> [2] https://lists.apache.org/thread/56px3kvn3tlwpc7sl12kx6notfmk9g7q
>

Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints

[Yuan Mei]

Please ignore this duplicated voting email due to infra outage
INFRA-24572[1].

Voting is happening in the thread [2]

[1] https://issues.apache.org/jira/browse/INFRA-24572
[2] https://www.mail-archive.com/dev@flink.apache.org/msg66500.html

On Wed, May 10, 2023 at 1:29 AM Zakelly Lan  wrote:

> Hi everyone,
>
> Thanks for all the feedback for FLIP-306: Unified File Merging Mechanism
> for Checkpoints[1] on the discussion thread[2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until May 12th, 10:00 AM GMT) unless there is an objection or an
> insufficient number of votes.
>
>
> Best regards,
> Zakelly
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints
> [2] https://lists.apache.org/thread/56px3kvn3tlwpc7sl12kx6notfmk9g7q

Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints

[Yuan Mei]

Please ignore this duplicated voting email for due to infra outage
INFRA-24572[1].

Voting is happening in the thread [2]

[1] https://issues.apache.org/jira/browse/INFRA-24572
[2] https://www.mail-archive.com/dev@flink.apache.org/msg66500.html


On Wed, May 10, 2023 at 1:47 AM Zakelly Lan  wrote:

> Hi everyone,
>
> (I'm resending this vote since the last email was blocked for some reason.)
>
> Thanks for all the feedback for FLIP-306: Unified File Merging
> Mechanism for Checkpoints[1] on the discussion thread[2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until May 11th, 5:00PM GMT) unless there is an objection or an
> insufficient number of votes.
>
>
> Best regards,
> Zakelly
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints
> [2] https://lists.apache.org/thread/56px3kvn3tlwpc7sl12kx6notfmk9g7q
>

Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints

[Yuan Mei]

Thanks for driving this, Zakelly.

As discussed in the thread,

+1 for the proposal (binding)

Best,

Yuan



On Wed, May 10, 2023 at 10:39 AM Zakelly Lan  wrote:

> Hi everyone,
>
> Sorry for the 4 duplicate emails. There was a problem with the dev
> mailing list blocking the mails from Gmail. I thought it was a network
> problem so I tried several times. The issue is addressed by
> INFRA-24572[1] and the piled up mails are delivered at one time.
>
> Based on the sending time, the vote will be open until May 12th at
> 11:00PM GMT. Please discuss and vote in the last thread (this one).
> Thanks!
>
>
> Best regards,
> Zakelly
>
> [1] https://issues.apache.org/jira/browse/INFRA-24572
>
> On Wed, May 10, 2023 at 10:30 AM Yanfei Lei  wrote:
> >
> > +1 (no-binding)
> >
> > Best,
> > Yanfei
> >
> >
> > Jing Ge  于2023年5月10日周三 07:03写道：
> >
> > >
> > > Hi Zakelly,
> > >
> > > I saw you sent at least 4 same emails for voting FLIP-306. I guess
> this one
> > > should be the last one and the right one for us to vote right? BTW,
> based
> > > on the sending time, 72 hours means to open the discussion until May
> 12th.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Tue, May 9, 2023 at 8:24 PM Zakelly Lan 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Thanks for all the feedback for FLIP-306: Unified File Merging
> > > > Mechanism for Checkpoints[1] on the discussion thread[2].
> > > >
> > > > I'd like to start a vote for it. The vote will be open for at least
> 72
> > > > hours (until May 11th, 12:00AM GMT) unless there is an objection or
> an
> > > > insufficient number of votes.
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > [1]
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints
> > > > [2] https://lists.apache.org/thread/56px3kvn3tlwpc7sl12kx6notfmk9g7q
> > > >
>

Re: [DISCUSS] FLIP-306: Unified File Merging Mechanism for Checkpoints

[Yuan Mei]

Hey all,

Thanks @Zakelly for driving this effort and thanks everyone for the warm
discussion. Sorry for the late response.

As I and Zakelly have already discussed and reviewed the design carefully
when drafting this FLIP, I do not have additional inputs here. But I want
to highlight several points that I've been quoted and explain why I think
the current design is a reasonable and clean one.

*Why this FLIP is proposed*
File Flooding is a problem for Flink I've seen many people bring up
throughout the years, especially for large clusters. Unfortunately, there
are not yet accepted solutions for the most commonly used state backend
like RocksDB. This FLIP was originally targeted to address merging
SST(KeyedState) checkpoint files.

While we are comparing different design choices, we found that different
types of checkpoint files (OPState, Unaligned CP channel state, Changelog
incremental state) share similar considerations, for example, file
management, file merging granularity, and e.t.c. That's why we want to
abstract a unified framework for merging these different types of
checkpoint files and provide flexibility to choose between merging
efficiency, rescaling/restoring cost, File system capabilities (affecting
File visibility), and e.t.c.

*File Ownership moved from JM to TM, WHY*
One of the major differences in the proposed design is moving file
ownership from JM to TM. A lot of questions/concerns are coming from here,
let me answer them one by one:

*1. Why the current JM SharedRegistry is not enough and do we have to
introduce more complexity?*
SharedRegistry maintains the mapping between *a file -> max CP ID using the
file*
For merging files, we have to introduce another level of mapping *a file ->
checkpoint file segment (merged files)*
So yes, no matter what, the second level of mapping has to be managed
somewhere, either JM or TM.

*2. Why the **complexity (second level of mapping)** cannot be maintained
in JM?*
- As a centralized service, JM has already been complicated and overloaded.
As mentioned by @Yanfei Lei , "triggering checkpoints
can be delayed by discarding shared state when JM manages a large number of
files FLINK-26590". This ends up setting the JM thread pool to 500!
- As explained by @Zakelly in the previous thread, the contract "for
Checkpoint N, only re-use shared state handles that have been already
referenced by checkpoint N-1" is not guaranteed for the concurrent
checkpoint in the current JM-owned design.  This problem can not be
addressed without significant changes in how SharedRegistry and checkpoint
subsume work, which, I do not think is worth it since "concurrent_CP>1" is
not used that much in prod.

*3. We have similar discussions before, moving ownership from JM to TM, why
it is not adopted at that time? *
As mentioned by Yun and Piotr, we have had similar discussions to move
ownership from JM to TM when designing the changelog state backend. The
reason why we stuck to JM ownership at that time is mainly due to
engineering time/effort constraints.
This time, since we need an extra level of mapping, which complicates the
JM logic even further, we indeed need to shade the complexity within the TM
to avoid more communications between JM and TM.
Zakelly has already shared the code branch (about 2000 lines), and it is
simple.

*4. Cloud-Native Trend*
The current centralized file management framework contradicts the
cloud-native trend. That's also one of the reasons moving ownership from JM
to TM was first proposed. The proposed design and implementation is a
worthy try-out in this direction. I'd like to put some more effort in this
direction if this really turns out working well.

One more thing I want to mention is that the proposed design shaded all the
code changes and complexities in the newly introduced File management in
TM. That says without enabling File merging, the code path of File managing
remains the same as before. So it is also a safe change in this sense.

Best,
Yuan



On Wed, Apr 12, 2023 at 5:23 PM Zakelly Lan  wrote:

> Hi Yun,
>
> I reorganized our discussion and added a comparison table of state
> ownership with some previous designs. Please take a look at section
> "4.9. State ownership comparison with other designs".
>
> But I don't see them as alternatives since the design of state
> ownership is integrated with this FLIP. That is to say, we are
> providing a file merging solution including file management for new
> merged files, other ownership models are not feasible for the current
> merging plan. If the state ownership changes, the design of merging
> files at different granularities also needs to be changed accordingly.
> WDYT?
>
>
> Best regards,
> Zakelly
>
> On Tue, Apr 11, 2023 at 10:18 PM Yun Tang  wrote:
> >
> > Hi Zakelly,
> >
> > Since we already had some discussions on this topic in the doc I
> mentioned, could you please describe the difference in your FLIP?
> >
> > I think we should better have a comparing table across different 

Re: RIC Incremental and GIC Incremental checkpoint use question

[Yuan Mei]

Hey

Thanks for trying out GIC! As Yanfei mentioned, GIC and RocksDB incremental
do not conflict with each other.

GIC is a generalized way to do incremental Flink checkpoints. At the same
time, RocksDB-incremental is a way to do incremental snapshots for backend
db (RocksDB).

Please let us know the results if convenient :-)

Best,
Yuan



On Sun, Mar 26, 2023 at 2:44 PM Yanfei Lei  wrote:

> Hi ConradJam,
>
> >  If Generic Incremental Checkpoint (GIC) enable, rocksdb Incremental
> Checkpoint can be disable or enable, Do they both have conflicting
> switches, does my turning on (GIC) mean I no longer need enable rocksdb
> Incremental Checkpoint ?
>
> The GIC and  rocksdb incremental Checkpoint are not in conflict, if
> GIC is enabled,  rocksdb Incremental Checkpoint can still be enabled
> to reduce the amount of data that needs to be uploaded.
> GIC is somewhat similar to the concept of WAL in the database, which
> can be regarded as a wrapper for original state backends(like heap,
> rocksdb, incremental rocksdb). The increment in GIC refers to
> **incremental log**, while the increment in the rocksdb state backend
> is the incremental SST file related to the specific state backend
> implementation.
>
> Here[1][2] are two blogs about GIC, hope it helps.
>
> [1]
> https://flink.apache.org/2022/05/30/improving-speed-and-stability-of-checkpointing-with-generic-log-based-incremental-checkpoints/
> [2]https://www.ververica.com/blog/generic-log-based-incremental-checkpoint
>
>
> ConradJam  于2023年3月26日周日 12:47写道：
> >
> > Hi Community . I would like to consult about some configurations of
> > Rocksdb incremental checkpoints and GIC. In Flink 1.17,I want to try
> > this feature . If Generic Incremental Checkpoint (GIC) enable, rocksdb
> > Incremental Checkpoint can be disable or enable, Do they both have
> > conflicting switches, does my turning on (GIC) mean I no longer need
> > enable rocksdb Incremental Checkpoint ? The community seems to have no
> > documentation to describe whether the two can be shared or only one of
> > them can be enabled, and the other does not need to be enabled
>
>
>
> --
> Best,
> Yanfei
>

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

[Yuan Mei]

Thanks Yanfei for driving the release!

Best
Yuan

On Mon, Jan 30, 2023 at 8:46 PM Jing Ge via user 
wrote:

> Hi Yanfei,
>
> Thanks for your effort. Looking forward to checking it.
>
> Best regards,
> Jing
>
> On Mon, Jan 30, 2023 at 1:42 PM Yanfei Lei  wrote:
>
>> It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0.
>>
>> Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86,
>> MacOS arm, and Windows are included in FRocksDB 6.20.3-ververica-2.0
>> jar, and the FRocksDB in Flink 1.17 would be updated to
>> 6.20.3-ververica-2.0.
>>
>> Release highlights:
>> - [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[1].
>> - [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[2].
>> - Avoid expensive ToString() call when not in debug[3].
>> - [FLINK-24932] Support build FRocksDB Java on Apple silicon[4].
>>
>> Maven artifacts for FRocksDB can be found at:
>> https://mvnrepository.com/artifact/com.ververica/frocksdbjni
>>
>> We would like to thank all efforts from the Apache Flink community
>> that made this release possible!
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-30457
>> [2] https://issues.apache.org/jira/browse/FLINK-30321
>> [3] https://github.com/ververica/frocksdb/pull/55
>> [4] https://issues.apache.org/jira/browse/FLINK-24932
>>
>> Best regards,
>> Yanfei
>> Ververica(Alibaba)
>>
>

Re: [VOTE] FLIP-290: Operator state compression (FLINK-30113)

[Yuan Mei]

+1 binding

Best
Yuan

On Sat, Jan 21, 2023 at 12:49 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1 (no-binding)
>
> Best
> Rui Fan
>
> On Fri, Jan 20, 2023 at 10:46 PM ConradJam  wrote:
>
> > +1 (no-binding)
> > thanks driving it
> >
> > Martijn Visser  于2023年1月20日周五 21:16写道：
> >
> > > +1 (binding)
> > >
> > > Thanks for driving this
> > >
> > > Best regards, Martijn
> > >
> > > Op vr 20 jan. 2023 om 12:53 schreef Piotr Nowojski <
> pnowoj...@apache.org
> > >:
> > >
> > > > +1 binding
> > > >
> > > > pt., 20 sty 2023 o 11:33 Dawid Wysakowicz 
> > > > napisał(a):
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > Best,
> > > > >
> > > > > Dawid
> > > > >
> > > > > On 20/01/2023 11:24, Etienne Chauchot wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > after the discussion in [1], I would like to open a voting thread
> > for
> > > > > > FLIP-290 [2] which discusses operator state compression.
> > > > > >
> > > > > > The vote will be open until January 25th (72h + weekend), unless
> > > there
> > > > > > is an objection or not enough votes.
> > > > > >
> > > > > > Best
> > > > > >
> > > > > > Etienne
> > > > > >
> > > > > >
> > > > > > (1)
> > https://lists.apache.org/thread/mor8vtc8s1w53mq884l80y3nt6zybw1w
> > > > > >
> > > > > > (2)
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-290+Operator+state+compression
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>

Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)

[Yuan Mei]

The proposal reads quite reasonable!

I do not have additional comments as long as the change can insure backward
compatibility. And many thanks to Dawid for catching this!

Best

Yuan

On Thu, Jan 19, 2023 at 6:03 PM Piotr Nowojski  wrote:

> Hi,
>
> The idea sounds like a nice improvement to complete the feature. I don't
> have any comments on top of what has been written already above.
>
> Bets,
> Piotrek
>
> czw., 19 sty 2023 o 09:57 Etienne Chauchot 
> napisał(a):
>
> > Hi,
> >
> > In the future we could add new compression algorithms by simply
> > extending /StreamCompressionDecorator/. For now there is only 2
> > extensions: /UncompressedStreamCompressionDecorator/ and
> > /SnappyStreamCompressionDecorator/.
> >
> > But I agree, I'd stick to /SnappyStreamCompressionDecorator /which is in
> > use for keyed state compression.
> >
> > I'll add a comment on the FLIP saying that.
> >
> > Best
> >
> > Etienne
> >
> > Le 18/01/2023 à 16:36, Dawid Wysakowicz a écrit :
> > > For the compression formats I'd stick with what is supported for the
> > > keyed state, which is either enable or disable compression. The format
> > > used for keyed state backend compression is snappy.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 18/01/2023 15:33, ConradJam wrote:
> > >> Thanks for driver this flip,I think have some question about this flip
> > >>
> > >> - Can we select the compression format, for example？ such as
> > >> snappy or
> > >> lz0
> > >> - If compression format is supported, I think the metadata will
> > >> probably
> > >> have to include which compression format
> > >>
> > >> and I agree idea with Dawid
> > >>
> > >> Dawid Wysakowicz  于2023年1月18日周三 20:55写道：
> > >>
> > >>
> > >>> It makes sense from my side.
> > >>>
> > >>> Could you, just for completeness, extend it with the info what will
> be
> > >>> the compression unit? If understand it correctly you envision each
> > >>> state
> > >>> to be compressed separately.
> > >>>
> > >>> Best,
> > >>>
> > >>> Dawid
> > >>>
> > >>> On 17/01/2023 15:47, Etienne Chauchot wrote:
> >  Hi everyone,
> > 
> >  I just published a new FLIP to introduce operator state compression
> >  (1). This feature is pretty straightforward to implement, so a PR is
> >  already under review (2) but it appeared during the review that to
> be
> >  able to insure backward compatibility of the operator state
> snapshots,
> >  we needed to modify the snapshot format hence the FLIP.
> > 
> >  Please tell me if you have any thoughts about it.
> > 
> > 
> >  [1]
> > 
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-290+Operator+state+compression
> > >>>
> > 
> >  [2] https://github.com/apache/flink/pull/21636
> > 
> > 
> >  Best,
> > 
> >  Etienne
> > 
> > >>
>

Re: [DISCUSS] Incorporate performance regression monitoring into routine process

[Yuan Mei]

Hey Yanfei,

Thanks so much for the efforts driving the whole process. It's great to see
that the performance benchmarks are indeed useful to help find regressions.
This is a discussion thread separated from the original performance
benchmark announcement thread [1]. Let's continue here so that more people
are aware of this change.

Overall, the instructions and proposal are good. Currently, the watching is
volunteered by Yanfei. However, she could only manage to check once or
twice every week, so I think it is important to integrate the
performance-watching process with the release-management process.

>From what I can see, there are still a couple of things that need to be
addressed:
- Improve the benchmark's stability [2], Yanfei is working on that.
- Someone other than Yanfei and me (maybe release managers) trying out the
instructions to see whether that's indeed clear how to find suspicious
commits causing regressions.

Looking forward to a more detailed discussion

Best
Yuan



[1] https://www.mail-archive.com/dev@flink.apache.org/msg61178.html
[2] https://issues.apache.org/jira/browse/FLINK-29825

On Thu, Jan 19, 2023 at 4:02 PM Yanfei Lei  wrote:

> Hi devs,
>
> I'd like to start a discussion about incorporating performance
> regression monitoring into the routine process. Flink benchmarks are
> periodically executed on http://codespeed.dak8s.net:8080 to monitor
> Flink performance. In late Oct'22, a new slack channel
> #flink-dev-benchmarks was created for notifications of performance
> regressions. It helped us find 2 build failures[1,2] and 5 performance
> regressions[3,4,5,6,7] in the past 3 months, which is very meaningful
> to ensuring the quality of the code.
>
> There are some release managers( cc @Matthias, @Martijn, @Qingsheng)
> proposing to incorporate performance regression monitoring into the
> release management, I think it makes sense for performance stabilities
> (like CI stabilities), since almost every release has some tickets
> about performance optimizations, the performance monitoring can
> effectively avoid performance regression and track the performance
> improvement of each release. So I start this discussion to pick
> everyone’s brain for some suggestions.
>
> In the past, I checked the slack notifications once a week, and I have
> summarized a draft[8](
> https://docs.google.com/document/d/1jTTJHoCTf8_LAjviyAY3Fi7p-tYtl_zw7rJKV4V6T_c/edit?usp=sharing
> )
> on how to deal with performance regressions according to some
> contributors and my own experience. If the above proposal is
> considered acceptable, I’d like to put it in the community wiki[9].
>
> Looking forward to your feedback!
>
> [1] https://issues.apache.org/jira/browse/FLINK-29883
> [2] https://issues.apache.org/jira/browse/FLINK-30015
> [3] https://issues.apache.org/jira/browse/FLINK-29886
> [4] https://issues.apache.org/jira/browse/FLINK-30181
> [5] https://issues.apache.org/jira/browse/FLINK-30623
> [6] https://issues.apache.org/jira/browse/FLINK-30624
> [7] https://issues.apache.org/jira/browse/FLINK-30625
> [8]
> https://docs.google.com/document/d/1jTTJHoCTf8_LAjviyAY3Fi7p-tYtl_zw7rJKV4V6T_c/edit?usp=sharing
> [9]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847
>
> Best,
> Yanfei
>

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Yuan Mei]

t; > >> > >It is unstable for a long time, see [5]
> > >> > >https://issues.apache.org/jira/browse/FLINK-27165 for possible
> > >> > reasons.
> > >> > >2. Regressions are detected by a simple script which may have
> > false
> > >> > >positives and false negatives, especially for benchmarks with
> > small
> > >> > >absolute values, small value changes cause large percentage
> > changes.
> > >> > see
> > >> > >[6] for details.
> > >> > >
> > >> > >  Maybe slidingWindow
> > >> > > <
> > >> > >
> > >> >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=6=slidingWindow=on=on=off=2=200
> > >> > > >(value~=600),
> > >> > > stateBackends.ROCKS
> > >> > > <
> > >> > >
> > >> >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=6=stateBackends.ROCKS=on=on=off=2=200
> > >> > > >
> > >> > > (value~=260) and serializerHeavyString
> > >> > > <
> > >> > >
> > >> >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=6=serializerHeavyString=on=on=off=2=200
> > >> > > >(value~=170)
> > >> > > are
> > >> > > not true regressions.
> > >> > >
> > >> > >1. For deployAllTasks.STREAMING
> > >> > ><
> > >> > >
> > >> >
> >
> http://codespeed.dak8s.net:8000/timeline/#/?exe=8=deployAllTasks.STREAMING=on=on=off=2=200
> > >> > > >,
> > >> > >this benchmark result is how much time it takes to deploy job,
> > the
> > >> > less
> > >> > >value the better performance, see [7] for details. FLINK-27571
> > >> > ><https://issues.apache.org/jira/browse/FLINK-27571>[8] would
> > fix this
> > >> > >problem.
> > >> > >
> > >> > >
> > >> > > As mentioned before, regressions are detected by a simple script
> > that is
> > >> > > less stable, FLINK-29825 <
> > >> > > https://issues.apache.org/jira/browse/FLINK-29825>[9]
> > >> > > is created to improve the benchmark's stability. I planned to
> > invite more
> > >> > > volunteers to monitor it after the checking of regression became
> > more
> > >> > > stable, but I've been stuck with something else lately, sorry for
> > the
> > >> > late
> > >> > > response.  Any suggestions on handling benchmark regressions/fails
> > are
> > >> > > welcome.
> > >> > >
> > >> > > [1] https://issues.apache.org/jira/browse/FLINK-29883
> > >> > >
> > >> > > [2] https://issues.apache.org/jira/browse/FLINK-29886
> > >> > >
> > >> > > [3] https://issues.apache.org/jira/browse/FLINK-30015
> > >> > >
> > >> > > [4] https://issues.apache.org/jira/browse/FLINK-30181
> > >> > >
> > >> > > [5] https://issues.apache.org/jira/browse/FLINK-27165
> > >> > >
> > >> > > [6]
> > >> > >
> > >> > >
> > >> >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py#L132-L136
> > >> > >
> > >> > > [7]
> > >> > >
> > >> > >
> > >> >
> >
> https://github.com/apache/flink-benchmarks/blob/master/src/main/java/org/apache/flink/scheduler/benchmark/deploying/DeployingTasksInStreamingJobBenchmarkExecutor.java#L58
> > >> > >
> > >> > > [8] https://issues.apache.org/jira/browse/FLINK-27571
> > >> > >
> > >> > > [9] https://issues.apache.org/jira/browse/FLINK-29825
> > >> > >
> > >> > >
> > >> > > Best,
> > >> > >
> > >> > > Yanfei
> > >> > >
> > >> > > Martijn Visser  于2022年11月29日周二 15:54写道：
> > >> > >
> > >> > > > Hi,
> > >> > > >
> > >> > > > Is there any update to be expected on the benchmark? I see
> > results of
> > >> > the
> > >

Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee

[Yuan Mei]

Congratulations, Lincoln!

Best,
Yuan

On Tue, Jan 10, 2023 at 12:23 PM Lijie Wang 
wrote:

> Congratulations, Lincoln!
>
> Best,
> Lijie
>
> Jingsong Li  于2023年1月10日周二 12:07写道：
>
> > Congratulations, Lincoln!
> >
> > Best,
> > Jingsong
> >
> > On Tue, Jan 10, 2023 at 11:56 AM Leonard Xu  wrote:
> > >
> > > Congratulations, Lincoln!
> > >
> > > Impressive work in streaming semantics, well deserved!
> > >
> > >
> > > Best,
> > > Leonard
> > >
> > >
> > > > On Jan 10, 2023, at 11:52 AM, Jark Wu  wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a new
> > Flink
> > > > committer.
> > > >
> > > > Lincoln Lee has been a long-term Flink contributor since 2017. He
> > mainly
> > > > works on Flink
> > > > SQL parts and drives several important FLIPs, e.g., FLIP-232 (Retry
> > Async
> > > > I/O), FLIP-234 (
> > > > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, He
> > also
> > > > contributed
> > > > much to Streaming Semantics, including the non-determinism problem
> and
> > the
> > > > message
> > > > ordering problem.
> > > >
> > > > Please join me in congratulating Lincoln for becoming a Flink
> > committer!
> > > >
> > > > Cheers,
> > > > Jark Wu
> > >
> >
>

Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0

[Yuan Mei]

+1

Best
Yuan

On Mon, Oct 31, 2022 at 5:01 PM Dawid Wysakowicz 
wrote:

> +1
>
> On 28/10/2022 16:57, Piotr Nowojski wrote:
> > Hi,
> >
> > As discussed on the dev mailing list [0] I would like to start a vote to
> > drop support of older savepoint formats (for Flink versions older than
> > 1.8). You can find the original explanation from the aforementioned dev
> > mailing list thread at the bottom of this message.
> >
> > Draft PR containing the proposed change you can find here:
> > https://github.com/apache/flink/pull/21056
> >
> > Vote will be open at least until Wednesday, November 2nd 18:00 CET.
> >
> > Best,
> > Piotrek
> >
> > [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w
> >
> > I would like to open a discussion to remove the long deprecated
> > (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the related
> > code.
> >
> > The motivation behind this move is two fold. One reason is that it
> > complicates our code base unnecessarily and creates confusion on how to
> > actually implement custom serializers. The immediate reason is that I
> > wanted to clean up Flink's configuration stack a bit and refactor the
> > ExecutionConfig class [2]. This refactor would keep the API compatibility
> > of the ExecutionConfig, but it would break savepoint compatibility with
> > snapshots written with some of the old serializers, which had
> > ExecutionConfig as a field and were serialized in the snapshot. This
> issue
> > has been resolved by the introduction of TypeSerializerSnapshot in Flink
> > 1.7 [3], where serializers are no longer part of the snapshot.
> >
> > TypeSerializerConfigSnapshot has been deprecated and no longer used by
> > built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged
> to
> > migrate to TypeSerializerSnapshot since then with their own custom
> > serializers. That has been plenty of time for the migration.
> >
> > This proposal would have the following impact for the users:
> > 1. we would drop support for recovery from savepoints taken with Flink <
> > 1.7.0 for all built in types serializers
> > 2. we would drop support for recovery from savepoints taken with Flink <
> > 1.8.0 for built in kryo serializers
> > 3. we would drop support for recovery from savepoints taken with Flink <
> > 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot
> >
> > 1. and 2. would have a simple migration path. Users migrating from those
> > old savepoints would have to first start his job using a Flink version
> from
> > the [1.8, 1.16] range, and take a new savepoint that would be compatible
> > with Flink 1.17.
> > 3. This is a bit more problematic, because users would have to first
> > migrate their own custom serializers to use TypeSerializerSnapshot
> (using a
> > Flink version from the [1.8, 1.16]), take a savepoint, and only then
> > migrate to Flink 1.17. However users had already 4 years to migrate,
> which
> > in my opinion has been plenty of time to do so.
> >
> > As a side effect, we could also drop support for some of the legacy
> > metadata serializers from LegacyStateMetaInfoReaders and potentially
> other
> > places that we are keeping for the sake of compatibility with old
> > savepoints.
> >
> > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html
> > [2] https://issues.apache.org/jira/browse/FLINK-29379
> > [3] https://issues.apache.org/jira/browse/FLINK-9377
> > [4] https://issues.apache.org/jira/browse/FLINK-9376
> > [5] https://issues.apache.org/jira/browse/FLINK-11323
> >
>

Re: [ANNOUNCE] Apache Flink 1.16.0 released

[Yuan Mei]

Congrats! Thanks everyone who is making this release happen!

Best
Yuan

On Mon, Oct 31, 2022 at 5:18 PM Danny Cranmer 
wrote:

> Nice work everyone!
>
> Congratulations to all involved :D
>
> Danny,
>
> On Mon, Oct 31, 2022 at 4:07 AM Paul Lam  wrote:
>
> > Congrats! Finally!
> >
> > Best,
> > Paul Lam
> >
> > > 2022年10月31日 11:58，Yuxin Tan  写道：
> > >
> > > Congrats! Glad to hear that.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > weijie guo  于2022年10月31日周一 11:51写道：
> > >
> > >> Congratulations, this is a version with many new features and thanks
> to
> > >> everyone involved!
> > >>
> > >> Best regards,
> > >>
> > >> Weijie
> > >>
> > >>
> > >> Yun Tang  于2022年10月31日周一 11:32写道：
> > >>
> > >>> Congratulations, and thanks to everyone involved!
> > >>>
> > >>>
> > >>> Best
> > >>> Yun Tang
> > >>> 
> > >>> From: Zakelly Lan 
> > >>> Sent: Monday, October 31, 2022 10:44
> > >>> To: dev@flink.apache.org 
> > >>> Subject: Re: [ANNOUNCE] Apache Flink 1.16.0 released
> > >>>
> > >>> Good to know & Congrats!
> > >>>
> > >>> Thanks everyone involved!
> > >>>
> > >>> Best,
> > >>> Zakelly
> > >>>
> > >>> On Mon, Oct 31, 2022 at 10:40 AM Becket Qin 
> > >> wrote:
> > 
> >  Hooray!! Congratulations to the team!
> > 
> >  Cheers,
> > 
> >  Jiangjie (Becket) Qin
> > 
> >  On Mon, Oct 31, 2022 at 9:57 AM Hang Ruan 
> > >>> wrote:
> > 
> > > Congratulations!
> > >
> > > Best,
> > > Hang
> > >
> > > Shengkai Fang  于2022年10月31日周一 09:40写道：
> > >
> > >> Congratulations!
> > >>
> > >> Best,
> > >> Shengkai
> > >>
> > >> Hangxiang Yu  于2022年10月31日周一 09:38写道：
> > >>
> > >>> Congratulations!
> > >>> Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the
> > >> release.
> > >>>
> > >>> On Fri, Oct 28, 2022 at 7:35 PM Jing Ge 
> > >>> wrote:
> > >>>
> >  Congrats!
> > 
> >  On Fri, Oct 28, 2022 at 1:22 PM 任庆盛 
> > >> wrote:
> > 
> > > Congratulations and a big thanks to Chesnay, Martijn, Godfrey
> > >>> and
> > >> Xingbo
> > > for the awesome work for 1.16!
> > >
> > > Best regards,
> > > Qingsheng Ren
> > >
> > >> On Oct 28, 2022, at 14:46, Xingbo Huang 
> > >>> wrote:
> > >>
> > >> The Apache Flink community is very happy to announce the
> > >>> release
> > > of
> > > Apache
> > >> Flink 1.16.0, which is the first release for the Apache
> > >> Flink
> > >>> 1.16
> > > series.
> > >>
> > >> Apache Flink® is an open-source stream processing framework
> > >>> for
> > >> distributed, high-performing, always-available, and accurate
> > >>> data
> > > streaming
> > >> applications.
> > >>
> > >> The release is available for download at:
> > >> https://flink.apache.org/downloads.html
> > >>
> > >> Please check out the release blog post for an overview of
> > >> the
> > >> improvements for this release:
> > >>
> > >>> https://flink.apache.org/news/2022/10/28/1.16-announcement.html
> > >>
> > >> The full release notes are available in Jira:
> > >>
> > >
> > >>>
> > >>
> > >
> > >>>
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275
> > >>
> > >> We would like to thank all contributors of the Apache Flink
> > >> community
> > >> who made this release possible!
> > >>
> > >> Regards,
> > >> Chesnay, Martijn, Godfrey & Xingbo
> > >
> > 
> > >>>
> > >>> --
> > >>> Best,
> > >>> Hangxiang.
> > >>>
> > >>
> > >
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] Delete useless branches from Flink repository

[Yuan Mei]

Hey Leonard,

Thanks for your efforts to clean up our repo!

Best
Yuan



On Thu, Oct 27, 2022 at 11:55 PM Leonard Xu  wrote:

> Thanks Matthias and Chesnay for the quick ACK.
>
> I’ve deleted following branches.
> > FLINK-29638-1.15
> > FLINK-29638-1.16
> > 28733
> > revert-16606-materialization_on_runtime
> > release0   Updated 2 years ago
> > docs_experimental__docsUpdated 2 years ago
> > docs_experimental__docs_compile Updated 3 years ago
>
> BTW, I fork a Flink repo [1] as a backup to avoid we delete required
> branches.
>
>
> Best,
> Leonard
> [1] https://github.com/flink-tpc-ds/flink-221027

Re: [VOTE] FLIP-263: Improve resolving schema compatibility

[Yuan Mei]

+1 (binding)

Thanks for driving this.

Best
Yuan

On Fri, Oct 28, 2022 at 11:17 AM yanfei lei  wrote:

> +1(non-binding) and thanks for Hangxiang's driving.
>
>
>
> Hangxiang Yu  于2022年10月28日周五 09:24写道：
>
> > Hi everyone,
> >
> > I'd like to start the vote for FLIP-263 [1].
> >
> > Thanks for your feedback and the discussion in [2][3].
> >
> > The vote will be open for at least 72 hours.
> >
> > Best regards,
> > Hangxiang.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility
> > [2] https://lists.apache.org/thread/4w36oof8dh28b9f593sgtk21o8qh8qx4
> >
> > [3] https://lists.apache.org/thread/t0bdkx1161rlbnsf06x0kswb05mch164
> >
>
>
> --
> Best,
> Yanfei
>

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Yuan Mei]

Thanks, Yanfei, to drive this and make the performance monitoring publicly
available.

Looking forward to seeing the workflow, and more details as Martijn
mentioned.

Best
Yuan

On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser 
wrote:

> Hi Yanfei Lei,
>
> Thanks for setting this up! It would be interesting to also know which
> aspects of Flink are monitored for "performance". I'm assuming there are
> specific pieces of functionality that are performance tested, but it would
> be great if this would be written down somewhere (next to a procedure how
> to detect a regression and what should be next steps).
>
> Best regards,
>
> Martijn
>
> On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan  wrote:
>
> > Hi yanfei,
> >
> > Thanks for driving this! It's a great help.
> >
> > I would like to join as a maintainer.
> >
> > Best,
> > Zakelly
> >
> > On Wed, Oct 26, 2022 at 11:32 AM yanfei lei  wrote:
> > >
> > > Hi everyone,
> > >
> > > As discussed earlier, we plan to create a benchmark channel in Apache
> > Flink
> > > slack[1], but the plan was shelved for a while[2]. So I went on with
> this
> > > work, and created the #flink-dev-benchmarks channel for performance
> > > regression notifications.
> > >
> > > We have a regression report script[3] that runs daily, and a
> notification
> > > would be sent to the slack channel when the last few benchmark results
> > are
> > > significantly worse than the baseline.
> > > Note, regressions are detected by a simple script which may have false
> > > positives and false negatives. And all benchmarks are executed on one
> > > physical machine[4] which is provided by Ververica(Alibaba)[5], it
> might
> > > happen that hardware issues affect performance, like "[FLINK-18614
> > > ] Performance
> > regression
> > > 2020.07.13"[6].
> > >
> > > After the migration, we need a procedure to watch over the entire
> > > performance of Flink code together. For example, if a regression
> > > occurs, investigating the cause and resolving the problem are needed.
> In
> > > the past, this procedure is maintained internally within Ververica, but
> > we
> > > think making the procedure public would benefit all. I volunteer to
> serve
> > > as one of the initial maintainers, and would be glad if more
> contributors
> > > can join me. I'd also prepare some guidelines to help others get
> familiar
> > > with the workflow. I will start a new thread to discuss the workflow
> > soon.
> > >
> > >
> > > [1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> > > [2] https://issues.apache.org/jira/browse/FLINK-28468
> > > [3]
> > >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
> > > [4] http://codespeed.dak8s.net:8080
> > > [5] https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro
> > >
> > > [6] https://issues.apache.org/jira/browse/FLINK-18614
> >
>

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

[Yuan Mei]

e new interface in
> all inner
> >   TypeSerializerSnapshots.
> >
> > Users may implement their own serializers based on inner serializers, we
> > should make sure that the new interface of inner TypeSerializerSnapshots
> is
> > usable.
> >
> >
> > Then I think it could work for both old custom serializers or new custom
> > serializers.
> >
> > No matter which interface the user implements, it could always work.
> >
> > Of course, we will deprecate the old interface and encourage users to use
> > the new one.
> >
> >
> >
> >1. Do we need to squash this with
> >https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w ?
> >
> > We will not break the compatibility based on 2, so it's not necessary to
> > squash them together.
> >
> >
> > Do you have any other suggestions ? Look forward to your reply!
> >
> > On Mon, Oct 24, 2022 at 8:30 PM Yuan Mei  wrote:
> >
> >> Hey Hangxiang,
> >>
> >> Thanks for driving this issue. I've read through all the discussions and
> >> suggestions in this thread, and here is my take:
> >>
> >> 1. I agree that the compatibility check should be done in the opposite
> >> direction.
> >> The current interface *causes some real issues* for users using
> >> their own `TypeSerializer`, and we should help resolve this issue.
> >>
> >> 2. The new opposite interface would better still stay within the class
> >> `TypeSerializerSnapshot`.
> >> I am suggesting this mostly considering the designed contract
> between
> >> `TypeSerializerSnapshot` and `TypeSerializer`. I do not think it is
> >> necessary to break the current contract because of this change.
> >>
> >> 3. It seems to me that "breaking changes" are unavoidable in each of the
> >> strategies proposed. However, *I'd be very concerned and worried *to
> >> make a breaking change *in the very first step*.
> >> A more user-friendly way is to make sure of compatibility and mark
> >> the method to drop as `@Deprecated`. After several versions of adoption,
> >> carefully and publicly discussing with users to drop the method. By
> >> compatibilities, I mean the following things:
> >> 1). upgrading from a lower Flink version to the new version does not
> >> need the user to do extra work for the change.
> >> 2). In the new version, both the old and the new method should be
> >> supported, and of course, we would encourage users to use the new
> method.
> >>For example, dropping "TypeSerializerConfigSnapshot" started to take
> >> into action after almost 10 releases the time it was marked deprecated.
> >>
> >>Due to the above reason, I am hesitant to squash this FLIP with
> >> https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w
> >>
> >> 4. What do I suggest doing?
> >>
> >>*STEP 1: In `TypeSerializerSnapshot`, we do something like this, and
> >> keep everything else exactly the same*
> >>
> >> @Deprecated
> >> default TypeSerializerSchemaCompatibility
> >> resolveSchemaCompatibility(
> >> TypeSerializer newSerializer) {
> >> // use new method
> >> return newSerializer.snapshotConfiguration().
> >>
> >> resolveSerializerSchemaCompatibility(this.restoreSerializer());
> >>
> >> }
> >>
> >> default TypeSerializerSchemaCompatibility
> >> resolveSerializerSchemaCompatibility(
> >> TypeSerializer oldSerializer) {
> >> return INCOMPATIBLE;
> >> }
> >>
> >>   *STEP 2: (This is a breaking change), after several releases*
> >>   1. Remove the deprecated method
> >>   2. Substitute every usage with the new method
> >>   3. Make the new method not have a default implementation
> >>
> >> STEP1 is compatible, and STEP2 is a breaking change.
> >>
> >> Best
> >> Yuan
> >>
> >>
> >>
> >> On Fri, Oct 21, 2022 at 10:01 AM godfrey he 
> wrote:
> >>
> >>> Hi Hangxiang, Dawid,
> >>>
> >>> I also prefer to add method into TypeSerializerSnapshot, which looks
> >>> more natural. TypeSerializerSnapshot has `Version` concept, which also
> >>> can be used for compatibility.
> >>> `
> >>> TypeSerializerSnapshot {
> >>>
> >>>

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

[Yuan Mei]

Hey Hangxiang,

Thanks for driving this issue. I've read through all the discussions and
suggestions in this thread, and here is my take:

1. I agree that the compatibility check should be done in the opposite
direction.
The current interface *causes some real issues* for users using their
own `TypeSerializer`, and we should help resolve this issue.

2. The new opposite interface would better still stay within the class
`TypeSerializerSnapshot`.
I am suggesting this mostly considering the designed contract between
`TypeSerializerSnapshot` and `TypeSerializer`. I do not think it is
necessary to break the current contract because of this change.

3. It seems to me that "breaking changes" are unavoidable in each of the
strategies proposed. However, *I'd be very concerned and worried *to make a
breaking change *in the very first step*.
A more user-friendly way is to make sure of compatibility and mark the
method to drop as `@Deprecated`. After several versions of adoption,
carefully and publicly discussing with users to drop the method. By
compatibilities, I mean the following things:
1). upgrading from a lower Flink version to the new version does not
need the user to do extra work for the change.
2). In the new version, both the old and the new method should be
supported, and of course, we would encourage users to use the new method.
   For example, dropping "TypeSerializerConfigSnapshot" started to take
into action after almost 10 releases the time it was marked deprecated.

   Due to the above reason, I am hesitant to squash this FLIP with
https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w

4. What do I suggest doing?

   *STEP 1: In `TypeSerializerSnapshot`, we do something like this, and
keep everything else exactly the same*

@Deprecated
default TypeSerializerSchemaCompatibility resolveSchemaCompatibility(
TypeSerializer newSerializer) {
// use new method
return newSerializer.snapshotConfiguration().

resolveSerializerSchemaCompatibility(this.restoreSerializer());

}

default TypeSerializerSchemaCompatibility
resolveSerializerSchemaCompatibility(
TypeSerializer oldSerializer) {
return INCOMPATIBLE;
}

  *STEP 2: (This is a breaking change), after several releases*
  1. Remove the deprecated method
  2. Substitute every usage with the new method
  3. Make the new method not have a default implementation

STEP1 is compatible, and STEP2 is a breaking change.

Best
Yuan



On Fri, Oct 21, 2022 at 10:01 AM godfrey he  wrote:

> Hi Hangxiang, Dawid,
>
> I also prefer to add method into TypeSerializerSnapshot, which looks
> more natural. TypeSerializerSnapshot has `Version` concept, which also
> can be used for compatibility.
> `
> TypeSerializerSnapshot {
>
> TypeSerializerSchemaCompatibility resolveSchemaCompatibility(
>  TypeSerializerSnapshot oldSnapshot);
>
> }
> `
>
> Best,
> Godfrey
>
> Dawid Wysakowicz  于2022年10月20日周四 16:59写道：
> >
> > That's the final status we will arrive at.
> >
> > IIUC, we cannot just remove the original method but just mark it as
> deprecated so two methods have to exist together in the short term.
> >
> > Users may also need to migrate their own external serializers which is a
> long run.
> >
> > I'd like to make jobs where users could always work without modifying
> any codes before removing the deprecated method so I provide a default
> implementation in the proposal.
> >
> >
> > I understand it works for old serializers, but it makes writing new
> serializers more complicated. It's not as simple as extending an interface
> and implementing all the methods it tells you to. You have to dig into
> documentation and check that you must implement one of the methods (either
> the old or the new one). Otherwise you end up in an ifinite loop.
> >
> >
> > If we break the compatibility, yes we cause some annoyance, but I feel
> it's a safer option. BTW, there is a proposal around serializer that also
> causes some incompatibilities so we could squash this together:
> https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w
> >
> >
> > I don't want to force that, but would be really happy to hear what
> others think.
> >
> >
> > Best,
> >
> > Dawid
> >
> > On 19/10/2022 04:05, Hangxiang Yu wrote:
> >
> > Hi, Dawid.
> >
> >
> > Thanks for your reply.
> >
> > Should we introduce the new method to the TypeSerializerSnapshot instead?
> >
> > You provided a valuable option, let me try to list the benefit and cost.
> >
> > benefit:
> >
> > TypeSerializerSnapshot still owns the responsibility of resolving schema
> compatibility so TypeSerializer could just pay attention to its
> serialization and deserialization as before.
> > It's very convenient to implement it based on current implementation by
> all information in TypeSerializerSnapshot and tools which is also helpful
> for users to migrate their external serializer.
> >
> > cost:
> >
> > The interface is a 

Re: [DISCUSS] Planning Flink 1.17

[Yuan Mei]

Thanks, Qingsheng for the kicking-off efforts.

1. January 17th, 2023 as feature freeze data sounds reasonable to me.
2. We will input our plan to the wiki link.

Thanks

Best
Yuan
Ververica (Alibaba)


On Fri, Oct 21, 2022 at 10:38 AM Xingbo Huang  wrote:

> Thanks Qingsheng, Leonard and Martijn for starting the discussion and
> volunteering.
> The timeline proposal sounds reasonable :+1:
>
> Best,
> Xingbo
>
> Jark Wu  于2022年10月21日周五 00:17写道：
>
> > Thanks for kicking off the 1.17 release.
> >
> > Targeting feature freeze on 1/17 for 1.17 release sounds pretty good to
> me.
> > +1 for the volunteers as release managers.
> >
> > Best,
> > Jark
> > Ververica (Alibaba)
> >
> > On Thu, 20 Oct 2022 at 18:09, Matthias Pohl  > .invalid>
> > wrote:
> >
> > > Thanks for starting the discussion about Flink 1.17. I would be
> > interested
> > > in helping out around the release as well.
> > >
> > > Best,
> > > Matthias
> > >
> > > On Thu, Oct 20, 2022 at 12:07 PM Xintong Song 
> > > wrote:
> > >
> > > > Thanks for kicking this off.
> > > >
> > > > +1 for the proposed timeline.
> > > >
> > > > Also +1 for Qingsheng, Leonard and Martijn as the release managers.
> > > Thanks
> > > > for volunteering.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Thu, Oct 20, 2022 at 3:59 PM Martijn Visser <
> > martijnvis...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi Qingsheng,
> > > > >
> > > > > I'm definitely interested in participating as a release manager
> > again.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Thu, Oct 20, 2022 at 9:47 AM Qingsheng Ren 
> > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > As we are approaching the official release of Flink 1.16, it’s a
> > good
> > > > > time
> > > > > > to kick off some discussions and march toward 1.17.
> > > > > >
> > > > > > - Release managers
> > > > > >
> > > > > > Leonard Xu and I would like to volunteer as release managers for
> > > 1.17,
> > > > > and
> > > > > > it would be great to have someone else working together on this
> > > > release.
> > > > > > Please let us know if you have any interest!
> > > > > >
> > > > > > - Timeline
> > > > > >
> > > > > > Having 1.16 will be released in the next few days and the 4
> months
> > > > > release
> > > > > > cycle after that, we propose to set the feature freezing date on
> > > > *January
> > > > > > 17th, 2023* (aligned with our version number 1.17 :-)), so that
> > > > everyone
> > > > > > could enjoy the holiday season and Chinese new year.
> > > > > >
> > > > > > - What we’ll be focusing
> > > > > >
> > > > > > Similar to our previous releases, we’d like to keep an eye on the
> > > > > > timeline, CI stability, release testing, and any communication
> and
> > > > > > coordination across teams and developers. One thing we’d like to
> > > > mention
> > > > > in
> > > > > > particular is compatibility, which is a frequent complaint from
> our
> > > > > > ecosystem developers and users. We encourage all committers to do
> > an
> > > > > extra
> > > > > > manual check to see if any public interface is touched before
> > > merging a
> > > > > PR.
> > > > > > We could discuss details in another thread later and update the
> > > > > > contributing guidelines to list which should be treated as public
> > > APIs.
> > > > > > Please feel free to raise any discussions if you have anything
> else
> > > to
> > > > > > emphasize specifically.
> > > > > >
> > > > > > - Collecting features
> > > > > >
> > > > > > We'll create a wiki page under this directory[1] for collecting
> new
> > > > > > features targeted in 1.17 as we always did before to give
> everyone
> > an
> > > > > > overview and track the progress. Please don’t hesitate to share
> > your
> > > > > ideas
> > > > > > on the page. In the meantime, we’d like to kindly invite our
> > > committers
> > > > > to
> > > > > > think about and plan what we could deliver to developers and
> users
> > in
> > > > > this
> > > > > > release.
> > > > > >
> > > > > > Looking forward to working with you all in the coming 1.17
> release!
> > > > > >
> > > > > > Best regards,
> > > > > > Qingsheng Ren and Leonard Xu
> > > > > > Ververica (Alibaba)
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Release+Management+and+Feature+Plan
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Drop Gelly

[Yuan Mei]

+1

On Tue, Oct 18, 2022 at 10:49 AM Dong Lin  wrote:

> +1
>
> On Thu, Oct 13, 2022 at 4:59 AM Martijn Visser 
> wrote:
>
> > Hi everyone,
> >
> > I would like to open a vote for dropping Gelly, which was discussed a
> long
> > time ago but never put to a vote [1].
> >
> > Voting will be open for at least 72 hours.
> >
> > Best regards,
> >
> > Martijn
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
> > [1] https://lists.apache.org/thread/2m6wtgjvxcogbf9d5q7mqt4ofqjf2ojc
> >
>

Re: Rescuing Queryable State from deprecation

[Yuan Mei]

Hey Ron,

Sorry for the late response. Thanks for the initiative to bring up this
topic and appreciate your efforts to rescue Queryable State :-). I will try
to answer your questions at my best!

*Why is Queryable State in the deprecation list?*
I've seen quite a few users bring up the requests to make "state" queryable
over the past few years, and I personally feel State Querying has a lot of
potential as well.
However, amongst all the use cases, I feel the current architecture of
queryable state is not aligned well with what is really needed to be
queried (I will explain why later, in the third section). I think this is
the main reason why the feature "queryable state" is not actively
maintained and improved, leading to the feature being added to the
deprecation list.

*What is the plan/future to support querying states?*
The motivation to introduce Queryable State was to provide external
applications with the capability to access real-time results from Flink
state without dependencies on external (k-v) stores. There are different
ways to achieve this. While Querable State can support some of the use
cases, I am not sure whether that's the way we want to head to before we
answer the following two questions:

1. What kind of data/query/result is expected from a user perspective
2. What does "real-time" mean here? Does that mean we have to read
uncommitted data (through the queryable state)?
Or if the checkpoint interval is short, reading committed data is good
enough (through state processor API)

>From my talking to users/customers (including myself), we do feel that
there are data already stored in the state that should not be wasted.
However, *Flink states are architected in a way optimized for fast
streaming processing access/fast failover/rescaling. Hence, state *
*querying **is far more complicated than simply "exposing states and making
the state accessible"*, as the current version of queryable state does.
Without scoping "state querying" well, it is difficult to make state
querying really usable and prod-ready.

*What is the problem with the queryable state?*
As I mentioned before, the problem mainly lies in the architecture. Here is
a short list based on my observation:

1. Queryable state service is unavailable during a failover.
2. Rollback to a previous checkpoint causing the state rollback as well.
We can solve this problem by only reading committed data. If this is the
case, why not simply use processor API then?
In the queryable state's current model,
it has to maintain multiple snapshots and wait for checkpoint completion.
This complicates the read model of the state store.
3. Have to support multi-thread reads from the state, which
complicates the accessing model and design
4. Cannot do complicated query analysis (other than simple look-up)
based on the current architecture.
5. The queryable state's life cycle is bound to the life cycle of the Flink
job. If a job is restarted, the job id is changed, and the query has to be
changed correspondingly even for the same state.

All in all, the queryable state is neither scalable nor high-available + it
has the potential to affect normal data processing if read access is
excessive.

Me and my colleagues have devoted ourselves to maintaining and improving
Flink state-related components in the Flink community during the last
couple of years. For reasons mentioned above: the past, the future, and the
problems faced with the queryable state, I'd be hesitant to remove the
queryable state from the deprecation list unless we have a clear vision of
where the "state querying" head (the two questions I proposed).

At the same time, I'd be open and happy to discuss this in more detail
online/offline.


Best Regards,

Yuan




On Tue, Aug 30, 2022 at 11:33 AM Yun Tang  wrote:

> Hi Ron,
>
> From my understanding, the feature of queryable state is introduced to
> Flink very early but lacks maintainers, in other words, this feature is
> currently a toy instead of a production-ready tool.
>
> I have heard from users many times asking for this feature in the
> production environment, and I believe it will bring benefits to the Flink
> community if someone could take it.
>
>
> Best
> Yun Tang
> 
> From: Konstantin Knauf 
> Sent: Monday, August 29, 2022 17:54
> To: dev 
> Subject: Re: Rescuing Queryable State from deprecation
>
> Hi Ron,
>
> thanks you for sharing your use case and your initiative to save Queryable
> State. Queryable State was deprecated due to a lack of maintainers and thus
> the community did not have resources to improve and develop it further. The
> deprecation notice was added to signal this lack of attention to our
> users.  On the other hand, I am not aware of anyone who is actively working
> towards actually dropping Queryable State. So, I don't think this will
> happen anytime soon.
>
> Personally, I see a lot of potential in Queryable State, but to make it
> really, really useful it still needs 

[jira] [Created] (FLINK-29082) Clean-up Leftovers for changelog pre-uploading files after failover

[Yuan Mei (Jira)]

Yuan Mei created FLINK-29082:


 Summary: Clean-up Leftovers for changelog pre-uploading files 
after failover
 Key: FLINK-29082
 URL: https://issues.apache.org/jira/browse/FLINK-29082
 Project: Flink
  Issue Type: Improvement
Reporter: Yuan Mei






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink Committer - Junhan Yang

[Yuan Mei]

Congratulations Junhan!

Best,
Yuan

On Sat, Aug 20, 2022 at 2:11 PM Danny Cranmer 
wrote:

> Congratulations Junhan! Welcome to the team.
>
> On Sat, 20 Aug 2022, 03:01 yuxia,  wrote:
>
> > Congratulations, Junhan!
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Aitozi" 
> > 收件人: "dev" 
> > 发送时间: 星期六, 2022年 8 月 20日 上午 12:18:29
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Junhan Yang
> >
> > Congratulations, Junhan!
> > Best,
> > Aitozi
> >
> > Guowei Ma  于2022年8月19日周五 13:18写道：
> >
> > > Congratulations, Junhan!
> > > Best,
> > > Guowei
> > >
> > >
> > > On Fri, Aug 19, 2022 at 6:01 AM Jing Ge  wrote:
> > >
> > > > Congrats Junhan!
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Thu, Aug 18, 2022 at 12:05 PM Jark Wu  wrote:
> > > >
> > > > > Congrats and welcome Junhan!
> > > > >
> > > > > Cheers,
> > > > > Jark
> > > > >
> > > > > > 2022年8月18日 17:59，Timo Walther  写道：
> > > > > >
> > > > > > Congratulations and welcome to the committer team :-)
> > > > > >
> > > > > > Regards,
> > > > > > Timo
> > > > > >
> > > > > > On 18.08.22 07:19, Lijie Wang wrote:
> > > > > >> Congratulations, Junhan!
> > > > > >> Best,
> > > > > >> Lijie
> > > > > >> Leonard Xu  于2022年8月18日周四 11:31写道：
> > > > > >>> Congratulations, Junhan!
> > > > > >>>
> > > > > >>> Best,
> > > > > >>>
> > > > >  2022年8月18日 上午11:27，Zhipeng Zhang 
> 写道：
> > > > > 
> > > > >  Congratulations, Junhan!
> > > > > 
> > > > >  Xintong Song  于2022年8月18日周四 11:21写道：
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > On behalf of the PMC, I'm very happy to announce Junhan Yang
> > as a
> > > > new
> > > > > >>> Flink
> > > > > > committer.
> > > > > >
> > > > > > Junhan has been contributing to the Flink project for more
> > than 1
> > > > > year.
> > > > > >>> His
> > > > > > contributions are mostly identified in the web frontend,
> > > including
> > > > > > FLIP-241, FLIP-249 and various maintenance efforts of Flink's
> > > > > frontend
> > > > > > frameworks.
> > > > > >
> > > > > > Please join me in congratulating Junhan for becoming a Flink
> > > > > committer!
> > > > > >
> > > > > > Best,
> > > > > > Xintong
> > > > > 
> > > > > 
> > > > > 
> > > > >  --
> > > > >  best,
> > > > >  Zhipeng
> > > > > >>>
> > > > > >>>
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Lijie Wang

[Yuan Mei]

Congratulations, Lijie!

Best,
Yuan

On Sat, Aug 20, 2022 at 2:12 PM Danny Cranmer 
wrote:

> Congratulations Lijie! Welcome to the team.
>
> On Sat, 20 Aug 2022, 03:25 Yun Tang,  wrote:
>
> > Congratulations, Lijie!
> >
> >
> > Best
> > Yun Tang
> > 
> > From: Geng Biao 
> > Sent: Saturday, August 20, 2022 10:03
> > To: dev@flink.apache.org 
> > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Lijie Wang
> >
> > Congratulations  Lijie!
> > Best,
> > Biao Geng
> >
> > 获取 Outlook for iOS
> > 
> > 发件人: yuxia 
> > 发送时间: Saturday, August 20, 2022 9:54:29 AM
> > 收件人: dev 
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Lijie Wang
> >
> > Congrats Lijie!
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Aitozi" 
> > 收件人: "dev" 
> > 发送时间: 星期六, 2022年 8 月 20日 上午 12:19:27
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Lijie Wang
> >
> > Congrats Lijie!
> >
> > Best regards,
> > Aitozi
> >
> > Jing Ge  于2022年8月19日周五 06:02写道：
> >
> > > Congrats Lijie!
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Thu, Aug 18, 2022 at 8:40 AM Terry Wang  wrote:
> > >
> > > > Congratulations, Lijie!
> > > >
> > > > On Thu, Aug 18, 2022 at 11:31 AM Leonard Xu 
> wrote:
> > > >
> > > > > Congratulations, Lijie!
> > > > >
> > > > > Best,
> > > > > Leonard
> > > > >
> > > > > > 2022年8月18日 上午11:26，Zhipeng Zhang  写道：
> > > > > >
> > > > > > Congratulations, Lijie!
> > > > > >
> > > > > > Xintong Song  于2022年8月18日周四 11:23写道：
> > > > > >>
> > > > > >> Congratulations Lijie, and welcome~!
> > > > > >>
> > > > > >> Best,
> > > > > >>
> > > > > >> Xintong
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Aug 18, 2022 at 11:12 AM Xingbo Huang <
> hxbks...@gmail.com
> > >
> > > > > wrote:
> > > > > >>
> > > > > >>> Congrats, Lijie
> > > > > >>>
> > > > > >>> Best,
> > > > > >>> Xingbo
> > > > > >>>
> > > > > >>> Lincoln Lee  于2022年8月18日周四 11:01写道：
> > > > > >>>
> > > > >  Congratulations, Lijie!
> > > > > 
> > > > >  Best,
> > > > >  Lincoln Lee
> > > > > 
> > > > > 
> > > > >  Benchao Li  于2022年8月18日周四 10:51写道：
> > > > > 
> > > > > > Congratulations Lijie!
> > > > > >
> > > > > > yanfei lei  于2022年8月18日周四 10:44写道：
> > > > > >
> > > > > >> Congratulations, Lijie!
> > > > > >>
> > > > > >> Best,
> > > > > >> Yanfei
> > > > > >>
> > > > > >> JunRui Lee  于2022年8月18日周四 10:35写道：
> > > > > >>
> > > > > >>> Congratulations, Lijie!
> > > > > >>>
> > > > > >>> Best,
> > > > > >>> JunRui
> > > > > >>>
> > > > > >>> Timo Walther  于2022年8月17日周三 19:30写道：
> > > > > >>>
> > > > >  Congratulations and welcome to the committer team :-)
> > > > > 
> > > > >  Regards,
> > > > >  Timo
> > > > > 
> > > > > 
> > > > >  On 17.08.22 12:50, Yuxin Tan wrote:
> > > > > > Congratulations, Lijie!
> > > > > >
> > > > > > Best,
> > > > > > Yuxin
> > > > > >
> > > > > >
> > > > > > Guowei Ma  于2022年8月17日周三 18:42写道：
> > > > > >
> > > > > >> Congratulations, Lijie. Welcome on board~!
> > > > > >> Best,
> > > > > >> Guowei
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Aug 17, 2022 at 6:25 PM Zhu Zhu <
> zh...@apache.org
> > >
> > > > >  wrote:
> > > > > >>
> > > > > >>> Hi everyone,
> > > > > >>>
> > > > > >>> On behalf of the PMC, I'm very happy to announce Lijie
> > Wang
> > > > > >>> as
> > > > > >>> a new Flink committer.
> > > > > >>>
> > > > > >>> Lijie has been contributing to Flink project for more
> > than
> > > 2
> > > > > > years.
> > > > > >>> He mainly works on the runtime/coordination part, doing
> > > > > >>> feature
> > > > > >>> development, problem debugging and code reviews. He has
> > > also
> > > > > >>> driven the work of FLIP-187(Adaptive Batch Scheduler)
> and
> > > > > >>> FLIP-224(Blocklist for Speculative Execution), which
> are
> > > > > > important
> > > > > >>> to run batch jobs.
> > > > > >>>
> > > > > >>> Please join me in congratulating Lijie for becoming a
> > Flink
> > > > > >>> committer!
> > > > > >>>
> > > > > >>> Cheers,
> > > > > >>> Zhu
> > > > > >>>
> > > > > >>
> > > > > >
> > > > > 
> > > > > 
> > > > > >>>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best,
> > > > > > Benchao Li
> > > > > >
> > > > > 
> > > > > >>>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > best,
> > > > > > Zhipeng
> > > > >
> > > > >
> > > >
> > > > --
> > > > Best Regards,
> > > > Terry Wang
> > > >
> > >
> >
>

Re: [VOTE] Creating benchmark channel in Apache Flink slack

[Yuan Mei]

+1 (binding) & thanks for the efforts!

Best
Yuan



On Mon, Jul 11, 2022 at 2:08 PM Yun Gao 
wrote:

> +1 (binding)
>
> Thanks Anton for driving this!
>
>
> Best,
> Yun Gao
>
>
> --
> From:Anton Kalashnikov 
> Send Time:2022 Jul. 8 (Fri.) 22:59
> To:undefined 
> Subject:[VOTE] Creating benchmark channel in Apache Flink slack
>
> Hi everyone,
>
>
> I would like to start a vote for creating the new channel in Apache
> Flink slack for sending benchamrk's result to it. This should help the
> community to notice the performance degradation on time.
>
> The discussion of this idea can be found here[1]. The ticket for this is
> here[2].
>
>
> [1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
>
> [2] https://issues.apache.org/jira/browse/FLINK-28468
>
> --
>
> Best regards,
> Anton Kalashnikov

Re: Re: Re: [DISCUSS ] Make state.backend.incremental as true by default

[Yuan Mei]

Hey Lihe Ma,

1. I do not think making different state backends with different default
setups (configs) is a bad idea. I have some similar ideas as Zakelly,
Hangxiang and you mentioned above.
Think about this: even though the incremental checkpoint for HashMap
Statebackend is released in 1.16, it may still take a while to stabilize,
and is probably not needed as a default for HashMap.

2. Incremental checkpoint for HashMap Statebackend is a feature for HashMap
Statebackend. Introducing a new state backend for this purpose does not
sound right here, as reasons have been discussed in my previous reply.

3. "Many guys show big +1" indicates that incremental checkpoint as default
is needed for RocksDB, and I am with big +1 on this as well. However, this
does not mean we can ignore the problems I've stated or ignore the needs
for other state backends. The right way is to find acceptable solutions and
to make "incremental checkpoint as default for RocksDB" happen.

Best,
Yuan

On Mon, Jun 27, 2022 at 12:28 PM Hangxiang Yu  wrote:

> The suggestion of Zakelly (Using noDefaultValue to make different
> StateBackend have its own default value) makes sense to me.
> I haven't gotten a better idea about it.
> Maybe @ro...@apache.org  who is the owner of incremental
> checkpoint support of HashMapStateBackend could share more ideas about it.
>
> On Sat, Jun 25, 2022 at 2:38 PM Lihe Ma  wrote:
>
> > Hi, Yuan Mei,
> >
> >
> > Thanks for your feedback.
> >
> > The configuration state.backend.incremental should work for all state
> > backends, which I think is reasonable. If we introduce more
> > configurations(such as state.backend.hashmap.incremental,
> > state.backend.rocksdb.incremental), it does not bring enough benefit ,
> and
> > users will be confused by the relationship of these configurations as
> > state.backend.incremental has been widely used for a long time.
> >
> >
> > If you prefer to reuse existing HashMapStateBackend to support the newly
> > incremental checkpoint feature, to be honest, I did not come up with an
> > idea to make everyone satisfied without introducing more configurations.
> It
> > seems that you have some ideas in your last reply, would you like to
> share
> > something here? Otherwise, I think we have to postpone the change to next
> > releases and let the beta feature of incremental checkpoint hashmap
> > state-backend merged first in the flink-1.16 release, even though many
> guys
> > show big +1 for the change of state.backend.incremental in this
> discussion.
> >
> >
> > Best,
> > Lihe Ma
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2022-06-19 20:39:45, "Yuan Mei"  wrote:
> > >Hey @Lihe Ma,
> > >
> > >Thanks and I appreciate your feedback!
> > >
> > >*What do I think of introducing a new state backend?*
> > >I am hesitant to follow this way of introducing a new state backend to
> > >implement `HashMapStateBackend with incremental checkpoint enabled`,
> > >because:
> > >- It causes more confusion for users: why incremental checkpoint can
> > >be *enabled
> > >through a flag for RocksDB* but having to *write a new Statebackend for
> > >HashMap*?
> > >- For the same reason above, this causes *inconsistent system behavior*
> > and
> > >potentially lead to more problems.
> > >For example, do we want to eventually keep both state backends
> > (HashMap
> > >and the newly introduced one) or deprecate one?
> > >How to deprecate one if both are widely used in prod, especially
> since
> > >people can set up state backends in the job code, which means they have
> to
> > >change/update their job code for depreciation?
> > >- Most importantly, I do not think introducing a new state backend to
> > >implement `HashMapStateBackend with incremental checkpoint enabled`
> > >resolves the problem. I will explain that later.
> > >
> > >Now let's come back to the original proposal
> > >*Make incremental checkpoint as default for RocksDB?*
> > >As I've already stated previously, I am +1 on this. And the reason is
> well
> > >explained (the motivation,  backward compatibilities, and previous
> > glitches
> > >solved by FLIP 193 e.t.c) in the original proposal and other
> > >threads/tickets.
> > >
> > >However, there is a gap between the above goal and the way proposed to
> > >achieve it: "set as the default state.backend.incremental = true". The

Re: [DISCUSS] Creating benchmark channel in Apache Flink slack

[Yuan Mei]

+1 for the proposal Anton, and thanks very much to move this effort forward!

In long term, I think it would be helpful to ask volunteers to help
watching-out the daily micro benchmark results.

But let's make it more visible as the very first step!

Best Regards,

Yuan

On Tue, Jul 5, 2022 at 10:18 PM Martijn Visser 
wrote:

> Hi Anton,
>
> I recalled that it was brought up before and it was indeed (when the
> announcement email was sent [1]).
>
> +1 for doing this.
>
> Best regards,
>
> Martijn
>
> [1] https://lists.apache.org/thread/kqjtrlnw33h1mybyx3gg4ff4390y5wbz
>
> Op di 5 jul. 2022 om 16:12 schreef Piotr Nowojski :
>
> > Hi Anton,
> >
> > Thanks for the proposal. +1 for this. The higher visibility in the
> > community of the benchmarks, benchmarking process and it's result the
> > better.
> >
> > Best,
> > Piotrek
> >
> > wt., 5 lip 2022 o 16:10 Anton Kalashnikov 
> napisał(a):
> >
> > > Hi everyone,
> > >
> > >
> > > As you know, we have Flink benchmarks[1], and all of these benchmarks
> > > constantly running several times per day. The result of this can be
> > > found in Codespeed panel[2].
> > >
> > > The problem is that only several people follow the result of these
> > > benchmarks since it is not possible to ask everybody constantly pay
> > > attention to this result page.
> > >
> > > The idea is to create `dev-benchmarks` channel(the name is
> discussable).
> > > Then we can create 'Slack app'[3] for benchmarks and send the result of
> > > each benchmark's run to this channel.
> > >
> > > I think it should make benchmarks more visible to the community.
> > >
> > > We can discuss the information that should be sent but the main idea is
> > > to follow all fails and regression of benchmarks.
> > >
> > > It also makes sense to mention that right now it is easy
> > > enough(especially after [4]) to configure our benchmarks in Jenkins
> > > since we have all scripts in the repository. So even in the case, we
> > > will lose our current server we can easily setup the new one.
> > >
> > >
> > > WDYT?
> > >
> > >
> > > [1] https://github.com/apache/flink-benchmarks
> > >
> > > [2] http://codespeed.dak8s.net:8000/changes/
> > >
> > > [3] https://plugins.jenkins.io/slack/#plugin-content-creating-your-app
> > >
> > > [4] https://issues.apache.org/jira/browse/FLINK-28407
> > >
> > > --
> > >
> > > Best regards,
> > > Anton Kalashnikov
> > >
> > >
> >
>

Re: Re: [ANNOUNCE] New Apache Flink Committers: Qingsheng Ren, Shengkai Fang

[Yuan Mei]

Congrats Qingsheng and ShengKai!

Best,

Yuan

On Tue, Jun 21, 2022 at 11:27 AM Terry Wang  wrote:

> Congratulations, Qingsheng and ShengKai!
>

Re: Re: [DISCUSS ] Make state.backend.incremental as true by default

[Yuan Mei]

Hey @Lihe Ma,

Thanks and I appreciate your feedback!

*What do I think of introducing a new state backend?*
I am hesitant to follow this way of introducing a new state backend to
implement `HashMapStateBackend with incremental checkpoint enabled`,
because:
- It causes more confusion for users: why incremental checkpoint can
be *enabled
through a flag for RocksDB* but having to *write a new Statebackend for
HashMap*?
- For the same reason above, this causes *inconsistent system behavior* and
potentially lead to more problems.
For example, do we want to eventually keep both state backends (HashMap
and the newly introduced one) or deprecate one?
How to deprecate one if both are widely used in prod, especially since
people can set up state backends in the job code, which means they have to
change/update their job code for depreciation?
- Most importantly, I do not think introducing a new state backend to
implement `HashMapStateBackend with incremental checkpoint enabled`
resolves the problem. I will explain that later.

Now let's come back to the original proposal
*Make incremental checkpoint as default for RocksDB?*
As I've already stated previously, I am +1 on this. And the reason is well
explained (the motivation,  backward compatibilities, and previous glitches
solved by FLIP 193 e.t.c) in the original proposal and other
threads/tickets.

However, there is a gap between the above goal and the way proposed to
achieve it: "set as the default state.backend.incremental = true". The way
proposed is to make incremental checkpoint default universally for all
state backends.

*Should we make incremental checkpoints as default universally* *for all
state backends?*
I am not sure about this, or at least before a few questions are
answered/discussed/solved.

1. Is incremental checkpoints as default needed for all state backends?
 Probably not, but if one state backend (like RocksDB) has strong
benefits, but others are neutral or non-neg effects, it is acceptable.
2. What is the impact of the default flag switch on the currently supported
state backends?
 RocksDB prefers incremental checkpoints, and HashMap (In Memory) does
not support incremental yet (no effect).
3. How about system extensibility and future migration?
There is a problem here. Nowadays, HashMap (In Memory) state
backends are widely used in prod as well when the state size is small
enough (in most cases based on user feedback). The support for incremental
checkpoints on HashMapStatebackend is asked by users. Then think about
this: if we set default state.backend.incremental = true, and launch the
incremental checkpoint feature for HashMapStatebackend, there is a big
upgrading cost for existing users that uses HashMapStatebackend.
 Introducing a new state backend does not solve the problem of "system
extensibility". It introduces new problems like "deprecating and migration"
as stated above.

*What is my suggestion?*
- Design a way to support different default configurations for different
state backends (I do not think it is difficult to do).
- Have a better way to solve the "system extensibility" problem.

Best,
Yuan


On Fri, Jun 17, 2022 at 11:49 AM LuNing Wang  wrote:

> Strongly +1
>
> Best,
> LuNing Wang
>
> Zakelly Lan  于2022年6月17日周五 00:15写道：
>
> > Thanks for bringing this up.
> > I'm +1 on enabling the incremental checkpoint  by default on RocksDB.
> But I
> > also agree with Yuan about not enabling this on newly implemented
> > incremental checkpoint for hashmap statebackend.
> > I am wondering can we make it behave differently for different state
> > backends when ```state.backend.incremental``` is not set? Although it may
> > bring some confusion in the default behaviour.
> >
> > On Thu, Jun 16, 2022 at 10:18 PM Lihe Ma  wrote:
> >
> > > Thanks for the suggestion, Yuan.
> > >
> > >
> > > How about naming the newly in-memory state-backend, which supports
> > > incremental checkpoint, as HeapStateBackend . And let the default
> > > state-backend still stay as HashMapStateBackend.
> > > By doing so, we can:
> > > 1) the default value of parameter state.backend.incremental could be
> set
> > > as true safely, which is so widely accepted according to the replies in
> > > this thread.
> > > 2) your newly in-memory state-backend could also be merged and users
> who
> > > want to try in-memory incremental checkpoints could also have a
> solution.
> > > We can also set the newly HeapStateBackend as default state-backend in
> > the
> > > future if the feature of incremental checkpoint is stable. What do you
> > > think?
> > >
> > >
> > > Best,
> > > Lihe Ma
> > >
> > >
> > >
> > >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

[Yuan Mei]

Congrats, Jingsong!

Best,
Yuan

On Thu, Jun 16, 2022 at 11:31 AM Yang Wang  wrote:

> Congrats, Jingsong!
>
> Best,
> Yang
>
> Zakelly Lan  于2022年6月16日周四 11:16写道：
>
> > Congrats & well deserved!
> >
> > Best,
> > Zakelly
> >
> > On Thu, Jun 16, 2022 at 10:36 AM Guowei Ma  wrote:
> >
> > > Congrats, Jingsong!
> > >
> > > Best,
> > > Guowei
> > >
> > >
> > > On Thu, Jun 16, 2022 at 9:49 AM Hangxiang Yu 
> > wrote:
> > >
> > > > Congrats, Jingsong!
> > > >
> > > > Best,
> > > > Hangxiang
> > > >
> > > > On Thu, Jun 16, 2022 at 9:46 AM Aitozi  wrote:
> > > >
> > > > > Congrats, Jingsong!
> > > > >
> > > > > Best,
> > > > > Aitozi
> > > > >
> > > > > Zhuoluo Yang  于2022年6月16日周四 09:26写道：
> > > > >
> > > > > > Many congratulations to teacher Lee!
> > > > > >
> > > > > > Thanks,
> > > > > > Zhuoluo
> > > > > >
> > > > > >
> > > > > > Dian Fu  于2022年6月16日周四 08:54写道：
> > > > > >
> > > > > > > Congratulations, Jingsong!
> > > > > > >
> > > > > > > Regards,
> > > > > > > Dian
> > > > > > >
> > > > > > > On Thu, Jun 16, 2022 at 1:08 AM Yu Li 
> wrote:
> > > > > > >
> > > > > > > > Congrats, Jingsong!
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Yu
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, 15 Jun 2022 at 15:26, Sergey Nuyanzin <
> > > snuyan...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations, Jingsong!
> > > > > > > > >
> > > > > > > > > On Wed, Jun 15, 2022 at 8:45 AM Jingsong Li <
> > > > > jingsongl...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks everyone.
> > > > > > > > > >
> > > > > > > > > > It's great to be with you in the Flink community!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Jingsong
> > > > > > > > > >
> > > > > > > > > > On Wed, Jun 15, 2022 at 2:11 PM Yun Gao
> > > > > >  > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Congratulations, Jingsong!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Yun Gao
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > >
> --
> > > > > > > > > > > From:Jing Zhang 
> > > > > > > > > > > Send Time:2022 Jun. 14 (Tue.) 11:05
> > > > > > > > > > > To:dev 
> > > > > > > > > > > Subject:Re: [ANNOUNCE] New Apache Flink PMC Member -
> > > Jingsong
> > > > > Lee
> > > > > > > > > > >
> > > > > > > > > > > Congratulations, Jingsong!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Jing Zhang
> > > > > > > > > > >
> > > > > > > > > > > Leonard Xu  于2022年6月14日周二 10:54写道：
> > > > > > > > > > >
> > > > > > > > > > > > Congratulations, Jingsong!
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Leonard
> > > > > > > > > > > >
> > > > > > > > > > > > > 2022年6月13日 下午6:52，刘首维  >
> > > 写道：
> > > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations and well deserved, Jingsong!
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Shouwei
> > > > > > > > > > > > >
> --原始邮件--
> > > > > > > > > > > > > 发件人:
> > > > > > > > > > > >
> >  "dev"
> > > > > > > > > > > >   <
> > > > > > > > > > luoyu...@alumni.sjtu.edu.cn
> > > > > > > > > > > > ;
> > > > > > > > > > > > > 发送时间:2022年6月13日(星期一) 晚上6:09
> > > > > > > > > > > > > 收件人:"dev" > > > > > > > dev@flink.apache.org
> > > > > > > > > > >;
> > > > > > > > > > > > >
> > > > > > > > > > > > > 主题:Re: [ANNOUNCE] New Apache Flink PMC
> Member -
> > > > > > Jingsong
> > > > > > > > Lee
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Jingsong!
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > >
> > > > > > > > > > > > > - 原始邮件 -
> > > > > > > > > > > > > 发件人: "Yun Tang"  > > > > > > > > > > > > 收件人: "dev"  > > > > > > > > > > > > 发送时间: 星期一, 2022年 6 月 13日 下午 6:12:24
> > > > > > > > > > > > > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member -
> > > Jingsong
> > > > > Lee
> > > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Jingsong! Well deserved.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best
> > > > > > > > > > > > > Yun Tang
> > > > > > > > > > > > > 
> > > > > > > > > > > > > From: Xingbo Huang  > > > > > > > > > > > > Sent: Monday, June 13, 2022 17:39
> > > > > > > > > > > > > To: dev  > > > > > > > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC
> Member -
> > > > > > Jingsong
> > > > > > > > Lee
> > > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Jingsong!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Xingbo
> > > > > > > > 

Re: [DISCUSS ] Make state.backend.incremental as true by default

[Yuan Mei]

Thanks for bringing this up.

I am +1 on making incremental checkpoints by default for RocksDB, but not
universally for all state backends.

Besides being widely used in prod, enabling incremental checkpoint for
RocksDB by default is also a pre-requisite when enabling task-local by
default FLINK-15507 

The incremental checkpoint for the hashmap statebackend is under review
right now. CC @ro...@ververica.com  , which is not a
good idea being enabled by default in the first version.

Best,

Yuan

On Tue, Jun 14, 2022 at 7:33 PM Jiangang Liu 
wrote:

> +1 for the suggestion. We have use the incremental checkpoint in our
> production for a long time.
>
> Hangxiang Yu  于2022年6月14日周二 15:41写道：
>
> > +1
> > It's basically enabled in most scenarios in production environments.
> > For HashMapStateBackend, it will adopt a full checkpoint even if we
> enable
> > incremental checkpoint. It will also support incremental checkpoint after
> > [1]. It's compatible.
> > BTW, I think we may also need to improve the documentation of incremental
> > checkpoints which users usually ask. There are some tickets like [2][3].
> >
> > Best,
> > Hangxiang.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-21648
> > [2] https://issues.apache.org/jira/browse/FLINK-22797
> > [3] https://issues.apache.org/jira/browse/FLINK-7449
> >
> > On Mon, Jun 13, 2022 at 7:48 PM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > > Strongly +1
> > >
> > > Best,
> > > Rui Fan
> > >
> > > On Mon, Jun 13, 2022 at 7:35 PM Martijn Visser <
> martijnvis...@apache.org
> > >
> > > wrote:
> > >
> > > > > BTW, from my knowledge, nothing would happen for
> HashMapStateBackend,
> > > > which does not support incremental checkpoint yet, when enabling
> > > > incremental checkpoints.
> > > >
> > > > Thanks Yun, if no errors would occur then definitely +1 to enable it
> by
> > > > default
> > > >
> > > > Op ma 13 jun. 2022 om 12:42 schreef Alexander Fedulov <
> > > > alexan...@ververica.com>:
> > > >
> > > > > +1
> > > > >
> > > > > From my experience, it is actually hard to come up with use cases
> > where
> > > > > incremental checkpoints should explicitly not be enabled with the
> > > RocksDB
> > > > > state backend. If the state is so small that the full snapshots do
> > not
> > > > > have any negative impact, one should consider using
> > HashMapStateBackend
> > > > > anyway.
> > > > >
> > > > > Best,
> > > > > Alexander Fedulov
> > > > >
> > > > >
> > > > > On Mon, Jun 13, 2022 at 12:26 PM Jing Ge 
> wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Glad to see the kickoff of this discussion. Thanks Lihe for
> driving
> > > > this!
> > > > > >
> > > > > > We have actually already discussed it internally a few months
> ago.
> > > > After
> > > > > > considering some corner cases, all agreed on enabling the
> > incremental
> > > > > > checkpoint as default.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Mon, Jun 13, 2022 at 12:17 PM Yun Tang 
> > wrote:
> > > > > >
> > > > > > > Strongly +1 for making incremental checkpoints as default. Many
> > > users
> > > > > > have
> > > > > > > ever been asking why this configuration is not enabled by
> > default.
> > > > > > >
> > > > > > > BTW, from my knowledge, nothing would happen for
> > > HashMapStateBackend,
> > > > > > > which does not support incremental checkpoint yet, when
> enabling
> > > > > > > incremental checkpoints.
> > > > > > >
> > > > > > >
> > > > > > > Best
> > > > > > > Yun Tang
> > > > > > > 
> > > > > > > From: Martijn Visser 
> > > > > > > Sent: Monday, June 13, 2022 18:05
> > > > > > > To: dev@flink.apache.org 
> > > > > > > Subject: Re: [DISCUSS ] Make state.backend.incremental as true
> by
> > > > > default
> > > > > > >
> > > > > > > Hi Lihe,
> > > > > > >
> > > > > > > What happens if we enable incremental checkpoints by default
> > while
> > > > the
> > > > > > used
> > > > > > > memory backend is HashMapStateBackend, which doesn't support
> > > > > incremental
> > > > > > > checkpoints?
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Martijn
> > > > > > >
> > > > > > > Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma :
> > > > > > >
> > > > > > > > Hi, Everyone,
> > > > > > > >
> > > > > > > > I would like to open a discussion on setting incremental
> > > checkpoint
> > > > > as
> > > > > > > > default behavior.
> > > > > > > >
> > > > > > > > Currently, the configuration of state.backend.incremental is
> > set
> > > as
> > > > > > false
> > > > > > > > by default. Incremental checkpoint has been adopted widely in
> > > > > industry
> > > > > > > > community for many years , and it is also well-tested from
> the
> > > > > feedback
> > > > > > > in
> > > > > > > > the community discussion. Incremental checkpointing is more
> > > > > > > light-weighted:
> > > > > > > > shorter checkpoint duration, less uploaded data and less
> > resource
> > > > > > 

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

[Yuan Mei]

I was thinking of the same thing about adding a dedicated channel for the
daily performance micro bench.

I was even thinking of asking volunteers to watch for the daily performance
micro bench.

Roman and I are drafting instructions on how to monitor performance
regression right now.

But I think this is a different topic and I will start a new thread to
discuss this once the details are finalized.

So, +1 for a dedicated channel for the daily performance micro bench notice.

Best,

Yuan

On Wed, Jun 8, 2022 at 11:38 AM Xingbo Huang  wrote:

> Thanks everyone for the great effort driving this.
>
> +1 for building a dedicated channel for CI builds.
>
> Best,
> Xingbo
>
> Martijn Visser  于2022年6月8日周三 01:32写道：
>
> > Hi Matthias,
> >
> > I've only thought about it, but I absolutely would +1 this. Making this
> > information available in dedicated channels will only improve things.
> >
> > Best regards,
> >
> > Martijn
> >
> > Op di 7 jun. 2022 om 18:12 schreef Matthias Pohl :
> >
> > > Thanks for driving the Slack channel efforts and setting everything up.
> > :-)
> > >
> > > I'm wondering whether we should extend the Slack space to also have a
> > > channel dedicated for CI builds to enable the release managers to
> monitor
> > > CI builds on master and the release branches through the Apache Flink
> > Slack
> > > workspace. It might help us in being more transparent with the release
> > > process. The same goes for the Flink performance tests [1].
> > >
> > > Are there plans around that already? I couldn't find anything related
> in
> > > [2]. Or is it worth opening another thread around that topic.
> > >
> > > Best,
> > > Matthias
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847
> > > [2] https://cwiki.apache.org/confluence/display/FLINK/Slack+Management
> > > On Tue, Jun 7, 2022 at 10:39 AM Jing Ge  wrote:
> > >
> > > > got it, thanks Martijn!
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > >
> > > > On Tue, Jun 7, 2022 at 10:38 AM Martijn Visser <
> > martijnvis...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi Jing,
> > > > >
> > > > > Yes, since this Flink Community workspace is treated as one
> company.
> > So
> > > > > everyone you want to invite, should considered a 'coworker'.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > Op za 4 jun. 2022 om 20:02 schreef Jing Ge :
> > > > >
> > > > >> Hi Xingtong,
> > > > >>
> > > > >> While inviting new members, there are two options: "From another
> > > > company"
> > > > >> vs "Your coworker". In this case, we should always choose "Your
> > > > coworker"
> > > > >> to add new members to the Apache Flink workspace, right?
> > > > >>
> > > > >> Best regards,
> > > > >> Jing
> > > > >>
> > > > >> On Fri, Jun 3, 2022 at 1:10 PM Yuan Mei 
> > > wrote:
> > > > >>
> > > > >>> Thanks, Xintong and Jark the great effort driving this, and
> > everyone
> > > > for
> > > > >>> making this possible.
> > > > >>>
> > > > >>> I've also Twittered this announcement on our Apache Flink Twitter
> > > > >>> account.
> > > > >>>
> > > > >>> Best
> > > > >>>
> > > > >>> Yuan
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Jun 3, 2022 at 12:54 AM Jing Ge 
> > wrote:
> > > > >>>
> > > > >>>> Thanks everyone for your effort!
> > > > >>>>
> > > > >>>> Best regards,
> > > > >>>> Jing
> > > > >>>>
> > > > >>>> On Thu, Jun 2, 2022 at 4:17 PM Martijn Visser <
> > > > martijnvis...@apache.org>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Thanks everyone for joining! It's good to see so many ha

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

[Yuan Mei]

Thanks, Xintong and Jark the great effort driving this, and everyone for
making this possible.

I've also Twittered this announcement on our Apache Flink Twitter account.

Best

Yuan



On Fri, Jun 3, 2022 at 12:54 AM Jing Ge  wrote:

> Thanks everyone for your effort!
>
> Best regards,
> Jing
>
> On Thu, Jun 2, 2022 at 4:17 PM Martijn Visser 
> wrote:
>
>> Thanks everyone for joining! It's good to see so many have joined in such
>> a short time already. I've just refreshed the link which you can always
>> find on the project website [1]
>>
>> Best regards, Martijn
>>
>> [1] https://flink.apache.org/community.html#slack
>>
>> Op do 2 jun. 2022 om 11:42 schreef Jingsong Li :
>>
>>> Thanks Xingtong, Jark, Martijn and Robert for making this possible!
>>>
>>> Best,
>>> Jingsong
>>>
>>>
>>> On Thu, Jun 2, 2022 at 5:32 PM Jark Wu  wrote:
>>>
 Thank Xingtong for making this possible!

 Cheers,
 Jark Wu

 On Thu, 2 Jun 2022 at 15:31, Xintong Song 
 wrote:

 > Hi everyone,
 >
 > I'm very happy to announce that the Apache Flink community has
 created a
 > dedicated Slack workspace [1]. Welcome to join us on Slack.
 >
 > ## Join the Slack workspace
 >
 > You can join the Slack workspace by either of the following two ways:
 > 1. Click the invitation link posted on the project website [2].
 > 2. Ask anyone who already joined the Slack workspace to invite you.
 >
 > We recommend 2), if available. Due to Slack limitations, the
 invitation
 > link in 1) expires and needs manual updates after every 100 invites.
 If it
 > is expired, please reach out to the dev / user mailing lists.
 >
 > ## Community rules
 >
 > When using the community Slack workspace, please follow these
 community
 > rules:
 > * *Be respectful* - This is the most important rule!
 > * All important decisions and conclusions *must be reflected back to
 the
 > mailing lists*. "If it didn’t happen on a mailing list, it didn’t
 happen."
 > - The Apache Mottos [3]
 > * Use *Slack threads* to keep parallel conversations from
 overwhelming a
 > channel.
 > * Please *do not direct message* people for troubleshooting, Jira
 assigning
 > and PR review. These should be picked-up voluntarily.
 >
 >
 > ## Maintenance
 >
 >
 > Committers can refer to this wiki page [4] for information needed for
 > maintaining the Slack workspace.
 >
 >
 > Thanks Jark, Martijn and Robert for helping setting up the Slack
 workspace.
 >
 >
 > Best,
 >
 > Xintong
 >
 >
 > [1] https://apache-flink.slack.com/
 >
 > [2] https://flink.apache.org/community.html#slack
 >
 > [3] http://theapacheway.com/on-list/
 >
 > [4]
 https://cwiki.apache.org/confluence/display/FLINK/Slack+Management
 >

>>>

Re: [VOTE] Creating an Apache Flink slack workspace

[Yuan Mei]

+1 (binding)

This facilitates people collaborating on the same project from different
organizations. I really like this idea.

On Thu, May 19, 2022 at 12:43 PM Peter Huang 
wrote:

> +1 (non-binding)
>
>
> Best Regards
> Peter Huang
>
> On Wed, May 18, 2022 at 9:33 PM Leonard Xu  wrote:
>
> > Thanks Xintong for driving this.
> >
> >  +1
> >
> > Best,
> > Leonard
> >
> > > 2022年5月19日 上午11:11，Zhou, Brian  写道：
> > >
> > > +1 (non-binding)  Slack is a better place for code sharing and quick
> > discussion.
> > >
> > > Regards,
> > > Brian Zhou
> > >
> > > -Original Message-
> > > From: Yun Tang 
> > > Sent: Thursday, May 19, 2022 10:32 AM
> > > To: dev
> > > Subject: Re: [VOTE] Creating an Apache Flink slack workspace
> > >
> > >
> > > [EXTERNAL EMAIL]
> > >
> > > Thanks Xintong for driving this. +1 from my side.
> > >
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Zhu Zhu 
> > > Sent: Wednesday, May 18, 2022 17:08
> > > To: dev 
> > > Subject: Re: [VOTE] Creating an Apache Flink slack workspace
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Timo Walther  于2022年5月18日周三 16:52写道：
> > >>
> > >> +1 (binding)
> > >>
> > >> Thanks,
> > >> Timo
> > >>
> > >>
> > >> On 17.05.22 20:44, Gyula Fóra wrote:
> > >>> +1 (binding)
> > >>>
> > >>> On Tue, 17 May 2022 at 19:52, Yufei Zhang 
> wrote:
> > >>>
> >  +1 (nonbinding)
> > 
> >  On Tue, May 17, 2022 at 5:29 PM Márton Balassi <
> > balassi.mar...@gmail.com>
> >  wrote:
> > 
> > > +1 (binding)
> > >
> > > On Tue, May 17, 2022 at 11:00 AM Jingsong Li <
> jingsongl...@gmail.com
> > >
> > > wrote:
> > >
> > >> Thank Xintong for driving this work.
> > >>
> > >> +1
> > >>
> > >> Best,
> > >> Jingsong
> > >>
> > >> On Tue, May 17, 2022 at 4:49 PM Martijn Visser <
> >  martijnvis...@apache.org
> > >>
> > >> wrote:
> > >>
> > >>> +1 (binding)
> > >>>
> > >>> On Tue, 17 May 2022 at 10:38, Yu Li  wrote:
> > >>>
> >  +1 (binding)
> > 
> >  Thanks Xintong for driving this!
> > 
> >  Best Regards,
> >  Yu
> > 
> > 
> >  On Tue, 17 May 2022 at 16:32, Robert Metzger <
> metrob...@gmail.com
> > >
> > >>> wrote:
> > 
> > > Thanks for starting the VOTE!
> > >
> > > +1 (binding)
> > >
> > >
> > >
> > > On Tue, May 17, 2022 at 10:29 AM Jark Wu 
> >  wrote:
> > >
> > >> Thank Xintong for driving this work.
> > >>
> > >> +1 from my side (binding)
> > >>
> > >> Best,
> > >> Jark
> > >>
> > >> On Tue, 17 May 2022 at 16:24, Xintong Song <
> > > tonysong...@gmail.com>
> > > wrote:
> > >>
> > >>> Hi everyone,
> > >>>
> > >>> As previously discussed in [1], I would like to open a vote
> >  on
> >  creating
> > >> an
> > >>> Apache Flink slack workspace channel.
> > >>>
> > >>> The proposed actions include:
> > >>> - Creating a dedicated slack workspace with the name Apache
> > > Flink
> >  that
> > > is
> > >>> controlled and maintained by the Apache Flink PMC
> > >>> - Updating the Flink website about rules for using various
> > > communication
> > >>> channels
> > >>> - Setting up an Archive for the Apache Flink slack
> > >>> - Revisiting this initiative by the end of 2022
> > >>>
> > >>> The vote will last for at least 72 hours, and will be
> >  accepted
> > >> by a
> > >>> consensus of active PMC members.
> > >>>
> > >>> Best,
> > >>>
> > >>> Xintong
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>
> >
> >
>

Re: [ANNOUNCE] New Flink PMC member: Yang Wang

[Yuan Mei]

Congrats and well Deserved, Yang!

Best,
Yuan

On Thu, May 5, 2022 at 8:21 PM Nicholas Jiang 
wrote:

> Congrats Yang!
>
> Best regards,
> Nicholas Jiang
>
> On 2022/05/05 11:18:10 Xintong Song wrote:
> > Hi all,
> >
> > I'm very happy to announce that Yang Wang has joined the Flink PMC!
> >
> > Yang has been consistently contributing to our community, by contributing
> > codes, participating in discussions, mentoring new contributors,
> answering
> > questions on mailing lists, and giving talks on Flink at
> > various conferences and events. He is one of the main contributors and
> > maintainers in Flink's Native Kubernetes / Yarn integrations and the
> Flink
> > Kubernetes Operator.
> >
> > Congratulations and welcome, Yang!
> >
> > Thank you~
> >
> > Xintong Song (On behalf of the Apache Flink PMC)
> >
>

Re: [ANNOUNCE] Apache Flink 1.15.0 released

[Yuan Mei]

Great!

Thanks, Yun Gao, Till, and Joe for driving the release, and thanks to
everyone for making this release happen!

Best
Yuan

On Thu, May 5, 2022 at 4:40 PM Leonard Xu  wrote:

> Congratulations!
>
> Thanks Yun Gao, Till and Joe for the great work as our release manager and
> everyone who involved.
>
> Best,
> Leonard
>
>
>
> > 2022年5月5日 下午4:30，Yang Wang  写道：
> >
> > Congratulations!
> >
> > Thanks Yun Gao, Till and Joe for driving this release and everyone who
> made
> > this release happen.
>
>

[jira] [Created] (FLINK-27214) Build Failed on state backend benchmark

[Yuan Mei (Jira)]

Yuan Mei created FLINK-27214:


 Summary: Build Failed on state backend benchmark
 Key: FLINK-27214
 URL: https://issues.apache.org/jira/browse/FLINK-27214
 Project: Flink
  Issue Type: Bug
Reporter: Yuan Mei


Failed build 72 of flink-statebackend-benchmarks-java11 
([Open|http://codespeed.dak8s.net:8080/job/flink-statebackend-benchmarks-java11/72/]):
 hudson.AbortException: script returned exit code 1
 
Failed build 214 of flink-statebackend-benchmarks-java8 
([Open|http://codespeed.dak8s.net:8080/job/flink-statebackend-benchmarks-java8/214/]):
 hudson.AbortException: script returned exit code 1



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Flink 1.15 Stabilisation Sync

[Yuan Mei]

FLINK-26985 was discovered just before last weekend.

We will get it resolved first thing after the holiday (tomorrow).

Best
Yuan

On Tue, Apr 5, 2022 at 5:37 PM Yun Gao  wrote:

> Hi Robert,
>
> Very sorry for the long delay before the rc1 could be published.
>
> For the open critical issues, I previously checked with the owners of the
> issues
> that they are optional to the release, thus the only blocker issue is the
> FLINK-26985.
>
> Besides to the blocker issues we are waiting for the release blog to be
> done. As a whole,
> if there is no new blocker issues that must be fixed, both of the current
> blockers are expected
> to be finished as soon as possible inside this week and we'll create the
> rc1 immediately
> after that. We'll also put more efforts to track and address the remaining
> blockers.
>
> Best,
> Yun Gao
>
>
> --
> From:Robert Metzger 
> Send Time:2022 Apr. 5 (Tue.) 13:34
> To:dev 
> Subject:Re: Flink 1.15 Stabilisation Sync
>
> Thanks a lot for the update.
>
> From the burndown board [1] it seems that there's only one blocker left,
> which has already a fix in review [2].
> Are we also planning to address the open Critical tickets before opening
> the first voting release candidate?
> When do you expect 1.15 to be out?
>
> [1]
>
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=505=detail=FLINK-26985
> [2] https://issues.apache.org/jira/browse/FLINK-26985
>
>
> On Mon, Mar 28, 2022 at 11:18 PM Johannes Moser  wrote:
>
> > Dear Community,
> >
> > We are fairly on track with stabilising the 1.15 release, that’s why Yun
> > Gao and me think we don’t need the sync anymore.
> >
> > So we skip it for now, starting from tomorrow. If the situation might
> > change. We will let you know.
> >
> > Best,
> > Joe
>
>

[jira] [Created] (FLINK-26992) PojoSerializer may cause concurrent exception passing directly between threads

[Yuan Mei (Jira)]

Yuan Mei created FLINK-26992:


 Summary: PojoSerializer may cause concurrent exception passing 
directly between threads
 Key: FLINK-26992
 URL: https://issues.apache.org/jira/browse/FLINK-26992
 Project: Flink
  Issue Type: Bug
Reporter: Yuan Mei






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - David Morávek

[Yuan Mei]

Congratulations, David!

Best Regards,
Yuan

On Sat, Mar 5, 2022 at 8:13 PM Roman Khachatryan  wrote:

> Congratulations, David!
>
> Regards,
> Roman
>
> On Fri, Mar 4, 2022 at 7:54 PM Austin Cawley-Edwards
>  wrote:
> >
> > Congrats David!
> >
> > On Fri, Mar 4, 2022 at 12:18 PM Zhilong Hong 
> wrote:
> >
> > > Congratulations, David!
> > >
> > > Best,
> > > Zhilong
> > >
> > > On Sat, Mar 5, 2022 at 1:09 AM Piotr Nowojski 
> > > wrote:
> > >
> > > > Congratulations :)
> > > >
> > > > pt., 4 mar 2022 o 16:04 Aitozi  napisał(a):
> > > >
> > > > > Congratulations David!
> > > > >
> > > > > Ingo Bürk  于2022年3月4日周五 22:56写道：
> > > > >
> > > > > > Congrats, David!
> > > > > >
> > > > > > On 04.03.22 12:34, Robert Metzger wrote:
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > On behalf of the PMC, I'm very happy to announce David Morávek
> as a
> > > > new
> > > > > > > Flink committer.
> > > > > > >
> > > > > > > His first contributions to Flink date back to 2019. He has been
> > > > > > > increasingly active with reviews and driving major initiatives
> in
> > > the
> > > > > > > community. David brings valuable experience from being a
> committer
> > > in
> > > > > the
> > > > > > > Apache Beam project to Flink.
> > > > > > >
> > > > > > >
> > > > > > > Please join me in congratulating David for becoming a Flink
> > > > committer!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Robert
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: Re: Change of focus

[Yuan Mei]

Thanks Till for everything you've done for the community!
Good luck with your new adventure and best wishes to your new life!

Best Regards,
Yuan

On Tue, Mar 1, 2022 at 10:35 AM Zhu Zhu  wrote:

> Thank you for all the efforts and good luck for the new adventure, Till!
>
> Thanks,
> Zhu
>
> Terry  于2022年3月1日周二 10:26写道：
>
> > Thanks a lot for your efforts! Good Luck!
> >
> > Jiangang Liu  于2022年3月1日周二 10:18写道：
> >
> > > Thanks for the efforts and help in flink, Till. Good luck!
> > >
> > > Best
> > > Liu Jiangang
> > >
> > > Lijie Wang  于2022年3月1日周二 09:53写道：
> > >
> > > > Thanks for all your efforts Till. Good luck !
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Yun Gao  于2022年3月1日周二 01:15写道：
> > > >
> > > > > Very thanks Till for all the efforts! Good luck for the next
> chapter~
> > > > >
> > > > > Best,
> > > > > Yun
> > > > >
> > > > > --
> > > > > Sender:Piotr Nowojski
> > > > > Date:2022/02/28 22:10:46
> > > > > Recipient:dev
> > > > > Theme:Re: Change of focus
> > > > >
> > > > > Good luck Till and thanks for all of your efforts.
> > > > >
> > > > > Best,
> > > > > Piotrek
> > > > >
> > > > > pon., 28 lut 2022 o 15:06 Aitozi 
> napisał(a):
> > > > >
> > > > > > Good luck with the next chapter, will miss you :)
> > > > > >
> > > > > > Best,
> > > > > > Aitozi
> > > > > >
> > > > > > Jark Wu  于2022年2月28日周一 21:28写道：
> > > > > >
> > > > > > > Thank you Till for every things. It's great to work with you.
> > Good
> > > > > luck!
> > > > > > >
> > > > > > > Best,
> > > > > > > Jark
> > > > > > >
> > > > > > > On Mon, 28 Feb 2022 at 21:26, Márton Balassi <
> > > > balassi.mar...@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thank you, Till. Good luck with the next chapter. :-)
> > > > > > > >
> > > > > > > > On Mon, Feb 28, 2022 at 1:49 PM Flavio Pompermaier <
> > > > > > pomperma...@okkam.it
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Good luck for your new adventure Till!
> > > > > > > > >
> > > > > > > > > On Mon, Feb 28, 2022 at 12:00 PM Till Rohrmann <
> > > > > trohrm...@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > I wanted to let you know that I will be less active in
> the
> > > > > > community
> > > > > > > > > > because I’ve decided to start a new chapter in my life.
> > > Hence,
> > > > > > please
> > > > > > > > > don’t
> > > > > > > > > > wonder if I might no longer be very responsive on mails
> and
> > > > JIRA
> > > > > > > > issues.
> > > > > > > > > >
> > > > > > > > > > It is great being part of such a great community with so
> > many
> > > > > > amazing
> > > > > > > > > > people. Over the past 7,5 years, I’ve learned a lot
> thanks
> > to
> > > > you
> > > > > > and
> > > > > > > > > > together we have shaped how people think about stream
> > > > processing
> > > > > > > > > nowadays.
> > > > > > > > > > This is something we can be very proud of. I am sure that
> > the
> > > > > > > community
> > > > > > > > > > will continue innovating and setting the pace for what is
> > > > > possible
> > > > > > > with
> > > > > > > > > > real time processing. I wish you all godspeed!
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Till
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao

[Yuan Mei]

Congratulations!

Best Regards,
Yuan

On Thu, Feb 17, 2022 at 5:17 PM Aitozi  wrote:

> Congratulations!
>
> Best,
> Aitozi
>
>
> Guowei Ma  于2022年2月17日周四 15:53写道：
>
> > Congratulations
> >
> > Best,
> > Guowei
> >
> >
> > On Thu, Feb 17, 2022 at 3:29 PM Yang Wang  wrote:
> >
> > > Congratulations to all of you!
> > >
> > > Best,
> > > Yang
> > >
> > > Lijie Wang  于2022年2月17日周四 14:20写道：
> > >
> > > > Congratulations to all of you!
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Leonard Xu  于2022年2月17日周四 12:13写道：
> > > >
> > > > > Congratulations !
> > > > >
> > > > > Best,
> > > > > Leonard
> > > > >
> > > > > > 2022年2月17日 上午11:09，Yun Tang  写道：
> > > > > >
> > > > > > Congratulations to all of you!
> > > > > >
> > > > > > Best,
> > > > > > Yun Tang
> > > > > > 
> > > > > > From: Yangze Guo 
> > > > > > Sent: Thursday, February 17, 2022 10:44
> > > > > > To: dev 
> > > > > > Cc: ro...@apache.org 
> > > > > > Subject: Re: [ANNOUNCE] New Flink PMC members: Igal Shilman,
> > > Konstantin
> > > > > Knauf and Yun Gao
> > > > > >
> > > > > > Congratulations! Well deserved.
> > > > > >
> > > > > > Best,
> > > > > > Yangze Guo
> > > > > >
> > > > > > On Thu, Feb 17, 2022 at 10:15 AM Paul Lam  >
> > > > wrote:
> > > > > >>
> > > > > >> Congrats to all of you!
> > > > > >>
> > > > > >> Best,
> > > > > >> Paul Lam
> > > > > >>
> > > > > >>> 2022年2月17日 01:24，Roman Khachatryan  写道：
> > > > > >>>
> > > > > >>> Congratulations!
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>> Roman
> > > > > >>>
> > > > > >>> On Wed, Feb 16, 2022 at 5:22 PM Matthias Pohl <
> > > > matth...@ververica.com>
> > > > > wrote:
> > > > > 
> > > > >  Congratulations to all of you :)
> > > > > 
> > > > >  On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann <
> > > trohrm...@apache.org
> > > > >
> > > > > wrote:
> > > > > 
> > > > > > Congratulations! Well deserved!
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser <
> > > > > mart...@ververica.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Congratulations to all of you!
> > > > > >>
> > > > > >> Best regards,
> > > > > >>
> > > > > >> Martijn Visser
> > > > > >> https://twitter.com/MartijnVisser82
> > > > > >>
> > > > > >>
> > > > > >> On Wed, 16 Feb 2022 at 14:29, Fabian Paul  >
> > > > wrote:
> > > > > >>
> > > > > >>> Congrats to all three of you, well deserved.
> > > > > >>>
> > > > > >>> Best,
> > > > > >>> Fabian
> > > > > >>>
> > > > > >>> On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger <
> > > > > rmetz...@apache.org>
> > > > > >>> wrote:
> > > > > 
> > > > >  Hi all,
> > > > > 
> > > > >  I would like to formally announce a few new Flink PMC
> > members
> > > on
> > > > > the
> > > > > >> dev@
> > > > >  list. The PMC has not done a good job of always announcing
> > new
> > > > PMC
> > > > > >>> members
> > > > >  (and committers) recently. I'll try to keep an eye on this
> > in
> > > > the
> > > > > >> future
> > > > > >>> to
> > > > >  improve the situation.
> > > > > 
> > > > >  Nevertheless, I'm very happy to announce some very active
> > > > > community
> > > > > >>> members
> > > > >  as new PMC members:
> > > > > 
> > > > >  - Igal Shilman, added to the PMC in October 2021
> > > > >  - Konstantin Knauf, added to the PMC in January 2022
> > > > >  - Yun Gao, added to the PMC in February 2022
> > > > > 
> > > > >  Please join me in welcoming them to the Flink PMC!
> > > > > 
> > > > >  Best,
> > > > >  Robert
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang

[Yuan Mei]

Congratulations!

Best Regards,
Yuan

On Thu, Feb 17, 2022 at 5:15 PM Aitozi  wrote:

> Congratulations!
>
> Best,
> Aitozi
>
> Guowei Ma  于2022年2月17日周四 15:52写道：
>
> > Congratulations to Feng and Zhipeng!
> > Best,
> > Guowei
> >
> >
> > On Thu, Feb 17, 2022 at 3:30 PM Yang Wang  wrote:
> >
> > > Congratulations!
> > >
> > > Best,
> > > Yang
> > >
> > > Lijie Wang  于2022年2月17日周四 14:21写道：
> > >
> > > > Congratulations to all of you!
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Zakelly Lan  于2022年2月17日周四 12:21写道：
> > > >
> > > > > Congratulations :-D
> > > > >
> > > > > On Thu, Feb 17, 2022 at 12:15 PM Leonard Xu 
> > wrote:
> > > > >
> > > > > > Congratulations !
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Leonard
> > > > > >
> > > > > > > 2022年2月17日 上午11:17，Paul Lam  写道：
> > > > > > >
> > > > > > > Congrats! Well deserved!
> > > > > > >
> > > > > > > Best,
> > > > > > > Paul Lam
> > > > > > >
> > > > > > >> 2022年2月16日 21:30，Robert Metzger  写道：
> > > > > > >>
> > > > > > >> Hi everyone,
> > > > > > >>
> > > > > > >> On behalf of the PMC, I'm very happy to announce two new Flink
> > > > > > >> committers: Feng Wang and Zhipeng Zhang!
> > > > > > >>
> > > > > > >> Feng is one of the most active Flink evangelists in China,
> with
> > > > plenty
> > > > > > of
> > > > > > >> public talks, blog posts and other evangelization activities.
> > The
> > > > PMC
> > > > > > wants
> > > > > > >> to recognize and value these efforts by making Feng a
> committer!
> > > > > > >>
> > > > > > >> Zhipeng Zhang has made significant contributions to flink-ml,
> > like
> > > > > most
> > > > > > of
> > > > > > >> the FLIPs for our ML efforts.
> > > > > > >>
> > > > > > >> Please join me in welcoming them as committers!
> > > > > > >>
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Robert
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Checkpointing (partially) failing jobs

[Yuan Mei]

ointing of the sources needs to support the
> > > rescaling case, otherwise it does not work. Is there currently a source
> > > implementation where this wouldn't work? For Kafka it should work
> because
> > > we store the offset per assigned partition. For Kinesis it is probably
> > the
> > > same. For the Filesource we store the set of unread input splits in the
> > > source coordinator and the state of the assigned splits in the sources.
> > > This should probably also work since new splits are only handed out to
> > > running tasks.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Feb 7, 2022 at 10:29 AM Yuan Mei 
> wrote:
> > >
> > > > Hey Till,
> > > >
> > > > > Why rescaling is a problem for pipelined regions/independent
> > execution
> > > > subgraphs:
> > > >
> > > > Take a simplified example :
> > > > job graph : source  (2 instances) -> sink (2 instances)
> > > > execution graph:
> > > > source (1/2)  -> sink (1/2)   [pieplined region 1]
> > > > source (2/2)  -> sink (2/2)   [pieplined region 2]
> > > >
> > > > Let's assume checkpoints are still triggered globally, meaning
> > different
> > > > pipelined regions share the global checkpoint id (PR1 CP1 matches
> with
> > PR2
> > > > CP1).
> > > >
> > > > Now let's assume PR1 completes CP10 and PR2 completes CP8.
> > > >
> > > > Let's say we want to rescale to parallelism 3 due to increased input.
> > > >
> > > > - Notice that we can not simply rescale based on the latest completed
> > > > checkpoint (CP8), because PR1 has already had data (CP8 -> CP10)
> output
> > > > externally.
> > > > - Can we take CP10 from PR1 and CP8 from PR2? I think it depends on
> > how the
> > > > source's offset redistribution is implemented.
> > > >The answer is yes if we treat each input partition as independent
> > from
> > > > each other, *but I am not sure whether we can make that assumption*.
> > > >
> > > > If not, the rescaling cannot happen until PR1 and PR2 are aligned
> with
> > CPs.
> > > >
> > > > Best
> > > > -Yuan
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Feb 7, 2022 at 4:17 PM Till Rohrmann 
> > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Yuan and Gen could you elaborate why rescaling is a problem if we
> say
> > > > that
> > > > > separate pipelined regions can take checkpoints independently?
> > > > > Conceptually, I somehow think that a pipelined region that is
> failed
> > and
> > > > > cannot create a new checkpoint is more or less the same as a
> > pipelined
> > > > > region that didn't get new input or a very very slow pipelined
> region
> > > > which
> > > > > couldn't read new records since the last checkpoint (assuming that
> > the
> > > > > checkpoint coordinator can create a global checkpoint by combining
> > > > > individual checkpoints (e.g. taking the last completed checkpoint
> > from
> > > > each
> > > > > pipelined region)). If this comparison is correct, then this would
> > mean
> > > > > that we have rescaling problems under the latter two cases.
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Mon, Feb 7, 2022 at 8:55 AM Gen Luo 
> wrote:
> > > > >
> > > > > > Hi Gyula,
> > > > > >
> > > > > > Thanks for sharing the idea. As Yuan mentioned, I think we can
> > discuss
> > > > > this
> > > > > > within two scopes. One is the job subgraph, the other is the
> > execution
> > > > > > subgraph, which I suppose is the same as PipelineRegion.
> > > > > >
> > > > > > An idea is to individually checkpoint the PipelineRegions, for
> the
> > > > > > recovering in a single run.
> > > > > >
> > > > > > Flink has now supported PipelineRegion based failover, with a
> > subset
> > > > of a
> > > > > > global checkpoint snapshot. The checkpoint b

Re: [DISCUSS] Checkpointing (partially) failing jobs

[Yuan Mei]

Hey Till,

> Why rescaling is a problem for pipelined regions/independent execution
subgraphs:

Take a simplified example :
job graph : source  (2 instances) -> sink (2 instances)
execution graph:
source (1/2)  -> sink (1/2)   [pieplined region 1]
source (2/2)  -> sink (2/2)   [pieplined region 2]

Let's assume checkpoints are still triggered globally, meaning different
pipelined regions share the global checkpoint id (PR1 CP1 matches with PR2
CP1).

Now let's assume PR1 completes CP10 and PR2 completes CP8.

Let's say we want to rescale to parallelism 3 due to increased input.

- Notice that we can not simply rescale based on the latest completed
checkpoint (CP8), because PR1 has already had data (CP8 -> CP10) output
externally.
- Can we take CP10 from PR1 and CP8 from PR2? I think it depends on how the
source's offset redistribution is implemented.
   The answer is yes if we treat each input partition as independent from
each other, *but I am not sure whether we can make that assumption*.

If not, the rescaling cannot happen until PR1 and PR2 are aligned with CPs.

Best
-Yuan







On Mon, Feb 7, 2022 at 4:17 PM Till Rohrmann  wrote:

> Hi everyone,
>
> Yuan and Gen could you elaborate why rescaling is a problem if we say that
> separate pipelined regions can take checkpoints independently?
> Conceptually, I somehow think that a pipelined region that is failed and
> cannot create a new checkpoint is more or less the same as a pipelined
> region that didn't get new input or a very very slow pipelined region which
> couldn't read new records since the last checkpoint (assuming that the
> checkpoint coordinator can create a global checkpoint by combining
> individual checkpoints (e.g. taking the last completed checkpoint from each
> pipelined region)). If this comparison is correct, then this would mean
> that we have rescaling problems under the latter two cases.
>
> Cheers,
> Till
>
> On Mon, Feb 7, 2022 at 8:55 AM Gen Luo  wrote:
>
> > Hi Gyula,
> >
> > Thanks for sharing the idea. As Yuan mentioned, I think we can discuss
> this
> > within two scopes. One is the job subgraph, the other is the execution
> > subgraph, which I suppose is the same as PipelineRegion.
> >
> > An idea is to individually checkpoint the PipelineRegions, for the
> > recovering in a single run.
> >
> > Flink has now supported PipelineRegion based failover, with a subset of a
> > global checkpoint snapshot. The checkpoint barriers are spread within a
> > PipelineRegion, so the checkpointing of individual PipelineRegions is
> > actually independent. Since in a single run of a job, the PipelineRegions
> > are fixed, we can individually checkpoint separated PipelineRegions,
> > despite what status the other PipelineRegions are, and use a snapshot of
> a
> > failing region to recover, instead of the subset of a global snapshot.
> This
> > can support separated job subgraphs as well, since they will also be
> > separated into different PipelineRegions. I think this can fulfill your
> > needs.
> >
> > In fact the individual snapshots of all PipelineRegions can form a global
> > snapshot, and the alignment of snapshots of individual regions is not
> > necessary. But rescaling this global snapshot can be potentially
> complex. I
> > think it's better to use the individual snapshots in a single run, and
> take
> > a global checkpoint/savepoint before restarting the job, rescaling it or
> > not.
> >
> > A major issue of this plan is that it breaks the checkpoint mechanism of
> > Flink. As far as I know, even in the approximate recovery, the snapshot
> > used to recover a single task is still a part of a global snapshot. To
> > implement the individual checkpointing of PipelineRegions, there may need
> > to be a checkpoint coordinator for each PipelineRegion, and a new global
> > checkpoint coordinator. When the scale goes up, there can be many
> > individual regions, which can be a big burden to the job manager. The
> > meaning of the checkpoint id will also be changed, which can affect many
> > aspects. There can be lots of work and risks, and the risks still exist
> if
> > we only individually checkpoint separated job subgraphs, since the
> > mechanism is still broken. If that is what you need, maybe separating
> them
> > into different jobs is an easier and better choice, as Caizhi and Yuan
> > mentioned.
> >
> > On Mon, Feb 7, 2022 at 11:39 AM Yuan Mei  wrote:
> >
> > > Hey Gyula,
> > >
> > > That's a very interesting idea. The discussion about the `Individual`
> vs
> > > `Global` checkpoint was raised before, but the main concern was from
> two
>

Re: [DISCUSS] Checkpointing (partially) failing jobs

[Yuan Mei]

Hey Gyula,

That's a very interesting idea. The discussion about the `Individual` vs
`Global` checkpoint was raised before, but the main concern was from two
aspects:

- Non-deterministic replaying may lead to an inconsistent view of checkpoint
- It is not easy to form a clear cut of past and future and hence no clear
cut of where the start point of the source should begin to replay from.

Starting from independent subgraphs as you proposed may be a good starting
point. However, when we talk about subgraph, do we mention it as a job
subgraph (each vertex is one or more operators) or execution subgraph (each
vertex is a task instance)?

If it is a job subgraph, then indeed, why not separate it into multiple
jobs as Caizhi mentioned.
If it is an execution subgraph, then it is difficult to handle rescaling
due to inconsistent views of checkpoints between tasks of the same operator.

`Individual/Subgraph Checkpointing` is definitely an interesting direction
to think of, and I'd love to hear more from you!

Best,

Yuan







On Mon, Feb 7, 2022 at 10:16 AM Caizhi Weng  wrote:

> Hi Gyula!
>
> Thanks for raising this discussion. I agree that this will be an
> interesting feature but I actually have some doubts about the motivation
> and use case. If there are multiple individual subgraphs in the same job,
> why not just distribute them to multiple jobs so that each job contains
> only one individual graph and can now fail without disturbing the others?
>
>
> Gyula Fóra  于2022年2月7日周一 05:22写道：
>
> > Hi all!
> >
> > At the moment checkpointing only works for healthy jobs with all running
> > (or some finished) tasks. This sounds reasonable in most cases but there
> > are a few applications where it would make sense to checkpoint failing
> jobs
> > as well.
> >
> > Due to how the checkpointing mechanism works, subgraphs that have a
> failing
> > task cannot be checkpointed without violating the exactly-once semantics.
> > However if the job has multiple independent subgraphs (that are not
> > connected to each other), even if one subgraph is failing, the other
> > completely running one could be checkpointed. In these cases the tasks of
> > the failing subgraph could simply inherit the last successful checkpoint
> > metadata (before they started failing). This logic would produce a
> > consistent checkpoint.
> >
> > The job as a whole could now make stateful progress even if some
> subgraphs
> > are constantly failing. This can be very valuable if for some reason the
> > job has a larger number of independent subgraphs that are expected to
> fail
> > every once in a while, or if some subgraphs can have longer downtimes
> that
> > would now cause the whole job to stall.
> >
> > What do you think?
> >
> > Cheers,
> > Gyula
> >
>

Re: [DISCUSS] Pushing Apache Flink 1.15 Feature Freeze

[Yuan Mei]

+1 extending feature freeze for one week.

Code Freeze on 6th (end of Spring Festival) is equivalent to say code
freeze at the end of this week for Chinese buddies, since Spring Festival
starts next week.
It also means they should be partially available during the holiday,
otherwise they would block the release if any unexpected issues arise.

The situation sounds a bit stressed and can be resolved very well by
extending the freeze date for a bit.

Best
Yuan

On Wed, Jan 26, 2022 at 11:18 AM Yun Tang  wrote:

> Since the official Spring Festival holidays in China starts from Jan 31th
> to Feb 6th, and many developers in China would enjoy the holidays at that
> time.
> +1 for extending the feature freeze.
>
> Best
> Yun Tang
> 
> From: Jingsong Li 
> Sent: Wednesday, January 26, 2022 10:32
> To: dev 
> Subject: Re: [DISCUSS] Pushing Apache Flink 1.15 Feature Freeze
>
> +1 for extending the feature freeze.
>
> Thanks Joe for driving.
>
> Best,
> Jingsong
>
> On Wed, Jan 26, 2022 at 12:04 AM Martijn Visser 
> wrote:
> >
> > Hi all,
> >
> > +1 for extending the feature freeze. We could use the time to try to wrap
> > up some important SQL related features and improvements.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Tue, 25 Jan 2022 at 16:38, Johannes Moser  wrote:
> >
> > > Dear Flink community,
> > >
> > > as mentioned in the summary mail earlier some contributors voiced that
> > > they would benefit from pushing the feature freeze for 1.15. by a week.
> > > This would mean Monday, 14th of February 2022, end of business CEST.
> > >
> > > Please let us know in case you got any concerns.
> > >
> > >
> > > Best,
> > > Till, Yun Gao & Joe
>

[jira] [Created] (FLINK-25512) Materialization Files are not cleaned up if no checkpoint is using it

[Yuan Mei (Jira)]

Yuan Mei created FLINK-25512:


 Summary: Materialization Files are not cleaned up if no checkpoint 
is using it
 Key: FLINK-25512
 URL: https://issues.apache.org/jira/browse/FLINK-25512
 Project: Flink
  Issue Type: Sub-task
Reporter: Yuan Mei






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25511) Leftovers after truncation are not be cleaned up if pre-uploading is enabled

[Yuan Mei (Jira)]

Yuan Mei created FLINK-25511:


 Summary: Leftovers after truncation are not be cleaned up if 
pre-uploading is enabled
 Key: FLINK-25511
 URL: https://issues.apache.org/jira/browse/FLINK-25511
 Project: Flink
  Issue Type: Sub-task
Reporter: Yuan Mei






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25470) Add/Expose/differentiate metrics of checkpoint size between changelog size vs materialization size

[Yuan Mei (Jira)]

Yuan Mei created FLINK-25470:


 Summary: Add/Expose/differentiate metrics of checkpoint size 
between changelog size vs materialization size
 Key: FLINK-25470
 URL: https://issues.apache.org/jira/browse/FLINK-25470
 Project: Flink
  Issue Type: Sub-task
Reporter: Yuan Mei






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Immediate dedicated Flink releases for log4j vulnerability

[Yuan Mei]

+1 for quick release.

On Mon, Dec 13, 2021 at 2:55 PM Martijn Visser 
wrote:

> +1 to address the issue like this
>
> On Mon, 13 Dec 2021 at 07:46, Jingsong Li  wrote:
>
> > +1 for fixing it in these versions and doing quick releases. Looks good
> to
> > me.
> >
> > Best,
> > Jingsong
> >
> > On Mon, Dec 13, 2021 at 2:18 PM Becket Qin  wrote:
> > >
> > > +1. The solution sounds good to me. There have been a lot of inquiries
> > > about how to react to this.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Mon, Dec 13, 2021 at 12:40 PM Prasanna kumar <
> > > prasannakumarram...@gmail.com> wrote:
> > >
> > > > 1+ for making Updates for 1.12.5 .
> > > > We are looking for fix in 1.12 version.
> > > > Please notify once the fix is done.
> > > >
> > > >
> > > > On Mon, Dec 13, 2021 at 9:45 AM Leonard Xu 
> wrote:
> > > >
> > > > > +1 for the quick release and the special vote period 24h.
> > > > >
> > > > > > 2021年12月13日 上午11:49，Dian Fu  写道：
> > > > > >
> > > > > > +1 for the proposal and creating a quick release.
> > > > > >
> > > > > > Regards,
> > > > > > Dian
> > > > > >
> > > > > >
> > > > > > On Mon, Dec 13, 2021 at 11:15 AM Kyle Bendickson <
> k...@tabular.io>
> > > > > wrote:
> > > > > >
> > > > > >> +1 to doing a release for this widely publicized vulnerability.
> > > > > >>
> > > > > >> In my experience, users will often update to the latest minor
> > patch
> > > > > version
> > > > > >> without much fuss. Plus, users have also likely heard about this
> > and
> > > > > will
> > > > > >> appreciate a simple fix (updating their version where possible).
> > > > > >>
> > > > > >> The work-around will need to still be noted for users who can’t
> > > > upgrade
> > > > > for
> > > > > >> whatever reason (EMR hasn’t caught up, etc).
> > > > > >>
> > > > > >> I also agree with your assessment to apply a patch on each of
> > those
> > > > > >> previous versions with only the log4j commit, so that they don’t
> > need
> > > > > to be
> > > > > >> as rigorously tested.
> > > > > >>
> > > > > >> Best,
> > > > > >> Kyle (GitHub @kbendick)
> > > > > >>
> > > > > >> On Sun, Dec 12, 2021 at 2:23 PM Stephan Ewen 
> > > > wrote:
> > > > > >>
> > > > > >>> Hi all!
> > > > > >>>
> > > > > >>> Without doubt, you heard about the log4j vulnerability [1].
> > > > > >>>
> > > > > >>> There is an advisory blog post on how to mitigate this in
> Apache
> > > > Flink
> > > > > >> [2],
> > > > > >>> which involves setting a config option and restarting the
> > processes.
> > > > > That
> > > > > >>> is fortunately a relatively simple fix.
> > > > > >>>
> > > > > >>> Despite this workaround, I think we should do an immediate
> > release
> > > > with
> > > > > >> the
> > > > > >>> updated dependency. Meaning not waiting for the next bug fix
> > releases
> > > > > >>> coming in a few weeks, but releasing asap.
> > > > > >>> The mood I perceive in the industry is pretty much panicky over
> > this,
> > > > > >> and I
> > > > > >>> expect we will see many requests for a patched release and many
> > > > > >> discussions
> > > > > >>> why the workaround alone would not be enough due to certain
> > > > guidelines.
> > > > > >>>
> > > > > >>> I suggest that we preempt those discussions and create releases
> > the
> > > > > >>> following way:
> > > > > >>>
> > > > > >>>  - we take the latest already released versions from each
> release
> > > > > >> branch:
> > > > > >>> ==> 1.14.0, 1.13.3, 1.12.5, 1.11.4
> > > > > >>>  - we add a single commit to those that just updates the log4j
> > > > > >> dependency
> > > > > >>>  - we release those as 1.14.1, 1.13.4, 1.12.6, 1.11.5, etc.
> > > > > >>>  - that way we don't need to do functional release tests,
> > because the
> > > > > >>> released code is identical to the previous release, except for
> > the
> > > > > log4j
> > > > > >>> dependency
> > > > > >>>  - we can then continue the work on the upcoming bugfix
> releases
> > as
> > > > > >>> planned, without high pressure
> > > > > >>>
> > > > > >>> I would suggest creating those RCs immediately and release them
> > with
> > > > a
> > > > > >>> special voting period (24h or so).
> > > > > >>>
> > > > > >>> WDYT?
> > > > > >>>
> > > > > >>> Best,
> > > > > >>> Stephan
> > > > > >>>
> > > > > >>> [1] https://nvd.nist.gov/vuln/detail/CVE-2021-44228
> > > > > >>> [2] https://flink.apache.org/2021/12/10/log4j-cve.html
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> >
> >
> >
> > --
> > Best, Jingsong Lee
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yingjie Cao

[Yuan Mei]

well deserved! Congratulations, Yingjie!

On Fri, Nov 19, 2021 at 12:39 PM Yu Li  wrote:

> Congrats and welcome, Yingjie!
>
> Best Regards,
> Yu
>
>
> On Thu, 18 Nov 2021 at 19:01, Yun Tang  wrote:
>
> > Congratulations, Yinjie!
> >
> > Best
> > Yun Tang
> >
> > On 2021/11/18 08:01:44 Martijn Visser wrote:
> > > Congratulations!
> > >
> > > On Thu, 18 Nov 2021 at 02:44, Leonard Xu  wrote:
> > >
> > > > Congratulations！Yingjie
> > > >
> > > > Best，
> > > > Leonard
> > > >
> > > > > 在 2021年11月18日，01:40，Till Rohrmann  写道：
> > > > >
> > > > > Congratulations Yingjie!
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Leonard Xu

[Yuan Mei]

Congrats Leonard!

Best
Yuan


On Fri, Nov 12, 2021 at 12:21 PM Paul Lam  wrote:

> Congrats! Well deserved!
>
> Best,
> Paul Lam
>
> > 2021年11月12日 12:16，Yangze Guo  写道：
> >
> > Congrats, Leonard!
> >
> > Jingsong Li  于 2021年11月12日周五 下午12:15写道：
> >
> >> Congrats & well deserved, Leonard!
> >>
> >> Thank you for your exciting contributions to Flink!
> >>
> >> Best,
> >> Jingsong
> >>
> >> On Fri, Nov 12, 2021 at 12:13 PM Jark Wu  wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> On behalf of the PMC, I'm very happy to announce Leonard Xu as a new
> >> Flink
> >>> committer.
> >>>
> >>> Leonard has been a very active contributor for more than two year,
> >> authored
> >>> 150+ PRs and reviewed many PRs which is quite outstanding.
> >>> Leonard mainly works on Flink SQL parts and drives several important
> >> FLIPs,
> >>> e.g. FLIP-132 (temporal table join) and FLIP-162 (correct time
> >> behaviors).
> >>> He is also the maintainer of flink-cdc-connectors[1] project which
> helps
> >> a
> >>> lot for users building a real-time data warehouse and data lake.
> >>>
> >>> Please join me in congratulating Leonard for becoming a Flink
> committer!
> >>>
> >>> Cheers,
> >>> Jark Wu
> >>>
> >>> [1]: https://github.com/ververica/flink-cdc-connectors
> >>
> >>
> >>
> >> --
> >> Best, Jingsong Lee
> >>
>
>

Re: [ANNOUNCE] New Apache Flink Committer - Yangze Guo

[Yuan Mei]

Congrats Yangze!

Best
Yuan

On Fri, Nov 12, 2021 at 11:57 AM Leonard Xu  wrote:

> Congrats & well deserved, Yangze!
>
> Best,
> Leonard
>
> > 在 2021年11月12日，11:51，godfrey he  写道：
> >
> > Congrats, Yangze!
> >
> > Best,
> > Godfrey
> >
> > Yuepeng Pan  于2021年11月12日周五 上午10:49写道：
> >>
> >> Congrats.
> >>
> >>
> >> Best,
> >> Yuepeng Pan.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> At 2021-11-12 10:10:43, "Xintong Song"  wrote:
> >>> Hi everyone,
> >>>
> >>> On behalf of the PMC, I'm very happy to announce Yangze Guo as a new
> Flink
> >>> committer.
> >>>
> >>> Yangze has been consistently contributing to this project for almost 3
> >>> years. His contributions are mainly in the resource management and
> >>> deployment areas, represented by the fine-grained resource management
> and
> >>> external resource framework. In addition to feature works, he's also
> active
> >>> in miscellaneous contributions, including PR reviews, document
> enhancement,
> >>> mailing list services and meetup/FF talks.
> >>>
> >>> Please join me in congratulating Yangze Guo for becoming a Flink
> committer!
> >>>
> >>> Thank you~
> >>>
> >>> Xintong Song
>
>

[jira] [Created] (FLINK-24436) FsStateChangelogWriter#lastAppendedSequenceNumber return different seq number with no writes

[Yuan Mei (Jira)]

Yuan Mei created FLINK-24436:


 Summary: FsStateChangelogWriter#lastAppendedSequenceNumber return 
different seq number with no writes
 Key: FLINK-24436
 URL: https://issues.apache.org/jira/browse/FLINK-24436
 Project: Flink
  Issue Type: Bug
Reporter: Yuan Mei






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24435) FsStateChangelogWriter#lastAppendedSequenceNumber return different seq number with no writes

[Yuan Mei (Jira)]

Yuan Mei created FLINK-24435:


 Summary: FsStateChangelogWriter#lastAppendedSequenceNumber return 
different seq number with no writes
 Key: FLINK-24435
 URL: https://issues.apache.org/jira/browse/FLINK-24435
 Project: Flink
  Issue Type: Bug
Reporter: Yuan Mei






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23765) PythonTableFunctionOperatorTestBase

[Yuan Mei (Jira)]

Yuan Mei created FLINK-23765:


 Summary: PythonTableFunctionOperatorTestBase
 Key: FLINK-23765
 URL: https://issues.apache.org/jira/browse/FLINK-23765
 Project: Flink
  Issue Type: Bug
Reporter: Yuan Mei


 
 
PythonTableFunctionOperatorTest>PythonTableFunctionOperatorTestBase.testFinishBundleTriggeredByTime:147
 » NullPointer 
PythonTableFunctionOperatorTest>PythonTableFunctionOperatorTestBase.testFinishBundleTriggeredOnCheckpoint:93
 » NullPointer 
 
PythonTableFunctionOperatorTest>PythonTableFunctionOperatorTestBase.testLeftJoin:179
 » NullPointer 
 
PythonTableFunctionOperatorTest>PythonTableFunctionOperatorTestBase.testRetractionFieldKept:68
 » NullPointer



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-183: Dynamic buffer size adjustment

[Yuan Mei]

+1 (binding)

Best
Yuan

On Wed, Jul 21, 2021 at 7:40 PM Piotr Nowojski  wrote:

> +1 (binding)
>
> Piotrek
>
> śr., 21 lip 2021 o 13:21 Anton Kalashnikov 
> napisał(a):
>
> > Hi everyone,
> >
> > I would like to start a vote on FLIP-183 [1] which was discussed in this
> > thread [2].
> > The vote will be open for at least 72 hours unless there is an objection
> > or not enough votes.
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-183%3A+Dynamic+buffer+size+adjustment
> > [2]
> >
> >
> https://lists.apache.org/thread.html/r0d06131b35fe641df787c16e8bcd3784161f901062c25778ed92871b%40%3Cdev.flink.apache.org%3E
> > --
> > Best regards,
> > Anton Kalashnikov
> >
>

[jira] [Created] (FLINK-23441) Remove CheckpointOptions Argument away from Snapshotable#snapshot

[Yuan Mei (Jira)]

Yuan Mei created FLINK-23441:


 Summary: Remove CheckpointOptions Argument away from 
Snapshotable#snapshot
 Key: FLINK-23441
 URL: https://issues.apache.org/jira/browse/FLINK-23441
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Yuan Mei






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23367) testKeyGroupedInternalPriorityQueue does not dispose rocksdb properly, and fails the test

[Yuan Mei (Jira)]

Yuan Mei created FLINK-23367:


 Summary: testKeyGroupedInternalPriorityQueue does not dispose 
rocksdb properly, and fails the test
 Key: FLINK-23367
 URL: https://issues.apache.org/jira/browse/FLINK-23367
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Yuan Mei


The set of `testKeyGroupedInternalPriorityQueue` for 
`ChangelogDelegateEmbeddedRocksDBStateBackendTest` does not dispose rocksdb 
properly. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

[Yuan Mei]

Thanks, everyone! ☺


Best Regards,
Yuan

On Fri, Jul 9, 2021 at 7:30 PM Benchao Li  wrote:

> Congratulations Yuan~
>
> Jun Qin  于2021年7月9日周五 下午4:27写道：
>
> > Congratulations, Yuan!
> >
> > > On Jul 8, 2021, at 11:49 AM, Jingsong Li 
> wrote:
> > >
> > > Congratulations Yuan!
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Thu, Jul 8, 2021 at 5:43 PM Arvid Heise  wrote:
> > >
> > >> Yay!
> > >>
> > >> On Thu, Jul 8, 2021 at 10:02 AM Jiayi Liao 
> > >> wrote:
> > >>
> > >>> Congratulations Yuan!
> > >>>
> > >>> Best,
> > >>> Jiayi Liao
> > >>>
> > >>> On Thu, Jul 8, 2021 at 3:55 PM Roman Khachatryan 
> > >> wrote:
> > >>>
> > >>>> Congratulations Yuan!
> > >>>>
> > >>>> Regards,
> > >>>> Roman
> > >>>>
> > >>>> On Thu, Jul 8, 2021 at 6:02 AM Yang Wang 
> > >> wrote:
> > >>>>>
> > >>>>> Congratulations Yuan!
> > >>>>>
> > >>>>> Best,
> > >>>>> Yang
> > >>>>>
> > >>>>> XING JIN  于2021年7月8日周四 上午11:46写道：
> > >>>>>
> > >>>>>> Congratulations Yuan~!
> > >>>>>>
> > >>>>>> Roc Marshal  于2021年7月8日周四 上午11:28写道：
> > >>>>>>
> > >>>>>>> Congratulations, Yuan!
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> At 2021-07-08 01:21:40, "Yu Li"  wrote:
> > >>>>>>>> Hi all,
> > >>>>>>>>
> > >>>>>>>> On behalf of the PMC, I’m very happy to announce Yuan Mei as a
> > >> new
> > >>>> Flink
> > >>>>>>>> committer.
> > >>>>>>>>
> > >>>>>>>> Yuan has been an active contributor for more than two years,
> > >> with
> > >>>> code
> > >>>>>>>> contributions on multiple components including kafka connectors,
> > >>>>>>>> checkpointing, state backends, etc. Besides, she has been
> > >> actively
> > >>>>>>> involved
> > >>>>>>>> in community activities such as helping manage releases,
> > >>> discussing
> > >>>>>>>> questions on dev@list, supporting users and giving talks at
> > >>>>>> conferences.
> > >>>>>>>>
> > >>>>>>>> Please join me in congratulating Yuan for becoming a Flink
> > >>>> committer!
> > >>>>>>>>
> > >>>>>>>> Cheers,
> > >>>>>>>> Yu
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>
> > >
> > >
> > > --
> > > Best, Jingsong Lee
> >
> >
>
> --
>
> Best,
> Benchao Li
>