Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Hangxiang Yu]

Congratulations, Rui !

On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee  wrote:

> Congratulations, Rui!
>
> Best,
> Lincoln Lee
>
>
> Lijie Wang  于2024年6月6日周四 09:11写道：
>
> > Congratulations, Rui!
> >
> > Best,
> > Lijie
> >
> > Rodrigo Meneses  于2024年6月5日周三 21:35写道：
> >
> > > All the best
> > >
> > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng 
> > wrote:
> > >
> > > > Congratulations, Rui!
> > > >
> > > > Regards,
> > > > Xiangyu Feng
> > > >
> > > > Feng Jin  于2024年6月5日周三 20:42写道：
> > > >
> > > > > Congratulations, Rui!
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng Jin
> > > > >
> > > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei 
> > wrote:
> > > > >
> > > > > > Congratulations, Rui!
> > > > > >
> > > > > > Best,
> > > > > > Yanfei
> > > > > >
> > > > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > > > >
> > > > > > > Congrats, Rui!
> > > > > > >
> > > > > > > Luke
> > > > > > >
> > > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun <
> jiabao...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Congrats, Rui. Well-deserved!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jiabao
> > > > > > > >
> > > > > > > > Zhanghao Chen  于2024年6月5日周三
> > 19:29写道：
> > > > > > > >
> > > > > > > > > Congrats, Rui!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Zhanghao Chen
> > > > > > > > > 
> > > > > > > > > From: Piotr Nowojski 
> > > > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > > > To: dev ; rui fan <
> > 1996fan...@gmail.com>
> > > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> > > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > On behalf of the PMC, I'm very happy to announce another
> new
> > > > Apache
> > > > > > Flink
> > > > > > > > > PMC Member - Fan Rui.
> > > > > > > > >
> > > > > > > > > Rui has been active in the community since August 2019.
> > During
> > > > this
> > > > > > time
> > > > > > > > he
> > > > > > > > > has contributed a lot of new features. Among others:
> > > > > > > > >   - Decoupling Autoscaler from Kubernetes Operator, and
> > > > supporting
> > > > > > > > > Standalone Autoscaler
> > > > > > > > >   - Improvements to checkpointing, flamegraphs, restart
> > > > strategies,
> > > > > > > > > watermark alignment, network shuffles
> > > > > > > > >   - Optimizing the memory and CPU usage of large operators,
> > > > greatly
> > > > > > > > > reducing the risk and probability of TaskManager OOM
> > > > > > > > >
> > > > > > > > > He reviewed a significant amount of PRs and has been active
> > > both
> > > > on
> > > > > > the
> > > > > > > > > mailing lists and in Jira helping to both maintain and grow
> > > > Apache
> > > > > > > > Flink's
> > > > > > > > > community. He is also our current Flink 1.20 release
> manager.
> > > > > > > > >
> > > > > > > > > In the last 12 months, Rui has been the most active
> > contributor
> > > > in
> > > > > > the
> > > > > > > > > Flink Kubernetes Operator project, while being the 2nd most
> > > > active
> > > > > > Flink
> > > > > > > > > contributor at the same time.
> > > > > > > > >
> > > > > > > > > Please join me in welcoming and congratulating Fan Rui!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
Best,
Hangxiang.

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

[Hangxiang Yu]

Congratulations, Weijie!

On Tue, Jun 4, 2024 at 11:40 PM Zhanghao Chen 
wrote:

> Congrats, Weijie!
>
> Best,
> Zhanghao Chen
> 
> From: Hang Ruan 
> Sent: Tuesday, June 4, 2024 16:37
> To: dev@flink.apache.org 
> Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo
>
> Congratulations Weijie!
>
> Best,
> Hang
>
> Yanfei Lei  于2024年6月4日周二 16:24写道：
>
> > Congratulations!
> >
> > Best,
> > Yanfei
> >
> > Leonard Xu  于2024年6月4日周二 16:20写道：
> > >
> > > Congratulations！
> > >
> > > Best,
> > > Leonard
> > >
> > > > 2024年6月4日 下午4:02，Yangze Guo  写道：
> > > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Yangze Guo
> > > >
> > > > On Tue, Jun 4, 2024 at 4:00 PM Weihua Hu 
> > wrote:
> > > >>
> > > >> Congratulations, Weijie!
> > > >>
> > > >> Best,
> > > >> Weihua
> > > >>
> > > >>
> > > >> On Tue, Jun 4, 2024 at 3:03 PM Yuxin Tan 
> > wrote:
> > > >>
> > > >>> Congratulations, Weijie!
> > > >>>
> > > >>> Best,
> > > >>> Yuxin
> > > >>>
> > > >>>
> > > >>> Yuepeng Pan  于2024年6月4日周二 14:57写道：
> > > >>>
> > >  Congratulations !
> > > 
> > > 
> > >  Best,
> > >  Yuepeng Pan
> > > 
> > >  At 2024-06-04 14:45:45, "Xintong Song" 
> > wrote:
> > > > Hi everyone,
> > > >
> > > > On behalf of the PMC, I'm very happy to announce that Weijie Guo
> > has
> > >  joined
> > > > the Flink PMC!
> > > >
> > > > Weijie has been an active member of the Apache Flink community
> for
> > many
> > > > years. He has made significant contributions in many components,
> > > >>> including
> > > > runtime, shuffle, sdk, connectors, etc. He has driven /
> > participated in
> > > > many FLIPs, authored and reviewed hundreds of PRs, been
> > consistently
> > >  active
> > > > on mailing lists, and also helped with release management of 1.20
> > and
> > > > several other bugfix releases.
> > > >
> > > > Congratulations and welcome Weijie!
> > > >
> > > > Best,
> > > >
> > > > Xintong (on behalf of the Flink PMC)
> > > 
> > > >>>
> > >
> >
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-35460) Check file size when position read for ForSt

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35460:


 Summary: Check file size when position read for ForSt
 Key: FLINK-35460
 URL: https://issues.apache.org/jira/browse/FLINK-35460
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35389) Implement List Async State API for ForStStateBackend

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35389:


 Summary: Implement List Async State API for ForStStateBackend
 Key: FLINK-35389
 URL: https://issues.apache.org/jira/browse/FLINK-35389
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends
Reporter: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-444: Native file copy support

[Hangxiang Yu]

>
> Note that for both recovery and checkpoints,  there are no retring
> mechanisms. If any part of downloading or
> uploading fails, the job fails over, so actually using such interface
> extension would be out of scope of this FLIP. In
> that case, maybe if this could be extended in the future without breaking
> compatibility we could leave it as a
> future improvement?
>

Thanks for the reply.
It makes sense to consider as a future optimization.

On Fri, May 10, 2024 at 4:31 PM Piotr Nowojski 
wrote:

> Hi!
>
> Thanks for your suggestions!
>
> > I'd prefer a unified one interface
>
> I have updated the FLIP to take that into account. In this case, I would
> also propose to completely drop `DuplicatingFileSystem` in favour of a
> basically renamed version of it `PathsCopyingFileSystem`.
> `DuplicatingFileSystem` was not marked as PublicEvolving/Experimental
> (probably by mistake), so technically we can do it. Even if not for that
> mistake, I would still vote to replace it to simplify the code, as any
> migration would be very easy. At the same time to the best of my knowledge,
> no one has ever implemented it.
>
> > The proposal mentions that s5cmd utilises 100% of CPU similar to Flink
> > 1.18. However, this will be a native process outside of the JVM. Are
> there
> > risk of large/long state download starving the TM of CPU cycle causing
> > issues such as heartbeat or ask timeout?
> >
> > Do you know if there is a way to limit the CPU utilisation of s5cmd? I
> see
> > worker and concurrency configuration but these do not map directly to cap
> > in CPU usage. The experience for feature user in this case will be one of
> > trial and error.
>
> Those are good points. As a matter of fact, shortly after publishing this
> FLIP, we started experimenting with using `cpulimit` to achieve just that.
> If everything will work out fine, we are planning to expose this as a
> configuration option for the S3 file system. I've added that to the FLIP.
>
> > 2. copyFiles is not an atomic operation, How could we handle the
> situation
> > when some partial files fail ?
> > Could we return the list of successful files then the caller could decide
> > to retry or just know them ?
>
> Do you have some suggestions on how that should be implemented in the
> interface and how should it be used?
>
> Note that for both recovery and checkpoints,  there are no retring
> mechanisms. If any part of downloading or
> uploading fails, the job fails over, so actually using such interface
> extension would be out of scope of this FLIP. In
> that case, maybe if this could be extended in the future without breaking
> compatibility we could leave it as a
> future improvement?
>
> Best,
> Piotrek
>
>
> pt., 10 maj 2024 o 07:40 Hangxiang Yu  napisał(a):
>
> > Hi Piotr.
> > Thanks for your proposal.
> >
> > I have some comments, PTAL:
> > 1. +1 about unifying the interface with DuplicatingFileSystem.
> > IIUC, DuplicatingFileSystem also covers the logic from/to both local and
> > remote paths.
> > The implementations could define their own logic about how to fast
> > copy/duplicate files, e.g. hard link or transfer manager.
> >
> > 2. copyFiles is not an atomic operation, How could we handle the
> situation
> > when some partial files fail ?
> > Could we return the list of successful files then the caller could decide
> > to retry or just know them ?
> >
> > On Thu, May 9, 2024 at 3:46 PM Keith Lee 
> > wrote:
> >
> > > Hi Piotr,
> > >
> > > Thank you for the proposal. Looks great.
> > >
> > > Along similar line of Aleks' question on memory usage.
> > >
> > > The proposal mentions that s5cmd utilises 100% of CPU similar to Flink
> > > 1.18. However, this will be a native process outside of the JVM. Are
> > there
> > > risk of large/long state download starving the TM of CPU cycle causing
> > > issues such as heartbeat or ask timeout?
> > >
> > > Do you know if there is a way to limit the CPU utilisation of s5cmd? I
> > see
> > > worker and concurrency configuration but these do not map directly to
> cap
> > > in CPU usage. The experience for feature user in this case will be one
> of
> > > trial and error.
> > >
> > > Thanks
> > > Keith
> > >
> > > On Wed, May 8, 2024 at 12:47 PM Ahmed Hamdy 
> > wrote:
> > >
> > > > Hi Piotr
> > > > +1 for the proposal, it seems to have a lot of gains.
> > > >
> > > > Best Regards
> &g

Re: [DISCUSS] FLIP-444: Native file copy support

[Hangxiang Yu]

Hi Piotr.
Thanks for your proposal.

I have some comments, PTAL:
1. +1 about unifying the interface with DuplicatingFileSystem.
IIUC, DuplicatingFileSystem also covers the logic from/to both local and
remote paths.
The implementations could define their own logic about how to fast
copy/duplicate files, e.g. hard link or transfer manager.

2. copyFiles is not an atomic operation, How could we handle the situation
when some partial files fail ?
Could we return the list of successful files then the caller could decide
to retry or just know them ?

On Thu, May 9, 2024 at 3:46 PM Keith Lee 
wrote:

> Hi Piotr,
>
> Thank you for the proposal. Looks great.
>
> Along similar line of Aleks' question on memory usage.
>
> The proposal mentions that s5cmd utilises 100% of CPU similar to Flink
> 1.18. However, this will be a native process outside of the JVM. Are there
> risk of large/long state download starving the TM of CPU cycle causing
> issues such as heartbeat or ask timeout?
>
> Do you know if there is a way to limit the CPU utilisation of s5cmd? I see
> worker and concurrency configuration but these do not map directly to cap
> in CPU usage. The experience for feature user in this case will be one of
> trial and error.
>
> Thanks
> Keith
>
> On Wed, May 8, 2024 at 12:47 PM Ahmed Hamdy  wrote:
>
> > Hi Piotr
> > +1 for the proposal, it seems to have a lot of gains.
> >
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Mon, 6 May 2024 at 12:06, Zakelly Lan  wrote:
> >
> > > Hi Piotrek,
> > >
> > > Thanks for your answers!
> > >
> > > Good question. The intention and use case behind
> `DuplicatingFileSystem`
> > is
> > > > different. It marks if `FileSystem` can quickly copy/duplicate files
> > > > in the remote `FileSystem`. For example an equivalent of a hard link
> or
> > > > bumping a reference count in the remote system. That's a bit
> different
> > > > to copy paths between remote and local file systems.
> > > >
> > > > However, it could arguably be unified under one interface where we
> > would
> > > > re-use or re-name `canFastDuplicate(Path, Path)` to
> > > > `canFastCopy(Path, Path)` with the following use cases:
> > > > - `canFastCopy(remoteA, remoteB)` returns true - current equivalent
> of
> > > > `DuplicatingFileSystem` - quickly duplicate/hard link remote path
> > > > - `canFastCopy(local, remote)` returns true - FS can natively upload
> > > local
> > > > file to a remote location
> > > > - `canFastCopy(remote, local)` returns true - FS can natively
> download
> > > > local file from a remote location
> > > >
> > > > Maybe indeed that's a better solution vs having two separate
> interfaces
> > > for
> > > > copying and duplicating?
> > > >
> > >
> > > I'd prefer a unified one interface, `canFastCopy(Path, Path)` looks
> good
> > to
> > > me. This also resolves my question 1 about the destination.
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Mon, May 6, 2024 at 6:36 PM Piotr Nowojski 
> > > wrote:
> > >
> > > > Hi All!
> > > >
> > > > Thanks for your comments.
> > > >
> > > > Muhammet and Hong, about the config options.
> > > >
> > > > > Could you please also add the configuration property for this? An
> > > example
> > > > showing how users would set this parameter would be helpful.
> > > >
> > > > > 1/ Configure the implementation of PathsCopyingFileSystem used
> > > > > 2/ Configure the location of the s5cmd binary (version control
> etc.)
> > > >
> > > > Ops, sorry I added the config options that I had in mind to the
> FLIP. I
> > > > don't know why I have omitted this. Basically I suggest that in order
> > to
> > > > use native file copying:
> > > > 1. `FileSystem` must support it via implementing
> > `PathsCopyingFileSystem`
> > > > interface
> > > > 2. That `FileSystem` would have to be configured to actually use it.
> > For
> > > > example S3 file system would return `true` that it can copy paths
> > > > only if `s3.s5cmd.path` has been specified.
> > > >
> > > > > Would this affect any filesystem connectors that use
> FileSystem[1][2]
> > > > dependencies?
> > > >
> > > > Definitely not out of the box. Any place in Flink that is currently
> > > > uploading/downloading files from a FileSystem could use this feature,
> > but
> > > > it
> > > > would have to be implemented. The same way this FLIP will implement
> > > native
> > > > files copying when downloading state during recovery,
> > > > but the old code path will be still used for uploading state files
> > > during a
> > > > checkpoint.
> > > >
> > > > > How adding a s5cmd will affect memory footprint? Since this is a
> > native
> > > > binary, memory consumption will not be controlled by JVM or Flink.
> > > >
> > > > As you mentioned the memory usage of `s5cmd` will not be controlled,
> so
> > > the
> > > > memory footprint will grow. S5cmd integration with Flink
> > > > has been tested quite extensively on our production environment
> > already,
> > > > and we haven't observed any issues so far despite the fact we
> > > > 

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

[Hangxiang Yu]

+1（binding）

On Mon, May 6, 2024 at 12:25 PM Yuan Mei  wrote:

> +1（binding）
>
> Best
> Yuan
>
> On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> > +1 (binding)
> >
> > Best,
> > Rui
> >
> > On Mon, May 6, 2024 at 11:01 AM Yanfei Lei  wrote:
> >
> > > +1 (binding)
> > >
> > > Best,
> > > Yanfei
> > >
> > > Zakelly Lan  于2024年5月6日周一 11:00写道：
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks for driving this!
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, May 6, 2024 at 10:54 AM yue ma  wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Thanks for all the feedback, I'd like to start a vote on the
> > FLIP-447:
> > > > > Upgrade FRocksDB from 6.20.3 to 8.10.0 [1]. The discussion thread
> is
> > > here
> > > > > [2].
> > > > >
> > > > > The vote will be open for at least 72 hours unless there is an
> > > objection or
> > > > > insufficient votes.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0
> > > > > [2]
> https://lists.apache.org/thread/lrxjfpjjwlq4sjzm1oolx58n1n8r48hw
> > > > >
> > > > > --
> > > > > Best,
> > > > > Yue
> > > > >
> > >
> >
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-35268) Support TTL for Async State API

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35268:


 Summary: Support TTL for Async State API
 Key: FLINK-35268
 URL: https://issues.apache.org/jira/browse/FLINK-35268
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends
Reporter: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35262) Bridge between AsyncKeyedStateBackend and AsyncExecutionController

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35262:


 Summary: Bridge between AsyncKeyedStateBackend and 
AsyncExecutionController
 Key: FLINK-35262
 URL: https://issues.apache.org/jira/browse/FLINK-35262
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends, Runtime / Task
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

[Hangxiang Yu]

Hi, Yue.
Very glad to see that IngestDB will be used to improve the rescaling
performance of RocksDB.
And +1 for the upgrade. Thanks for the great work!

On Thu, Apr 25, 2024 at 5:16 AM Martijn Visser 
wrote:

> +1
>
> On Wed, Apr 24, 2024 at 5:31 PM Congxian Qiu 
> wrote:
>
> > Thanks for driving this,  yue
> >
> > We also observed significant performance improvements in some cases after
> > bumped the Rocksdb version, +1 for this work
> >
> > Best,
> > Congxian
> >
> >
> > yue ma  于2024年4月24日周三 19:16写道：
> >
> > > hi Yanfei,
> > >
> > > Thanks for your feedback and reminders I have updated related
> > information.
> > > In fact, most of them use the default Configrations.
> > >
> > > Yanfei Lei  于2024年4月23日周二 12:51写道：
> > >
> > > > Hi Yue & Roman,
> > > >
> > > > Thanks for initiating this FLIP and all the efforts for the upgrade.
> > > >
> > > > 8.10.0 introduces some new features, making it possible for Flink to
> > > > implement some new exciting features, and the upgrade also makes
> > > > FRocksDB easier to maintain, +1 for upgrading.
> > > >
> > > > I read the FLIP and have a minor comment, it would be better to add
> > > > some description about the environment/configuration of the nexmark's
> > > > result.
> > > >
> > > > Roman Khachatryan  于2024年4月23日周二 12:07写道：
> > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > Thanks for writing the proposal and preparing the upgrade.
> > > > >
> > > > > FRocksDB  definitely needs to be kept in sync with the upstream and
> > the
> > > > new
> > > > > APIs are necessary for faster rescaling.
> > > > > We're already using a similar version internally.
> > > > >
> > > > > I reviewed the FLIP and it looks good to me (disclaimer: I took
> part
> > in
> > > > > some steps of this effort).
> > > > >
> > > > >
> > > > > Regards,
> > > > > Roman
> > > > >
> > > > > On Mon, Apr 22, 2024, 08:11 yue ma  wrote:
> > > > >
> > > > > > Hi Flink devs,
> > > > > >
> > > > > > I would like to start a discussion on FLIP-447: Upgrade FRocksDB
> > from
> > > > > > 6.20.3 to 8.10.0
> > > > > >
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0
> > > > > >
> > > > > > This FLIP proposes upgrading the version of FRocksDB in the Flink
> > > > Project
> > > > > > from 6.20.3 to 8.10.0.
> > > > > > The FLIP mainly introduces the main benefits of upgrading
> FRocksDB,
> > > > > > including the use of IngestDB which can improve Rescaling
> > performance
> > > > by
> > > > > > more than 10 times in certain scenarios, as well as other
> potential
> > > > > > optimization points such as async_io, blob db, and tiered
> > storage.The
> > > > > > FLIP also presented test results based on RocksDB 8.10, including
> > > > > > StateBenchmark and Nexmark tests.
> > > > > > Overall, upgrading FRocksDB may result in a small regression of
> > write
> > > > > > performance( which is a very small part of the overall overhead),
> > but
> > > > it
> > > > > > can bring many important performance benefits.
> > > > > > So we hope to upgrade the version of FRocksDB through this FLIP.
> > > > > >
> > > > > > Looking forward to everyone's feedback and suggestions. Thank
> you!
> > > > > > --
> > > > > > Best regards,
> > > > > > Yue
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Yanfei
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Yue
> > >
> >
>


-- 
Best,
Hangxiang.

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

[Hangxiang Yu]

Congratulations, Jing!

On Mon, Apr 15, 2024 at 11:11 AM spoon_lz  wrote:

>
>
> Congratulations!
>
> Best,
> lz
>
> | |
> spoon_lz
> |
> |
> spoon...@126.com
> |
>
>
>  Replied Message 
> | From | Biao Geng |
> | Date | 04/15/2024 10:40 |
> | To |  |
> | Subject | Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge |
> Congrats, Jing!
>
> Best,
> Biao Geng
>
> Zakelly Lan  于2024年4月15日周一 10:17写道：
>
> Congratulations, Jing!
>
>
> Best,
> Zakelly
>
> On Sat, Apr 13, 2024 at 12:47 AM Ferenc Csaky 
> wrote:
>
> Congratulations, Jing!
>
> Best,
> Ferenc
>
>
>
> On Friday, April 12th, 2024 at 13:54, Ron liu 
> wrote:
>
>
>
> Congratulations, Jing!
>
> Best,
> Ron
>
> Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
>
> Congratulations, Jing!
>
> Best,
> Junrui
>
> Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:28写道：
>
> Congratulations, Jing!
>
> Best Regards,
> Aleksandr
>
>
>

-- 
Best,
Hangxiang.

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

[Hangxiang Yu]

Congratulations, Lincoln!

On Mon, Apr 15, 2024 at 10:17 AM Zakelly Lan  wrote:

> Congratulations, Lincoln!
>
>
> Best,
> Zakelly
>
> On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky 
> wrote:
>
> > Congratulations, Lincoln!
> >
> > Best,
> > Ferenc
> >
> >
> >
> >
> > On Friday, April 12th, 2024 at 15:54, lorenzo.affe...@ververica.com
> .INVALID
> >  wrote:
> >
> > >
> > >
> > > Huge congrats! Well done!
> > > On Apr 12, 2024 at 13:56 +0200, Ron liu ron9@gmail.com, wrote:
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > Best,
> > > > Ron
> > > >
> > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > Best,
> > > > > Junrui
> > > > >
> > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:29写道：
> > > > >
> > > > > > > Congratulations, Lincoln!
> > > > > > >
> > > > > > > Best Regards
> > > > > > > Aleksandr
> >
>


-- 
Best,
Hangxiang.

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

[Hangxiang Yu]

Congratulations, Zakelly!

On Mon, Apr 15, 2024 at 1:58 PM Yun Tang  wrote:

> Congratulations, Zakelly!
>
> Best
> Yun Tang
> 
> From: Yanquan Lv 
> Sent: Monday, April 15, 2024 13:23
> To: dev@flink.apache.org 
> Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan
>
> Congratulations, Zakelly!
>
> Best,
> YanQuan
>
> Yuan Mei  于2024年4月15日周一 10:51写道：
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm happy to let you know that Zakelly Lan has
> become
> > a new Flink Committer!
> >
> > Zakelly has been continuously contributing to the Flink project since
> 2020,
> > with a focus area on Checkpointing, State as well as frocksdb (the
> default
> > on-disk state db).
> >
> > He leads several FLIPs to improve checkpoints and state APIs, including
> > File Merging for Checkpoints and configuration/API reorganizations. He is
> > also one of the main contributors to the recent efforts of "disaggregated
> > state management for Flink 2.0" and drives the entire discussion in the
> > mailing thread, demonstrating outstanding technical depth and breadth of
> > knowledge.
> >
> > Beyond his technical contributions, Zakelly is passionate about helping
> the
> > community in numerous ways. He spent quite some time setting up the Flink
> > Speed Center and rebuilding the benchmark pipeline after the original one
> > was out of lease. He helps build frocksdb and tests for the upcoming
> > frocksdb release (bump rocksdb from 6.20.3->8.10).
> >
> > Please join me in congratulating Zakelly for becoming an Apache Flink
> > committer!
> >
> > Best,
> > Yuan (on behalf of the Flink PMC)
> >
>


-- 
Best,
Hangxiang.

Re: [VOTE] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

[Hangxiang Yu]

+1  (binding)

On Fri, Apr 12, 2024 at 10:22 AM Jinzhong Li 
wrote:

> +1  (non binding)
>
> Bests,
> Jinzhong
>
> On Thu, Apr 11, 2024 at 7:26 AM Muhammet Orazov
>  wrote:
>
> > Hey Rui,
> >
> > +1 (non-binding).
> >
> > Thanks for driving it!
> >
> > Best,
> > Muhammet
> >
> > On 2024-04-10 04:36, Rui Fan wrote:
> > > Hi devs,
> > >
> > > Thank you to everyone for the feedback on FLIP-441: Show
> > > the JobType and remove Execution Mode on Flink WebUI[1]
> > > which has been discussed in this thread [2].
> > >
> > > I would like to start a vote for it. The vote will be open for at least
> > > 72
> > > hours unless there is an objection or not enough votes.
> > >
> > > [1] https://cwiki.apache.org/confluence/x/agrPEQ
> > > [2] https://lists.apache.org/thread/0s52w17w24x7m2zo6ogl18t1fy412vcd
> > >
> > > Best,
> > > Rui
> >
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-35049) Implement Async State API for ForStStateBackend

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35049:


 Summary: Implement Async State API for ForStStateBackend
 Key: FLINK-35049
 URL: https://issues.apache.org/jira/browse/FLINK-35049
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends
Reporter: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35048) Implement all methods of AsyncKeyedStateBakend

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35048:


 Summary: Implement all methods of AsyncKeyedStateBakend 
 Key: FLINK-35048
 URL: https://issues.apache.org/jira/browse/FLINK-35048
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends
Reporter: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35047) Introduce ForStStateBackend

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35047:


 Summary: Introduce ForStStateBackend
 Key: FLINK-35047
 URL: https://issues.apache.org/jira/browse/FLINK-35047
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35046) Introduce New KeyedStateBackend related Async interfaces

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35046:


 Summary: Introduce New KeyedStateBackend related Async interfaces
 Key: FLINK-35046
 URL: https://issues.apache.org/jira/browse/FLINK-35046
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends
Reporter: Hangxiang Yu


Since we have introduced new State API, the async version of some classes 
should be introduced to support it, e.g. AsyncKeyedStateBackend, new State 
Descriptor.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35045) Introduce ForStFileSystem to support reading and writing with ByteBuffer

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35045:


 Summary: Introduce ForStFileSystem to support reading and writing 
with ByteBuffer
 Key: FLINK-35045
 URL: https://issues.apache.org/jira/browse/FLINK-35045
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35044) Introduce statebackend-forst module

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35044:


 Summary: Introduce statebackend-forst module
 Key: FLINK-35044
 URL: https://issues.apache.org/jira/browse/FLINK-35044
 Project: Flink
  Issue Type: Sub-task
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35043) Release beta version of ForSt

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-35043:


 Summary: Release beta version of ForSt
 Key: FLINK-35043
 URL: https://issues.apache.org/jira/browse/FLINK-35043
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34987) Introduce Internal State Interface for Async API

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34987:


 Summary: Introduce Internal State Interface for Async API
 Key: FLINK-34987
 URL: https://issues.apache.org/jira/browse/FLINK-34987
 Project: Flink
  Issue Type: Sub-task
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34975) FLIP-427: ForSt - Disaggregated state Store

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34975:


 Summary: FLIP-427: ForSt - Disaggregated state Store
 Key: FLINK-34975
 URL: https://issues.apache.org/jira/browse/FLINK-34975
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / State Backends
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


This is a sub-FLIP for the disaggregated state management and its related work, 
please read the [FLIP-423|https://cwiki.apache.org/confluence/x/R4p3EQ] first 
to know the whole story.

As described in FLIP-423, there are some tough issues about embedded state 
backend on local file system, respecially when dealing with extremely large 
state:
 # {*}Constraints of local disk space complicate the prediction of storage 
requirements, potentially leading to job failures{*}: Especially in cloud 
native deployment mode, pre-allocated local disks typically face strict 
capacity constraints, making it challenging to forecast the size requirements 
of job states. Over-provisioning disk space results in unnecessary resource 
overhead, while under-provisioning risks job failure due to insufficient space.
 # *The tight coupling of compute and storage resources leads to 
underutilization and increased waste:* Jobs can generally be categorized as 
either CPU-intensive or IO-intensive. In a coupled architecture, CPU-intensive 
jobs leave a significant portion of storage resources underutilized, whereas 
IO-intensive jobs result in idle computing resources.

By considering remote storage as the primary storage, all working states are 
maintained on the remote file system, which brings several advantages:
 # *Remote storages e.g. S3/HDFS typically offer elastic scalability, 
theoretically providing unlimited space.*
 # *The allocation of remote storage resources can be optimized by reducing 
them for CPU-intensive jobs and augmenting them for IO-intensive jobs, thus 
enhancing overall resource utilization.*
 # *This architecture facilitates a highly efficient and lightweight process 
for checkpointing, recovery, and rescaling through fast copy or simple move.*

This FLIP aims to realize disaggregated state for our new key-value store named 
*ForSt* which evloves from RocksDB and supports remote file system. This makes 
Flink get rid of the disadvantages by coupled state architecture and embrace 
the scalable as well as flexible cloud-native storage.

Please see [FLIP-427 |https://cwiki.apache.org/confluence/x/T4p3EQ]for more 
details.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-426: Grouping Remote State Access

[Hangxiang Yu]

+1 (binding)

On Sun, Mar 31, 2024 at 9:02 PM Yanfei Lei  wrote:

> +1 (binding)
>
> Best,
> Yanfei
>
> yue ma  于2024年3月29日周五 16:10写道：
> >
> > +1 (non-binding)
> >
> > Jinzhong Li  于2024年3月27日周三 18:57写道：
> >
> > > Hi devs,
> > >
> > > I'd like to start a vote on the FLIP-426: Grouping Remote State Access
> [1].
> > > The discussion thread is here [2].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection or
> > > insufficient votes.
> > >
> > >
> > > [1] https://cwiki.apache.org/confluence/x/TYp3EQ
> > >
> > > [2] https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf
> > >
> > >
> > > Best,
> > >
> > > Jinzhong
> > >
> >
> >
> > --
> > Best,
> > Yue
>


-- 
Best,
Hangxiang.

[RESULT][VOTE] FLIP-427: Disaggregated state Store

[Hangxiang Yu]

I'm happy to announce that FLIP-427: Disaggregated state Store[1]
has been accepted with 7 approving votes (4 binding) [2]:

- Yuan Mei (binding)
- Feifan Wang (non-binding)
- Piotr Nowojski (binding)
- Rui Fan (binding)
- Yun Tang (binding)
- Yuepeng Pan (non-binding)
- yue ma (non-binding)

There are no disapproving votes. Thanks to everyone who participated
in the discussion and voting.

[1] https://cwiki.apache.org/confluence/x/T4p3EQ
[2] https://lists.apache.org/thread/lqpovwx3kpszgfcgl0olfhxotfo9d6nz


Best,
Hangxiang

Re: [VOTE] FLIP-427: Disaggregated state Store

[Hangxiang Yu]

Thanks all for the votes!
I'm closing the vote and the result will be posted in a separate mail.

On Fri, Mar 29, 2024 at 4:29 PM yue ma  wrote:

> +1(non-binding)
>
> Hangxiang Yu  于2024年3月27日周三 18:37写道：
>
> > Hi devs,
> >
> > Thanks all for your valuable feedback about FLIP-427: Disaggregated state
> > Store [1].
> > I'd like to start a vote on it.  The discussion thread is here [2].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> or
> > insufficient votes.
> >
> > [1] https://cwiki.apache.org/confluence/x/T4p3EQ
> > [2] https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft
> >
> >
> > Best,
> > Hangxiang
> >
>
>
> --
> Best,
> Yue
>


-- 
Best,
Hangxiang.

Re: [VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State

[Hangxiang Yu]

+1 (binding)

On Mon, Apr 1, 2024 at 9:24 AM gongzhongqiang 
wrote:

> +1(non-binding)
>
> Best,
>
> Zhongqiang Gong
>
> Jinzhong Li  于2024年3月27日周三 19:31写道：
>
> > Hi devs,
> >
> >
> > I'd like to start a vote on the FLIP-428: Fault Tolerance/Rescale
> > Integration for Disaggregated State [1]. The discussion thread is here
> [2].
> >
> >
> > The vote will be open for at least 72 hours unless there is an objection
> or
> > insufficient votes.
> >
> > [1] https://cwiki.apache.org/confluence/x/UYp3EQ
> >
> > [2] https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b
> >
> > Best,
> >
> > Jinzhong
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-427: Disaggregated State Store

[Hangxiang Yu]

Hi Yun.
Thanks for the great suggestion.
I just added related information into the FLIP.

On Sat, Mar 30, 2024 at 10:49 AM Yun Tang  wrote:

> Hi Feifan,
>
> I just replied in the discussion of FLIP-428. I agree that we could leave
> the clean-up optimization in the future FLIP, however, I think we should
> mention this topic explicitly in the current FLIP to make the overall
> design complete and more sophisticated.
>
> Best
> Yun Tang
> 
> From: Feifan Wang 
> Sent: Thursday, March 28, 2024 12:35
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS] FLIP-427: Disaggregated State Store
>
> Thanks for your reply, Hangxiang. I totally agree with you about the jni
> part.
>
> Hi Yun Tang, I just noticed that FLIP-427 mentions “The life cycle of
> working dir is managed as before local strategy.” IIUC, the working dir
> will be deleted after TaskManager exit. And I think that's enough for
> current stage, WDYT ?
>
> ——
>
> Best regards,
>
> Feifan Wang
>
>
>
>
> At 2024-03-28 12:18:56, "Hangxiang Yu"  wrote:
> >Hi, Feifan.
> >
> >Thanks for your reply.
> >
> >What if we only use jni to access DFS that needs to reuse Flink
> FileSystem?
> >> And all local disk access through native api. This idea is based on the
> >> understanding that jni overhead is not worth mentioning compared to DFS
> >> access latency. It might make more sense to consider avoiding jni
> overhead
> >> for faster local disks. Since local disk as secondary is already under
> >> consideration [1], maybe we can discuss in that FLIP whether to use
> native
> >> api to access local disk?
> >>
> >This is a good suggestion. It's reasonable to use native api to access
> >local disk cache since it requires lower latency compared to remote
> access.
> >I also believe that the jni overhead is relatively negligible when weighed
> >against the latency of remote I/O as mentioned in the FLIP.
> >So I think we could just go on proposal 2 and keep proposal 1 as a
> >potential future optimization, which could work better when there is a
> >higher performance requirement or some native libraries of filesystems
> have
> >significantly higher performance and resource usage compared to their java
> >libs.
> >
> >
> >On Thu, Mar 28, 2024 at 11:39 AM Feifan Wang  wrote:
> >
> >> Thanks for this valuable proposal Hangxiang !
> >>
> >>
> >> > If we need to introduce a JNI call during each filesystem call, that
> >> would be N times JNI cost compared with the current RocksDB
> state-backend's
> >> JNI cost.
> >> What if we only use jni to access DFS that needs to reuse Flink
> >> FileSystem? And all local disk access through native api. This idea is
> >> based on the understanding that jni overhead is not worth mentioning
> >> compared to DFS access latency. It might make more sense to consider
> >> avoiding jni overhead for faster local disks. Since local disk as
> secondary
> >> is already under consideration [1], maybe we can discuss in that FLIP
> >> whether to use native api to access local disk?
> >>
> >>
> >> >I'd suggest keeping `state.backend.forSt.working-dir` as it is for now.
> >> >Different disaggregated state storages may have their own semantics
> about
> >> >this configuration, e.g. life cycle, supported file systems or
> storages.
> >> I agree with considering moving this configuration up to the engine
> level
> >> until there are other disaggreated backends.
> >>
> >>
> >> [1] https://cwiki.apache.org/confluence/x/U4p3EQ
> >>
> >> ——
> >>
> >> Best regards,
> >>
> >> Feifan Wang
> >>
> >>
> >>
> >>
> >> At 2024-03-28 09:55:48, "Hangxiang Yu"  wrote:
> >> >Hi, Yun.
> >> >Thanks for the reply.
> >> >
> >> >The JNI cost you considered is right. As replied to Yue, I agreed to
> leave
> >> >space and consider proposal 1 as an optimization in the future, which
> is
> >> >also updated in the FLIP.
> >> >
> >> >The other question is that the configuration of
> >> >> `state.backend.forSt.working-dir` looks too coupled with the ForSt
> >> >> state-backend, how would it be if we introduce another disaggregated
> >> state
> >> >> storage? Thus, I think `state.backend.disaggregated.working-dir`
> might
> >> be a
>

Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

[Hangxiang Yu]

Congratulations!

On Fri, Mar 29, 2024 at 10:27 AM Benchao Li  wrote:

> Congratulations!
>
> Zakelly Lan  于2024年3月29日周五 10:25写道：
> >
> > Congratulations!
> >
> >
> > Best,
> > Zakelly
> >
> > On Thu, Mar 28, 2024 at 10:13 PM Jing Ge 
> wrote:
> >
> > > Congrats!
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Thu, Mar 28, 2024 at 1:27 PM Feifan Wang 
> wrote:
> > >
> > > > Congratulations!——
> > > >
> > > > Best regards,
> > > >
> > > > Feifan Wang
> > > >
> > > >
> > > >
> > > >
> > > > At 2024-03-28 20:02:43, "Yanfei Lei"  wrote:
> > > > >Congratulations!
> > > > >
> > > > >Best,
> > > > >Yanfei
> > > > >
> > > > >Zhanghao Chen  于2024年3月28日周四 19:59写道：
> > > > >>
> > > > >> Congratulations!
> > > > >>
> > > > >> Best,
> > > > >> Zhanghao Chen
> > > > >> 
> > > > >> From: Yu Li 
> > > > >> Sent: Thursday, March 28, 2024 15:55
> > > > >> To: d...@paimon.apache.org 
> > > > >> Cc: dev ; user 
> > > > >> Subject: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level
> > > Project
> > > > >>
> > > > >> CC the Flink user and dev mailing list.
> > > > >>
> > > > >> Paimon originated within the Flink community, initially known as
> Flink
> > > > >> Table Store, and all our incubating mentors are members of the
> Flink
> > > > >> Project Management Committee. I am confident that the bonds of
> > > > >> enduring friendship and close collaboration will continue to
> unite the
> > > > >> two communities.
> > > > >>
> > > > >> And congratulations all!
> > > > >>
> > > > >> Best Regards,
> > > > >> Yu
> > > > >>
> > > > >> On Wed, 27 Mar 2024 at 20:35, Guojun Li 
> > > > wrote:
> > > > >> >
> > > > >> > Congratulations!
> > > > >> >
> > > > >> > Best,
> > > > >> > Guojun
> > > > >> >
> > > > >> > On Wed, Mar 27, 2024 at 5:24 PM wulin 
> wrote:
> > > > >> >
> > > > >> > > Congratulations~
> > > > >> > >
> > > > >> > > > 2024年3月27日 15:54，王刚  写道：
> > > > >> > > >
> > > > >> > > > Congratulations~
> > > > >> > > >
> > > > >> > > >> 2024年3月26日 10:25，Jingsong Li  写道：
> > > > >> > > >>
> > > > >> > > >> Hi Paimon community,
> > > > >> > > >>
> > > > >> > > >> I’m glad to announce that the ASF board has approved a
> > > > resolution to
> > > > >> > > >> graduate Paimon into a full Top Level Project. Thanks to
> > > > everyone for
> > > > >> > > >> your help to get to this point.
> > > > >> > > >>
> > > > >> > > >> I just created an issue to track the things we need to
> modify
> > > > [2],
> > > > >> > > >> please comment on it if you feel that something is
> missing. You
> > > > can
> > > > >> > > >> refer to apache documentation [1] too.
> > > > >> > > >>
> > > > >> > > >> And, we already completed the GitHub repo migration [3],
> please
> > > > update
> > > > >> > > >> your local git repo to track the new repo [4].
> > > > >> > > >>
> > > > >> > > >> You can run the following command to complete the remote
> repo
> > > > tracking
> > > > >> > > >> migration.
> > > > >> > > >>
> > > > >> > > >> git remote set-url origin
> https://github.com/apache/paimon.git
> > > > >> > > >>
> > > > >> > > >> If you have a different name, please change the 'origin' to
> > > your
> > > > remote
> > > > >> > > name.
> > > > >> > > >>
> > > > >> > > >> Please join me in celebrating!
> > > > >> > > >>
> > > > >> > > >> [1]
> > > > >> > >
> > > >
> > >
> https://incubator.apache.org/guides/transferring.html#life_after_graduation
> > > > >> > > >> [2] https://github.com/apache/paimon/issues/3091
> > > > >> > > >> [3] https://issues.apache.org/jira/browse/INFRA-25630
> > > > >> > > >> [4] https://github.com/apache/paimon
> > > > >> > > >>
> > > > >> > > >> Best,
> > > > >> > > >> Jingsong Lee
> > > > >> > >
> > > > >> > >
> > > >
> > >
>
>
>
> --
>
> Best,
> Benchao Li
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State

[Hangxiang Yu]

Hi, Yun and Feifan.

Thanks for your reply.

About the cleanup of working dir, as mentioned in FLIP-427, "The life cycle
of working dir is managed as before local strategy.".
Since the current working dir and checkpoint dir are separate, The life
cycle including creating and cleanup of working dir could be aligned with
before easily.

On Thu, Mar 28, 2024 at 12:07 PM Feifan Wang  wrote:

> And I think the cleanup of working dir should be discussion in FLIP-427[1]
> ( this mail list [2]) ?
>
>
> [1] https://cwiki.apache.org/confluence/x/T4p3EQ
> [2] https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft
>
> ——
>
> Best regards,
>
> Feifan Wang
>
>
>
>
> At 2024-03-28 11:56:22, "Feifan Wang"  wrote:
> >Hi Jinzhong :
> >
> >
> >> I suggest that we could postpone this topic for now and consider it
> comprehensively combined with the TM ownership file management in the
> future FLIP.
> >
> >
> >Sorry I still think we should consider the cleanup of the working dir in
> this FLIP, although we may come up with a better solution in a subsequent
> flip, I think it is important to maintain the integrity of the current
> changes. Otherwise we may suffer from wasted DFS space for some time.
> >Perhaps we only need a simple cleanup strategy at this stage, such as
> proactive cleanup when TM exits. While this may fail in the case of a TM
> crash, it already alleviates the problem.
> >
> >
> >
> >
> >——
> >
> >Best regards,
> >
> >Feifan Wang
> >
> >
> >
> >
> >At 2024-03-28 11:15:11, "Jinzhong Li"  wrote:
> >>Hi Yun,
> >>
> >>Thanks for your reply.
> >>
> >>> 1. Why must we have another 'subTask-checkpoint-sub-dir'
> >>> under the shared directory? if we don't consider making
> >>> TM ownership in this FLIP, this design seems unnecessary.
> >>
> >> Good catch! We will not change the directory layout of shared directory
> in
> >>this FLIP. I have already removed this part from this FLIP. I think we
> >>could revisit this topic in a future FLIP about TM ownership.
> >>
> >>> 2. This FLIP forgets to mention the cleanup of the remote
> >>> working directory in case of the taskmanager crushes,
> >>> even though this is an open problem, we can still leave
> >>> some space for future optimization.
> >>
> >>Considering that we have plans to merge TM working dir and checkpoint dir
> >>into one directory, I suggest that we could postpone this topic for now
> and
> >>consider it comprehensively combined with the TM ownership file
> management
> >>in the future FLIP.
> >>
> >>Best,
> >>Jinzhong
> >>
> >>
> >>
> >>On Wed, Mar 27, 2024 at 11:49 PM Yun Tang  wrote:
> >>
> >>> Hi Jinzhong,
> >>>
> >>> The overall design looks good.
> >>>
> >>> I have two minor questions:
> >>>
> >>> 1. Why must we have another 'subTask-checkpoint-sub-dir' under the
> shared
> >>> directory? if we don't consider making TM ownership in this FLIP, this
> >>> design seems unnecessary.
> >>> 2. This FLIP forgets to mention the cleanup of the remote working
> >>> directory in case of the taskmanager crushes, even though this is an
> open
> >>> problem, we can still leave some space for future optimization.
> >>>
> >>> Best,
> >>> Yun Tang
> >>>
> >>> 
> >>> From: Jinzhong Li 
> >>> Sent: Monday, March 25, 2024 10:41
> >>> To: dev@flink.apache.org 
> >>> Subject: Re: [DISCUSS] FLIP-428: Fault Tolerance/Rescale Integration
> for
> >>> Disaggregated State
> >>>
> >>> Hi Yue,
> >>>
> >>> Thanks for your comments.
> >>>
> >>> The CURRENT is a special file that points to the latest manifest log
> >>> file. As Zakelly explained above, we could record the latest manifest
> >>> filename during sync phase, and write the filename into CURRENT
> snapshot
> >>> file during async phase.
> >>>
> >>> Best,
> >>> Jinzhong
> >>>
> >>> On Fri, Mar 22, 2024 at 11:16 PM Zakelly Lan 
> >>> wrote:
> >>>
> >>> > Hi Yue,
> >>> >
> >>> > Thanks for bringing this up!
> >>> >
> >>> > The CURRENT FILE is the special one, which should be snapshot during
> the
> >>> > sync phase (temporary load into memory). Thus we can solve this.
> >>> >
> >>> >
> >>> > Best,
> >>> > Zakelly
> >>> >
> >>> > On Fri, Mar 22, 2024 at 4:55 PM yue ma  wrote:
> >>> >
> >>> > > Hi jinzhong,
> >>> > > Thanks for you reply. I still have some doubts about the first
> >>> question.
> >>> > Is
> >>> > > there such a case
> >>> > > When you made a snapshot during the synchronization phase, you
> recorded
> >>> > the
> >>> > > current and manifest 8, but before asynchronous phase, the manifest
> >>> > reached
> >>> > > the size threshold and then the CURRENT FILE pointed to the new
> >>> manifest
> >>> > 9,
> >>> > > and then uploaded the incorrect CURRENT file ?
> >>> > >
> >>> > > Jinzhong Li  于2024年3月20日周三 20:13写道：
> >>> > >
> >>> > > > Hi Yue,
> >>> > > >
> >>> > > > Thanks for your feedback!
> >>> > > >
> >>> > > > > 1. If we choose Option-3 for ForSt , how would we handle
> Manifest
> >>> > File
> >>> > > > > ? Should we take a snapshot of the 

Re: Re: [DISCUSS] FLIP-427: Disaggregated State Store

[Hangxiang Yu]

Hi, Feifan.

Thanks for your reply.

What if we only use jni to access DFS that needs to reuse Flink FileSystem?
> And all local disk access through native api. This idea is based on the
> understanding that jni overhead is not worth mentioning compared to DFS
> access latency. It might make more sense to consider avoiding jni overhead
> for faster local disks. Since local disk as secondary is already under
> consideration [1], maybe we can discuss in that FLIP whether to use native
> api to access local disk?
>
This is a good suggestion. It's reasonable to use native api to access
local disk cache since it requires lower latency compared to remote access.
I also believe that the jni overhead is relatively negligible when weighed
against the latency of remote I/O as mentioned in the FLIP.
So I think we could just go on proposal 2 and keep proposal 1 as a
potential future optimization, which could work better when there is a
higher performance requirement or some native libraries of filesystems have
significantly higher performance and resource usage compared to their java
libs.


On Thu, Mar 28, 2024 at 11:39 AM Feifan Wang  wrote:

> Thanks for this valuable proposal Hangxiang !
>
>
> > If we need to introduce a JNI call during each filesystem call, that
> would be N times JNI cost compared with the current RocksDB state-backend's
> JNI cost.
> What if we only use jni to access DFS that needs to reuse Flink
> FileSystem? And all local disk access through native api. This idea is
> based on the understanding that jni overhead is not worth mentioning
> compared to DFS access latency. It might make more sense to consider
> avoiding jni overhead for faster local disks. Since local disk as secondary
> is already under consideration [1], maybe we can discuss in that FLIP
> whether to use native api to access local disk?
>
>
> >I'd suggest keeping `state.backend.forSt.working-dir` as it is for now.
> >Different disaggregated state storages may have their own semantics about
> >this configuration, e.g. life cycle, supported file systems or storages.
> I agree with considering moving this configuration up to the engine level
> until there are other disaggreated backends.
>
>
> [1] https://cwiki.apache.org/confluence/x/U4p3EQ
>
> ——
>
> Best regards,
>
> Feifan Wang
>
>
>
>
> At 2024-03-28 09:55:48, "Hangxiang Yu"  wrote:
> >Hi, Yun.
> >Thanks for the reply.
> >
> >The JNI cost you considered is right. As replied to Yue, I agreed to leave
> >space and consider proposal 1 as an optimization in the future, which is
> >also updated in the FLIP.
> >
> >The other question is that the configuration of
> >> `state.backend.forSt.working-dir` looks too coupled with the ForSt
> >> state-backend, how would it be if we introduce another disaggregated
> state
> >> storage? Thus, I think `state.backend.disaggregated.working-dir` might
> be a
> >> better configuration name.
> >
> >I'd suggest keeping `state.backend.forSt.working-dir` as it is for now.
> >Different disaggregated state storages may have their own semantics about
> >this configuration, e.g. life cycle, supported file systems or storages.
> >Maybe it's more suitable to consider it together when we introduce other
> >disaggregated state storages in the future.
> >
> >On Thu, Mar 28, 2024 at 12:02 AM Yun Tang  wrote:
> >
> >> Hi Hangxiang,
> >>
> >> The design looks good, and I also support leaving space for proposal 1.
> >>
> >> As you know, loading index/filter/data blocks for querying across levels
> >> would introduce high IO access within the LSM tree for old data. If we
> need
> >> to introduce a JNI call during each filesystem call, that would be N
> times
> >> JNI cost compared with the current RocksDB state-backend's JNI cost.
> >>
> >> The other question is that the configuration of
> >> `state.backend.forSt.working-dir` looks too coupled with the ForSt
> >> state-backend, how would it be if we introduce another disaggregated
> state
> >> storage? Thus, I think `state.backend.disaggregated.working-dir` might
> be a
> >> better configuration name.
> >>
> >>
> >> Best
> >> Yun Tang
> >>
> >> 
> >> From: Hangxiang Yu 
> >> Sent: Wednesday, March 20, 2024 11:32
> >> To: dev@flink.apache.org 
> >> Subject: Re: [DISCUSS] FLIP-427: Disaggregated State Store
> >>
> >> Hi, Yue.
> >> Thanks for the reply.
> >>
> >> If we use proposal1, we can easily reuse these optimizati

Re: [DISCUSS] FLIP-427: Disaggregated State Store

[Hangxiang Yu]

Hi, Yun.
Thanks for the reply.

The JNI cost you considered is right. As replied to Yue, I agreed to leave
space and consider proposal 1 as an optimization in the future, which is
also updated in the FLIP.

The other question is that the configuration of
> `state.backend.forSt.working-dir` looks too coupled with the ForSt
> state-backend, how would it be if we introduce another disaggregated state
> storage? Thus, I think `state.backend.disaggregated.working-dir` might be a
> better configuration name.

I'd suggest keeping `state.backend.forSt.working-dir` as it is for now.
Different disaggregated state storages may have their own semantics about
this configuration, e.g. life cycle, supported file systems or storages.
Maybe it's more suitable to consider it together when we introduce other
disaggregated state storages in the future.

On Thu, Mar 28, 2024 at 12:02 AM Yun Tang  wrote:

> Hi Hangxiang,
>
> The design looks good, and I also support leaving space for proposal 1.
>
> As you know, loading index/filter/data blocks for querying across levels
> would introduce high IO access within the LSM tree for old data. If we need
> to introduce a JNI call during each filesystem call, that would be N times
> JNI cost compared with the current RocksDB state-backend's JNI cost.
>
> The other question is that the configuration of
> `state.backend.forSt.working-dir` looks too coupled with the ForSt
> state-backend, how would it be if we introduce another disaggregated state
> storage? Thus, I think `state.backend.disaggregated.working-dir` might be a
> better configuration name.
>
>
> Best
> Yun Tang
>
> 
> From: Hangxiang Yu 
> Sent: Wednesday, March 20, 2024 11:32
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS] FLIP-427: Disaggregated State Store
>
> Hi, Yue.
> Thanks for the reply.
>
> If we use proposal1, we can easily reuse these optimizations .It is even
> > possible to discuss and review the solution together in the Rocksdb
> > community.
>
> We also saw these useful optimizations which could be applied to ForSt in
> the future.
> But IIUC, it's not binding to proposal 1, right? We could also
> implement interfaces about temperature and secondary cache to reuse them,
> or organize a more complex HybridEnv based on proposal 2.
>
> My point is whether we should retain the potential of proposal 1 in the
> > design.
> >
> This is a good suggestion. We choose proposal 2 firstly due to its
> maintainability and scalability, especially because it could leverage all
> filesystems flink supported conveniently.
> Given the indelible advantage in performance, I think we could also
> consider proposal 1 as an optimization in the future.
> For the interface on the DB side, we could also expose more different Envs
> in the future.
>
>
> On Tue, Mar 19, 2024 at 9:14 PM yue ma  wrote:
>
> > Hi Hangxiang,
> >
> > Thanks for bringing this discussion.
> > I have a few questions about the Proposal you mentioned in the FLIP.
> >
> > The current conclusion is to use proposal 2, which is okay for me. My
> point
> > is whether we should retain the potential of proposal 1 in the design.
> > There are the following reasons:
> > 1. No JNI overhead, just like the Performance Part mentioned in Flip
> > 2. RocksDB currently also provides an interface for Env, and there are
> also
> > some implementations, such as HDFS-ENV, which seem to be easily scalable.
> > 3. The RocksDB community continues to support LSM for different storage
> > media, such as  Tiered Storage
> > <
> >
> https://github.com/facebook/rocksdb/wiki/Tiered-Storage-%28Experimental%29
> > >
> >   And some optimizations have been made for this scenario, such as
> Per
> > Key Placement Comparison
> > <https://rocksdb.org/blog/2022/11/09/time-aware-tiered-storage.html>.
> >  *Secondary cache
> > <
> >
> https://github.com/facebook/rocksdb/wiki/SecondaryCache-%28Experimental%29
> > >*,
> > similar to the Hybrid Block Cache mentioned in Flip-423
> >  If we use proposal1, we can easily reuse these optimizations .It is even
> > possible to discuss and review the solution together in the Rocksdb
> > community.
> >  In fact, we have already implemented some production practices using
> > Proposal1 internally. We have integrated HybridEnv, Tiered Storage, and
> > Secondary Cache on RocksDB and optimized the performance of Checkpoint
> and
> > State Restore. It seems work well for us.
> >
> > --
> > Best,
> > Yue
> >
>
>
> --
> Best,
> Hangxiang.
>


-- 
Best,
Hangxiang.

[VOTE] FLIP-427: Disaggregated state Store

[Hangxiang Yu]

Hi devs,

Thanks all for your valuable feedback about FLIP-427: Disaggregated state
Store [1].
I'd like to start a vote on it.  The discussion thread is here [2].

The vote will be open for at least 72 hours unless there is an objection or
insufficient votes.

[1] https://cwiki.apache.org/confluence/x/T4p3EQ
[2] https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft


Best,
Hangxiang

Re: [DISCUSS] Flink Website Menu Adjustment

[Hangxiang Yu]

Thanks Zhongqiang for driving this.
+1 for the proposal.

On Tue, Mar 26, 2024 at 1:36 PM Shawn Huang  wrote:

> +1 for the proposal
>
> Best,
> Shawn Huang
>
>
> Hongshun Wang  于2024年3月26日周二 11:56写道：
>
> > +1 for the proposal
> >
> > Best Regards,
> > Hongshun Wang
> >
> > On Tue, Mar 26, 2024 at 11:37 AM gongzhongqiang <
> gongzhongqi...@apache.org
> > >
> > wrote:
> >
> > > Hi Martijn,
> > >
> > > Thank you for your feedback.
> > >
> > > I agree with your point that we should make a one-time update to the
> > menu,
> > > rather than continuously updating it. This will be done unless some
> > > sub-projects are moved or archived.
> > >
> > > Best regards,
> > >
> > > Zhongqiang Gong
> > >
> > >
> > > Martijn Visser  于2024年3月25日周一 23:35写道：
> > >
> > > > Hi Zhongqiang Gong,
> > > >
> > > > Are you suggesting to continuously update the menu based on the
> number
> > of
> > > > releases, or just this one time? I wouldn't be in favor of
> continuously
> > > > updating: returning customers expect a certain order in the menu,
> and I
> > > > don't see a lot of value in continuously changing that. I do think
> that
> > > the
> > > > order that you have currently proposed is better then the one we have
> > > right
> > > > now, so I would +1 a one-time update but not a continuously updating
> > > order.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > On Mon, Mar 25, 2024 at 4:15 PM Yanquan Lv 
> > wrote:
> > > >
> > > > > +1 for this proposal.
> > > > >
> > > > > gongzhongqiang  于2024年3月25日周一 15:49写道：
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I'd like to start a discussion on adjusting the Flink website [1]
> > > menu
> > > > to
> > > > > > improve accuracy and usability.While migrating Flink CDC
> > > documentation
> > > > > > to the website, I found outdated links, need to review and update
> > > menus
> > > > > > for the most relevant information for our users.
> > > > > >
> > > > > >
> > > > > > Proposal:
> > > > > >
> > > > > > - Remove Paimon [2] from the "Getting Started" and
> "Documentation"
> > > > menus:
> > > > > > Paimon [2] is now an independent top project of ASF. CC： jingsong
> > > lees
> > > > > >
> > > > > > - Sort the projects in the subdirectory by the activity of the
> > > > projects.
> > > > > > Here I list the number of releases for each project in the past
> > year.
> > > > > >
> > > > > > Flink Kubernetes Operator : 7
> > > > > > Flink CDC : 5
> > > > > > Flink ML  : 2
> > > > > > Flink Stateful Functions : 1
> > > > > >
> > > > > >
> > > > > > Expected Outcome :
> > > > > >
> > > > > > - Menu "Getting Started"
> > > > > >
> > > > > > Before:
> > > > > >
> > > > > > With Flink
> > > > > >
> > > > > > With Flink Stateful Functions
> > > > > >
> > > > > > With Flink ML
> > > > > >
> > > > > > With Flink Kubernetes Operator
> > > > > >
> > > > > > With Paimon(incubating) (formerly Flink Table Store)
> > > > > >
> > > > > > With Flink CDC
> > > > > >
> > > > > > Training Course
> > > > > >
> > > > > >
> > > > > > After:
> > > > > >
> > > > > > With Flink
> > > > > > With Flink Kubernetes Operator
> > > > > >
> > > > > > With Flink CDC
> > > > > >
> > > > > > With Flink ML
> > > > > >
> > > > > > With Flink Stateful Functions
> > > > > >
> > > > > > Training Course
> > > > > >
> > > > > >
> > > > > > - Menu "Documentation" will same with "Getting Started"
> > > > > >
> > > > > >
> > > > > > I look forward to hearing your thoughts and suggestions on this
> > > > proposal.
> > > > > >
> > > > > > [1] https://flink.apache.org/
> > > > > > [2] https://github.com/apache/incubator-paimon
> > > > > > [3] https://github.com/apache/flink-statefun
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Zhongqiang Gong
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
Best,
Hangxiang.

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

[Hangxiang Yu]

Congratulations!
Thanks for the efforts.

On Fri, Mar 22, 2024 at 10:00 AM Yanfei Lei  wrote:

> Congratulations!
>
> Best regards,
> Yanfei
>
> Xuannan Su  于2024年3月22日周五 09:21写道：
> >
> > Congratulations!
> >
> > Best regards,
> > Xuannan
> >
> > On Fri, Mar 22, 2024 at 9:17 AM Charles Zhang 
> wrote:
> > >
> > > Congratulations!
> > >
> > > Best wishes,
> > > Charles Zhang
> > > from Apache InLong
> > >
> > >
> > > Jeyhun Karimov  于2024年3月22日周五 04:16写道：
> > >
> > > > Great news! Congratulations!
> > > >
> > > > Regards,
> > > > Jeyhun
> > > >
> > > > On Thu, Mar 21, 2024 at 2:00 PM Yuxin Tan 
> wrote:
> > > >
> > > > > Congratulations! Thanks for the efforts.
> > > > >
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > > >
> > > > > Samrat Deb  于2024年3月21日周四 20:28写道：
> > > > >
> > > > > > Congratulations !
> > > > > >
> > > > > > Bests
> > > > > > Samrat
> > > > > >
> > > > > > On Thu, 21 Mar 2024 at 5:52 PM, Ahmed Hamdy <
> hamdy10...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Congratulations, great work and great news.
> > > > > > > Best Regards
> > > > > > > Ahmed Hamdy
> > > > > > >
> > > > > > >
> > > > > > > On Thu, 21 Mar 2024 at 11:41, Benchao Li  >
> > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, and thanks for the great work!
> > > > > > > >
> > > > > > > > Yuan Mei  于2024年3月21日周四 18:31写道：
> > > > > > > > >
> > > > > > > > > Thanks for driving these efforts!
> > > > > > > > >
> > > > > > > > > Congratulations
> > > > > > > > >
> > > > > > > > > Best
> > > > > > > > > Yuan
> > > > > > > > >
> > > > > > > > > On Thu, Mar 21, 2024 at 4:35 PM Yu Li 
> wrote:
> > > > > > > > >
> > > > > > > > > > Congratulations and look forward to its further
> development!
> > > > > > > > > >
> > > > > > > > > > Best Regards,
> > > > > > > > > > Yu
> > > > > > > > > >
> > > > > > > > > > On Thu, 21 Mar 2024 at 15:54, ConradJam <
> jam.gz...@gmail.com>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Congrattulations!
> > > > > > > > > > >
> > > > > > > > > > > Leonard Xu  于2024年3月20日周三 21:36写道：
> > > > > > > > > > >
> > > > > > > > > > > > Hi devs and users,
> > > > > > > > > > > >
> > > > > > > > > > > > We are thrilled to announce that the donation of
> Flink CDC
> > > > > as a
> > > > > > > > > > > > sub-project of Apache Flink has completed. We invite
> you to
> > > > > > > explore
> > > > > > > > > > the new
> > > > > > > > > > > > resources available:
> > > > > > > > > > > >
> > > > > > > > > > > > - GitHub Repository:
> https://github.com/apache/flink-cdc
> > > > > > > > > > > > - Flink CDC Documentation:
> > > > > > > > > > > >
> https://nightlies.apache.org/flink/flink-cdc-docs-stable
> > > > > > > > > > > >
> > > > > > > > > > > > After Flink community accepted this donation[1], we
> have
> > > > > > > completed
> > > > > > > > > > > > software copyright signing, code repo migration, code
> > > > > cleanup,
> > > > > > > > website
> > > > > > > > > > > > migration, CI migration and github issues migration
> etc.
> > > > > > > > > > > > Here I am particularly grateful to Hang Ruan,
> Zhongqaing
> > > > > Gong,
> > > > > > > > > > Qingsheng
> > > > > > > > > > > > Ren, Jiabao Sun, LvYanquan, loserwang1024 and other
> > > > > > contributors
> > > > > > > > for
> > > > > > > > > > their
> > > > > > > > > > > > contributions and help during this process！
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > For all previous contributors: The contribution
> process has
> > > > > > > > slightly
> > > > > > > > > > > > changed to align with the main Flink project. To
> report
> > > > bugs
> > > > > or
> > > > > > > > > > suggest new
> > > > > > > > > > > > features, please open tickets
> > > > > > > > > > > > Apache Jira (https://issues.apache.org/jira).  Note
> that
> > > > we
> > > > > > will
> > > > > > > > no
> > > > > > > > > > > > longer accept GitHub issues for these purposes.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Welcome to explore the new repository and
> documentation.
> > > > Your
> > > > > > > > feedback
> > > > > > > > > > and
> > > > > > > > > > > > contributions are invaluable as we continue to
> improve
> > > > Flink
> > > > > > CDC.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks everyone for your support and happy exploring
> Flink
> > > > > CDC!
> > > > > > > > > > > >
> > > > > > > > > > > > Best，
> > > > > > > > > > > > Leonard
> > > > > > > > > > > > [1]
> > > > > > > >
> https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Best
> > > > > > > > > > >
> > > > > > > > > > > ConradJam
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Benchao Li
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
>


-- 
Best,
Hangxiang.

Re: [VOTE] FLIP-433: State Access on DataStream API V2

[Hangxiang Yu]

+1 (binding)

On Thu, Mar 21, 2024 at 10:04 AM Xintong Song  wrote:

> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Wed, Mar 20, 2024 at 8:30 PM weijie guo 
> wrote:
>
> > Hi everyone,
> >
> >
> > Thanks for all the feedback about the FLIP-433: State Access on
> > DataStream API V2 [1]. The discussion thread is here [2].
> >
> >
> > The vote will be open for at least 72 hours unless there is an
> > objection or insufficient votes.
> >
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-433%3A+State+Access+on+DataStream+API+V2
> >
> > [2] https://lists.apache.org/thread/8ghzqkvt99p4k7hjqxzwhqny7zb7xrwo
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-427: Disaggregated State Store

[Hangxiang Yu]

Hi, Yue.
Thanks for the reply.

If we use proposal1, we can easily reuse these optimizations .It is even
> possible to discuss and review the solution together in the Rocksdb
> community.

We also saw these useful optimizations which could be applied to ForSt in
the future.
But IIUC, it's not binding to proposal 1, right? We could also
implement interfaces about temperature and secondary cache to reuse them,
or organize a more complex HybridEnv based on proposal 2.

My point is whether we should retain the potential of proposal 1 in the
> design.
>
This is a good suggestion. We choose proposal 2 firstly due to its
maintainability and scalability, especially because it could leverage all
filesystems flink supported conveniently.
Given the indelible advantage in performance, I think we could also
consider proposal 1 as an optimization in the future.
For the interface on the DB side, we could also expose more different Envs
in the future.


On Tue, Mar 19, 2024 at 9:14 PM yue ma  wrote:

> Hi Hangxiang,
>
> Thanks for bringing this discussion.
> I have a few questions about the Proposal you mentioned in the FLIP.
>
> The current conclusion is to use proposal 2, which is okay for me. My point
> is whether we should retain the potential of proposal 1 in the design.
> There are the following reasons:
> 1. No JNI overhead, just like the Performance Part mentioned in Flip
> 2. RocksDB currently also provides an interface for Env, and there are also
> some implementations, such as HDFS-ENV, which seem to be easily scalable.
> 3. The RocksDB community continues to support LSM for different storage
> media, such as  Tiered Storage
> <
> https://github.com/facebook/rocksdb/wiki/Tiered-Storage-%28Experimental%29
> >
>   And some optimizations have been made for this scenario, such as Per
> Key Placement Comparison
> .
>  *Secondary cache
> <
> https://github.com/facebook/rocksdb/wiki/SecondaryCache-%28Experimental%29
> >*,
> similar to the Hybrid Block Cache mentioned in Flip-423
>  If we use proposal1, we can easily reuse these optimizations .It is even
> possible to discuss and review the solution together in the Rocksdb
> community.
>  In fact, we have already implemented some production practices using
> Proposal1 internally. We have integrated HybridEnv, Tiered Storage, and
> Secondary Cache on RocksDB and optimized the performance of Checkpoint and
> State Restore. It seems work well for us.
>
> --
> Best,
> Yue
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-427: Disaggregated State Store

[Hangxiang Yu]

Hi everyone,

Thanks for your valuable feedback!

Our discussions have been going on for a while.
As a sub-FLIP of FLIP-423 which is nearing a consensus, I would like to
start a vote after 72 hours.

Please let me know if you have any concerns, thanks!

On Mon, Mar 11, 2024 at 11:48 AM Hangxiang Yu  wrote:

> Hi, Jeyhun.
>
> Thanks for the reply.
>
> Is this argument true for all workloads? Or does this argument also hold
> for workloads with many small files, which is quite a common case [1] ?
>
> Yes, I think so. The overhead should still be considered negligible,
> particularly in comparison to remote I/O, and other benefits of this
> proposal may be more significant than this one.
>
> Additionally, there is JNI overhead when Flink calls RocksDB methods
> currently. The frequency of these calls could surpass that of actual file
> system interface calls, given that not all state requests require accessing
> the file system.
>
> BTW, the issue with small files can also impact the performance of db with
> the local file system at runtime, so we usually resolve this firstly in the
> production environment.
>
> the engine spawns huge amount of scan range requests to the
> file system to retrieve different parts of a file.
>
> Indeed, frequent requests to the remote file system can significantly
> affect performance. To address this, other FLIPs have introduced various
> strategies:
>
> 1. Local disk cache to minimize remote requests as described in FLIP-423
> which we will introduce in FLIP-429 as you mentioned. With effective cache
> utilization, the performance will not be inferior to the local strategy
> when cache hits.
>
> 2. Grouping remote access to decrease the number of remote I/O requests,
> as proposed in "FLIP-426: Grouping Remote State Access."
>
> 3. Parallel I/O to maximize network bandwidth usage, outlined in
> "FLIP-425: Asynchronous Execution Model."
>
> The PoC implements a simple file cache and asynchronous execution which
> improves the performance a lot. You could also refer to the PoC results in
> FLIP-423.
>
> On Mon, Mar 11, 2024 at 3:11 AM Jeyhun Karimov 
> wrote:
>
>> Hi Hangxiang,
>>
>> Thanks for the proposal. +1 for it.
>> I have a few comments.
>>
>> Proposal 2 has additional JNI overhead, but the overhead is relatively
>> > negligible when weighed against the latency of remote I/O.
>>
>> - Is this argument true for all workloads? Or does this argument also hold
>> for workloads with many small files, which is quite a common case [1] ?
>>
>> - Also, in many workloads the engine does not need the whole file either
>> because of the query forces it or
>> file type supports efficient filtering (e.g. ORC, parquet, arrow files),
>> or
>> simply one file is "divided" among multiple workers.
>> In these cases, the engine spawns huge amount of scan range requests to
>> the
>> file system to retrieve different parts of a file.
>> How the proposed solution would work with these workloads?
>>
>> - The similar question related to the above applies also for caching ( I
>> know caching is subject of FLIP-429, asking here becasue of the related
>> section in this FLIP).
>>
>> Regards,
>> Jeyhun
>>
>> [1] https://blog.min.io/challenge-big-data-small-files/
>>
>>
>>
>> On Thu, Mar 7, 2024 at 10:09 AM Hangxiang Yu  wrote:
>>
>> > Hi devs,
>> >
>> >
>> > I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated
>> > State Storage and Management[1], which is a joint work of Yuan Mei,
>> Zakelly
>> > Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:
>> >
>> > - FLIP-427: Disaggregated State Store
>> >
>> > This FLIP introduces the initial version of the ForSt disaggregated
>> state
>> > store.
>> >
>> > Please make sure you have read the FLIP-423[1] to know the whole story,
>> and
>> > we'll discuss the details of FLIP-427[2] under this mail. For the
>> > discussion of overall architecture or topics related with multiple
>> > sub-FLIPs, please post in the previous mail[3].
>> >
>> > Looking forward to hearing from you!
>> >
>> > [1] https://cwiki.apache.org/confluence/x/R4p3EQ
>> >
>> > [2] https://cwiki.apache.org/confluence/x/T4p3EQ
>> >
>> > [3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
>> >
>> >
>> > Best,
>> >
>> > Hangxiang.
>> >
>>
>
>
> --
> Best,
> Hangxiang.
>


-- 
Best,
Hangxiang.

Re: [ANNOUNCE] Apache Flink 1.19.0 released

[Hangxiang Yu]

Congratulations!
Thanks release managers and all involved!

On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan  wrote:

> Congratulations!
>
> Best,
> Hang
>
> Paul Lam  于2024年3月18日周一 17:18写道：
>
> > Congrats! Thanks to everyone involved!
> >
> > Best,
> > Paul Lam
> >
> > > 2024年3月18日 16:37，Samrat Deb  写道：
> > >
> > > Congratulations !
> > >
> > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li 
> > wrote:
> > >
> > >> Congratulations!
> > >>
> > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com> wrote:
> > >>>
> > >>> Congratulations, thanks for the great work!
> > >>>
> > >>> Best,
> > >>> Rui
> > >>>
> > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee 
> > >> wrote:
> > 
> >  The Apache Flink community is very happy to announce the release of
> > >> Apache Flink 1.19.0, which is the fisrt release for the Apache Flink
> > 1.19
> > >> series.
> > 
> >  Apache Flink® is an open-source stream processing framework for
> > >> distributed, high-performing, always-available, and accurate data
> > streaming
> > >> applications.
> > 
> >  The release is available for download at:
> >  https://flink.apache.org/downloads.html
> > 
> >  Please check out the release blog post for an overview of the
> > >> improvements for this bugfix release:
> > 
> > >>
> >
> https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/
> > 
> >  The full release notes are available in Jira:
> > 
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282
> > 
> >  We would like to thank all contributors of the Apache Flink
> community
> > >> who made this release possible!
> > 
> > 
> >  Best,
> >  Yun, Jing, Martijn and Lincoln
> > >>
> >
> >
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-34660) AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34660:


 Summary: AutoRescalingITCase#testCheckpointRescalingInKeyedState 
AssertionError
 Key: FLINK-34660
 URL: https://issues.apache.org/jira/browse/FLINK-34660
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Reporter: Hangxiang Yu


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58249=ms.vss-test-web.build-test-results-tab=4036370=100718=debug]

 
{code:java}
expected:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000), (1,3), 
(1,54000), (0,2000), (0,1), (0,5), (0,66000), (0,74000), (0,82000), 
(1,8), (1,0), (1,16000), (1,24000), (1,4), (1,56000), (1,64000), 
(0,12000), (0,28000), (0,52000), (0,6), (0,68000), (0,76000), (1,18000), 
(1,26000), (1,34000), (1,42000), (1,58000), (0,6000), (0,14000), (0,22000), 
(0,38000), (0,46000), (0,62000), (0,7), (1,4000), (1,2), (1,36000), 
(1,44000)]> but was:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000), 
(1,3), (1,54000), (0,2000), (0,1), (0,5), (0,66000), (0,74000), 
(0,82000), (0,23000), (0,31000), (1,8), (1,0), (1,16000), (1,24000), 
(1,4), (1,56000), (1,64000), (0,12000), (0,28000), (0,52000), (0,6), 
(0,68000), (0,76000), (1,18000), (1,26000), (1,34000), (1,42000), (1,58000), 
(0,6000), (0,14000), (0,22000), (0,19000), (0,35000), (1,4000), (1,2), 
(1,36000), (1,44000)]> {code}
 

This maybe related to FLINK-34624 as we could see from the log:
{code:java}
03:31:02,073 [ main] INFO 
org.apache.flink.runtime.testutils.PseudoRandomValueSelector [] - Randomly 
selected true for state.changelog.enabled
03:31:02,163 [jobmanager-io-thread-2] INFO 
org.apache.flink.state.changelog.AbstractChangelogStateBackend [] - 
ChangelogStateBackend is used, delegating EmbeddedRocksDBStateBackend. {code}
FLINK-34624 disables changelog since it doesn't support local rescaling 
currently.

Even if disabling changelog for AutoRescalingITCase manually, 
randomization may still be applied to it.
We should apply randomization only when it's not pre-defined.
 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34652) Use CheckpointStreamFactory for StateChangeFsUploader

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34652:


 Summary: Use CheckpointStreamFactory for StateChangeFsUploader
 Key: FLINK-34652
 URL: https://issues.apache.org/jira/browse/FLINK-34652
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: Hangxiang Yu


As discussed before, we may consider supporting CheckpointStreamFactory for 
StateChangeFsUploader:
 * need for checkpointId in the current API to resolve the location
 * 
different settings for materialized/changelog (e.g. timeouts)
 * 
re-use closeAndGetHandle
 * 
re-use in-memory handles (.metadata)
 * 
handle in-memory handles duplication

It could be considered together with FLINK-32085



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-433: State Access on DataStream API V2

[Hangxiang Yu]

concept of partition.
> In the
> > > > > case of KeyedPartitionStream, one key corresponds to one
> partition.For
> > > > > NonKeyedPartitionStream one parallelism/subtask corresponds to one
> > > > > partition. All states are considered to be confined within the
> > > partition.
> > > > > On this basis, an obvious question is whether and how the state
> should
> > > be
> > > > > redistribution when the partition changes? So we divide the state
> into
> > > > > three categories:
> > > > > >
> > > > > >- Don't need to redistribute states when the partition
> changes.
> > > > > >- Has to decide how to distribute states when the partition
> > > changes.
> > > > > >- Always has the same state across different partitions.
> > > > > >
> > > > > > After introducing the concept of partition, the redistribution
> > > > > pattern/mode of state is the more essential difference between
> states.
> > > > For
> > > > > this reason, we don't want to emphasize keyed/operator state in
> the V2
> > > > API
> > > > > > any
> > > > > > more. Keep in mind, partition are first-class citizens. And,
> even in
> > > > V1,
> > > > > we have to let the user know that split/union are two different
> > > > strategies
> > > > > for list state.
> > > > > >
> > > > > >
> > > > > >
> > > > > > As for whether or not to expose RedistributionMode to users, I
> have
> > > an
> > > > > open mind. But as I said just now, we still can't avoid this
> problem in
> > > > the
> > > > > splitRedistributionListState and unionRedistributionListState. IMO,
> > > it's
> > > > > better to explain it in the API level instead of avoiding it. WDTY?
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Weijie
> > > > > >
> > > > > >
> > > > > > weijie guo  于2024年3月7日周四 16:39写道：
> > > > > >
> > > > > >> Hi Gyula,
> > > > > >>
> > > > > >>
> > > > > >> Thanks for your reply!
> > > > > >>
> > > > > >>
> > > > > >> Let me answer these questions:
> > > > > >>
> > > > > >>
> > > > > >> > What is the semantics of the usesStates method? When is it
> called?
> > > > Can
> > > > > >> the used state change dynamically at runtime? Can the logic
> depend
> > > on
> > > > > something computed in open(..) for example?
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> useStates is used to predefine all the states that the process
> > > > function
> > > > > needs to access. In other words, we want to avoid declaring the
> state
> > > > > dynamically at runtime and this allows the SQL planner and JM to
> > > optimize
> > > > > the job better. As a result, this logic must be fully available at
> > > > compile
> > > > > time (when the JobGraph is generated), so it can't rely on
> computations
> > > > > that are executed after deploy to TM.
> > > > > >>
> > > > > >>
> > > > > >> >
> > > > > >> Currently state access is pretty dynamic in Flink and I would
> assume
> > > > > many jobs create states on the fly based on some required logic.
> Are we
> > > > > planning to address these use-cases?
> > > > > >>
> > > > > >>
> > > > > >> It depends on what type of context we need. If the type and
> number
> > > of
> > > > > states depend on runtime context, that's something we want to
> avoid. If
> > > > it
> > > > > only depended on information available at compile time, I think we
> > > could
> > > > > support
> > > > > >> it.
> > > > > >>
> > > > > >>
> > > > > >> >
> > > > > >> Are we planning to support deleting/dropping states that are not
> > > > 

Re: [VOTE] Release 1.19.0, release candidate #2

[Hangxiang Yu]

+1 (non-binding)

- Verified signatures and checksums
- Reviewed Web PR
- Built from source successfully
- Ran a wordcount job which worked well



On Tue, Mar 12, 2024 at 1:00 AM Jeyhun Karimov  wrote:

> +1 (non binding)
>
> - verified that source distribution does not contain binaries
> - verified signatures and checksums
> - built source code successfully
>
> Regards,
> Jeyhun
>
>
> On Mon, Mar 11, 2024 at 3:08 PM Samrat Deb  wrote:
>
> > +1 (non binding)
> >
> > - verified signatures and checksums
> > - ASF headers are present in all expected file
> > - No unexpected binaries files found in the source
> > - Build successful locally
> > - tested basic word count example
> >
> >
> >
> >
> > Bests,
> > Samrat
> >
> > On Mon, 11 Mar 2024 at 7:33 PM, Ahmed Hamdy 
> wrote:
> >
> > > Hi Lincoln
> > > +1 (non-binding) from me
> > >
> > > - Verified Checksums & Signatures
> > > - Verified Source dists don't contain binaries
> > > - Built source successfully
> > > - reviewed web PR
> > >
> > >
> > > Best Regards
> > > Ahmed Hamdy
> > >
> > >
> > > On Mon, 11 Mar 2024 at 15:18, Lincoln Lee 
> > wrote:
> > >
> > > > Hi Robin,
> > > >
> > > > Thanks for helping verifying the release note[1], FLINK-14879 should
> > not
> > > > have been included, after confirming this
> > > > I moved all unresolved non-blocker issues left over from 1.19.0 to
> > 1.20.0
> > > > and reconfigured the release note [1].
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282
> > > >
> > > >
> > > > Robin Moffatt  于2024年3月11日周一 19:36写道：
> > > >
> > > > > Looking at the release notes [1] it lists `DESCRIBE DATABASE`
> > > > (FLINK-14879)
> > > > > and `DESCRIBE CATALOG` (FLINK-14690).
> > > > > When I try these in 1.19 RC2 the behaviour is as in 1.18.1, i.e. it
> > is
> > > > not
> > > > > supported:
> > > > >
> > > > > ```
> > > > > [INFO] Execute statement succeed.
> > > > >
> > > > > Flink SQL> show catalogs;
> > > > > +-+
> > > > > |catalog name |
> > > > > +-+
> > > > > |   c_new |
> > > > > | default_catalog |
> > > > > +-+
> > > > > 2 rows in set
> > > > >
> > > > > Flink SQL> DESCRIBE CATALOG c_new;
> > > > > [ERROR] Could not execute SQL statement. Reason:
> > > > > org.apache.calcite.sql.validate.SqlValidatorException: Column
> 'c_new'
> > > not
> > > > > found in any table
> > > > >
> > > > > Flink SQL> show databases;
> > > > > +--+
> > > > > |database name |
> > > > > +--+
> > > > > | default_database |
> > > > > +--+
> > > > > 1 row in set
> > > > >
> > > > > Flink SQL> DESCRIBE DATABASE default_database;
> > > > > [ERROR] Could not execute SQL statement. Reason:
> > > > > org.apache.calcite.sql.validate.SqlValidatorException: Column
> > > > > 'default_database' not found in
> > > > > any table
> > > > > ```
> > > > >
> > > > > Is this an error in the release notes, or my mistake in
> interpreting
> > > > them?
> > > > >
> > > > > thanks, Robin.
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282
> > > > >
> > > > > On Thu, 7 Mar 2024 at 10:01, Lincoln Lee 
> > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Please review and vote on the release candidate #2 for the
> version
> > > > > 1.19.0,
> > > > > > as follows:
> > > > > > [ ] +1, Approve the release
> > > > > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > > >
> > > > > > The complete staging area is available for your review, which
> > > includes:
> > > > > >
> > > > > > * JIRA release notes [1], and the pull request adding release
> note
> > > for
> > > > > > users [2]
> > > > > > * the official Apache source release and binary convenience
> > releases
> > > to
> > > > > be
> > > > > > deployed to dist.apache.org [3], which are signed with the key
> > with
> > > > > > fingerprint E57D30ABEE75CA06  [4],
> > > > > > * all artifacts to be deployed to the Maven Central Repository
> [5],
> > > > > > * source code tag "release-1.19.0-rc2" [6],
> > > > > > * website pull request listing the new release and adding
> > > announcement
> > > > > blog
> > > > > > post [7].
> > > > > >
> > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > majority
> > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282
> > > > > > [2] https://github.com/apache/flink/pull/24394
> > > > > > [3]
> https://dist.apache.org/repos/dist/dev/flink/flink-1.19.0-rc2/
> > > > > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > [5]
> > > > >
> > 

Re: [DISCUSS] FLIP-427: Disaggregated State Store

[Hangxiang Yu]

Hi, Jeyhun.

Thanks for the reply.

Is this argument true for all workloads? Or does this argument also hold
for workloads with many small files, which is quite a common case [1] ?

Yes, I think so. The overhead should still be considered negligible,
particularly in comparison to remote I/O, and other benefits of this
proposal may be more significant than this one.

Additionally, there is JNI overhead when Flink calls RocksDB methods
currently. The frequency of these calls could surpass that of actual file
system interface calls, given that not all state requests require accessing
the file system.

BTW, the issue with small files can also impact the performance of db with
the local file system at runtime, so we usually resolve this firstly in the
production environment.

the engine spawns huge amount of scan range requests to the
file system to retrieve different parts of a file.

Indeed, frequent requests to the remote file system can significantly
affect performance. To address this, other FLIPs have introduced various
strategies:

1. Local disk cache to minimize remote requests as described in FLIP-423
which we will introduce in FLIP-429 as you mentioned. With effective cache
utilization, the performance will not be inferior to the local strategy
when cache hits.

2. Grouping remote access to decrease the number of remote I/O requests, as
proposed in "FLIP-426: Grouping Remote State Access."

3. Parallel I/O to maximize network bandwidth usage, outlined in "FLIP-425:
Asynchronous Execution Model."

The PoC implements a simple file cache and asynchronous execution which
improves the performance a lot. You could also refer to the PoC results in
FLIP-423.

On Mon, Mar 11, 2024 at 3:11 AM Jeyhun Karimov  wrote:

> Hi Hangxiang,
>
> Thanks for the proposal. +1 for it.
> I have a few comments.
>
> Proposal 2 has additional JNI overhead, but the overhead is relatively
> > negligible when weighed against the latency of remote I/O.
>
> - Is this argument true for all workloads? Or does this argument also hold
> for workloads with many small files, which is quite a common case [1] ?
>
> - Also, in many workloads the engine does not need the whole file either
> because of the query forces it or
> file type supports efficient filtering (e.g. ORC, parquet, arrow files), or
> simply one file is "divided" among multiple workers.
> In these cases, the engine spawns huge amount of scan range requests to the
> file system to retrieve different parts of a file.
> How the proposed solution would work with these workloads?
>
> - The similar question related to the above applies also for caching ( I
> know caching is subject of FLIP-429, asking here becasue of the related
> section in this FLIP).
>
> Regards,
> Jeyhun
>
> [1] https://blog.min.io/challenge-big-data-small-files/
>
>
>
> On Thu, Mar 7, 2024 at 10:09 AM Hangxiang Yu  wrote:
>
> > Hi devs,
> >
> >
> > I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated
> > State Storage and Management[1], which is a joint work of Yuan Mei,
> Zakelly
> > Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:
> >
> > - FLIP-427: Disaggregated State Store
> >
> > This FLIP introduces the initial version of the ForSt disaggregated state
> > store.
> >
> > Please make sure you have read the FLIP-423[1] to know the whole story,
> and
> > we'll discuss the details of FLIP-427[2] under this mail. For the
> > discussion of overall architecture or topics related with multiple
> > sub-FLIPs, please post in the previous mail[3].
> >
> > Looking forward to hearing from you!
> >
> > [1] https://cwiki.apache.org/confluence/x/R4p3EQ
> >
> > [2] https://cwiki.apache.org/confluence/x/T4p3EQ
> >
> > [3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
> >
> >
> > Best,
> >
> > Hangxiang.
> >
>


-- 
Best,
Hangxiang.

[DISCUSS] FLIP-427: Disaggregated State Store

[Hangxiang Yu]

Hi devs,


I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated
State Storage and Management[1], which is a joint work of Yuan Mei, Zakelly
Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:

- FLIP-427: Disaggregated State Store

This FLIP introduces the initial version of the ForSt disaggregated state
store.

Please make sure you have read the FLIP-423[1] to know the whole story, and
we'll discuss the details of FLIP-427[2] under this mail. For the
discussion of overall architecture or topics related with multiple
sub-FLIPs, please post in the previous mail[3].

Looking forward to hearing from you!

[1] https://cwiki.apache.org/confluence/x/R4p3EQ

[2] https://cwiki.apache.org/confluence/x/T4p3EQ

[3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0


Best,

Hangxiang.

Re: [DISCUSS] FLIP-433: State Access on DataStream API V2

[Hangxiang Yu]

Hi, Weijie.
Thanks for your proposal.
I'd like to start the discussion with some questions:
1. We have also discussed in FLIP-359/FLINK-32658 about limiting the user
operation to avoid creating state when processElement. Could current
interfaces also help this?

2. Could you provide more examples about how useStates() works ? Since some
operations may change their used states at runtime, the value this method
returns will be modified at runtime, right ?
If so, I'm thinking if we could get some deterministic State Declaration
Set before running which could help a lot for some state operations e.g.
pre-check schema compatibility, queryable schema.

3. IIUC, RedistributionMode/Strategy should not be used by users, right ?
If so, I'm +1 to move them to inner interfaces which seems a bit confusing
to users.

On Thu, Mar 7, 2024 at 11:39 AM Zakelly Lan  wrote:

> Hi Weijie,
>
> Thanks for proposing this!
>
> Unifying and optimizing state definitions is a very good thing. I like the
> idea of 'definition goes before using', so overall +1 for this proposal.
>
> However, I think the current definition is somewhat unclear. From a user's
> point of view, I believe that state can be characterized along two
> relatively independent axes: the scenario (keyed, non-keyed, or broadcast)
> and the data structure (single value, list, map). I recommend that we fully
> decouple these aspects, rather than linking the nature of the definition to
> specific assumptions, such as equating broadcast states with maps, or
> considering list states could be non-keyed.
> Furthermore, the concept of 'Redistribution' may impose a cognitive burden
> on general users. My advice would be to conceal RedistributionMode/Strategy
> from the standard user interface, particularly within the helper class
> 'State'. But I'm OK to keep it in `StateDeclaration` since its interfaces
> are basically used by the framework. My preferred syntax would be:
> ```
> StateDeclaration a = State.declare(name).keyed().listState(type);
> StateDeclaration b = State.declare(name).broadcast().mapState(typeK,
> typeV);
> StateDeclaration c = State.declare(name).keyed().aggregatingState(type,
> function);
> ```
> WDYT?
>
>
> Best,
> Zakelly
>
> On Wed, Mar 6, 2024 at 11:04 PM Gyula Fóra  wrote:
>
> > Hi Weijie!
> >
> > Thank you for the proposal.
> >
> > I have some initial questions to start the discussion:
> >
> > 1. What is the semantics of the usesStates method? When is it called? Can
> > the used state change dynamically at runtime? Can the logic depend on
> > something computed in open(..) for example?
> >
> > Currently state access is pretty dynamic in Flink and I would assume many
> > jobs create states on the fly based on some required logic. Are we
> planning
> > to address these use-cases?
> >
> > Are we planning to support deleting/dropping states that are not required
> > anymore?
> >
> > 2. Get state now returns an optional, but you mention that:
> > " If you want to get a state that is not declared or has no access,
> > Option#empty is returned."
> >
> > I think if a state is not declared or otherwise cannot be accessed, an
> > exceptions must be thrown. We cannot confuse empty value with something
> > inaccessible.
> >
> > 3. The RedistributionMode enum sounds a bit strange to me, as it doesn't
> > actually specify a mode of redistribution. It feels more like a flag. Can
> > we simply have an Optional instead?
> >
> > 4. BroadcastStates are currently very limited by only Map-like states,
> and
> > the new interface also enforces that.
> > Can we remove this limitation? If not, should broadcastState declaration
> > extend mapstate declaration?
> >
> > Cheers,
> > Gyula
> >
> > Cheers
> > Gyuka
> >
> > On Wed, Mar 6, 2024 at 11:18 AM weijie guo 
> > wrote:
> >
> > > Hi devs,
> > >
> > > I'd like to start a discussion about FLIP-433: State Access on
> > > DataStream API V2
> > > [1]. This is the third sub-FLIP of DataStream API V2.
> > >
> > >
> > > After FLIP-410 [2], we can already write a simple stateless job using
> the
> > > DataStream V2 API.  But as we all know, stateful computing is Flink's
> > trump
> > > card. In this FLIP, we will discuss how to declare and access state on
> > > DataStream API V2 and we manage to avoid some of the shortcomings of V1
> > in
> > > this regard.
> > >
> > > You can find more details in this FLIP. Its relationship with other
> > > sub-FLIPs can be found in the umbrella FLIP
> > > [3]. Looking forward to hearing from you, thanks!
> > >
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-433%3A+State+Access+on+DataStream+API+V2
> > >
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-410%3A++Config%2C+Context+and+Processing+Timer+Service+of+DataStream+API+V2
> > >
> > > [3]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-408%3A+%5BUmbrella%5D+Introduce+DataStream+API+V2
> > >

Re: [VOTE] FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

[Hangxiang Yu]

+1 (binding)

On Thu, Mar 7, 2024 at 9:34 AM Yun Tang  wrote:

> > +1 for this FLIP.
> Sorry for not being clear in my previous reply, it's a binding vote.
>
> Best
> Yun Tang
> 
> From: Jeyhun Karimov 
> Sent: Thursday, March 7, 2024 4:40
> To: dev@flink.apache.org 
> Subject: Re: [VOTE] FLIP-420: Add API annotations for RocksDB StateBackend
> user-facing classes
>
> Hi Jinzhong,
>
> Thanks for the FLIP.
>
> +1 (non-binding)
>
> Regards,
> Jeyhun
>
> On Wed, Mar 6, 2024 at 5:09 PM Yun Tang  wrote:
>
> > +1 for this FLIP.
> >
> >
> >
> > Best
> > Yun Tang
> > 
> > From: Jinzhong Li 
> > Sent: Wednesday, March 6, 2024 20:29
> > To: dev@flink.apache.org 
> > Subject: [VOTE] FLIP-420: Add API annotations for RocksDB StateBackend
> > user-facing classes
> >
> > Hi All,
> >
> > I'd like to start a vote on the FLIP-420: Add API annotations for RocksDB
> > StateBackend user-facing classes[1].
> > The discussion thread is here [2].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> or
> > not enough votes.
> >
> >
> > [1]https://cwiki.apache.org/confluence/x/JQs4EQ
> > [2]https://lists.apache.org/thread/4t71lz2j2ft8hf90ylvtomynhr2qthoo
> >
> >
> > Best,
> > Jinzhong Li
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] Move CheckpointingMode to flink-core

[Hangxiang Yu]

Hi, Zakelly.
Thanks for driving this.
Moving this class to flink-core makes sense to me which could make the code
path and configs clearer.
It's marked as @Public from 1.0 and 1.20 should be the next long-term
version, so 1.19 should have been a suitable version to do it.
And also look forward to thoughts of other developers/RMs since 1.19 is
currently under a feature freeze status.

On Mon, Feb 26, 2024 at 6:42 PM Zakelly Lan  wrote:

> Hi devs,
>
> When working on the FLIP-406[1], I realized that moving all options of
> ExecutionCheckpointingOptions(flink-streaming-java) to
> CheckpointingOptions(flink-core) depends on relocating the
> enum CheckpointingMode(flink-streaming-java) to flink-core module. However,
> the CheckpointingMode is annotated as @Public and used by datastream api
> like 'CheckpointConfig#setCheckpointingMode'. So I'd like to start a
> discussion on moving the CheckpointingMode to flink-core. It is in a little
> bit of a hurry if we want the old enum to be entirely removed in Flink 2.x
> series, since the deprecation should be shipped in the upcoming Flink 1.19.
> I suggest not creating a dedicated FLIP and treating this as a sub-task of
> FLIP-406.
>
> I prepared a minimal change of providing new APIs and deprecating the old
> ones[2], which could be merged to 1.19 if we agree to do so.
>
> Looking forward to your thoughts! Also cc RMs of 1.19 about this.
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> [2]
>
> https://github.com/apache/flink/commit/9bdd237d0322df8853f1b9e6ae658f77b9175237
>
> Best,
> Zakelly
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-34512) Thrown root cause for HandlerRequestException

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34512:


 Summary: Thrown root cause for  HandlerRequestException
 Key: FLINK-34512
 URL: https://issues.apache.org/jira/browse/FLINK-34512
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / REST
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


My Flink job only thrown below exception without the root cause:

 
{code:java}
org.apache.flink.runtime.rest.handler.HandlerRequestException: Cannot resolve 
path parameter (triggerid) from value "4c729cc8-ccc1-80dc-6c8d-9d84d52cfe7b". 
{code}
It's better for HandlerRequest to thrown the root cause of 
HandlerRequestException.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34479) Fix missed changelog configs in the documentation

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34479:


 Summary: Fix missed changelog configs in the documentation
 Key: FLINK-34479
 URL: https://issues.apache.org/jira/browse/FLINK-34479
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.19.0, 1.20.0
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


state_backend_changelog_section has been missed in the documentation



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34352) Improve the documentation of allowNonRestoredState

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34352:


 Summary: Improve the documentation of allowNonRestoredState
 Key: FLINK-34352
 URL: https://issues.apache.org/jira/browse/FLINK-34352
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


Current documentation of allowNonRestoredState is not clear, we should clarify:
 # It can lead to serious issues with correctness if it's used incorrectly.
 # The correctness is related to the topological order and the logic of job 
when removing operator by default.
 # For DataStream Job, the operator uid could be assigned explicitly to avoid 
the reassignment of operator uid.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

[Hangxiang Yu]

Hi Jinzhong.
Thanks for driving this!
Some suggestions:
1. As RocksDBStateBackend marked as Deprecated, We should also
mark RocksDBStateBackendFactory as Deprecated
2. Since 1.19 will be freezed in 1.26. Let's adjust the target version to
1.20


On Wed, Jan 24, 2024 at 11:50 PM Zakelly Lan  wrote:

> Hi Jinzhong,
>
> Thanks for driving this! +1 for fixing the lack of annotation.
>
> I'm wondering if we really need to annotate *RocksDBStateUploader* and
> *RocksDBStateDownloader
> *with @Internal, as they seem to be ordinary classes without interacting
> with other modules.
> Also, I have reservations about annotating *SingleStateIterator*, but I'd
> like to hear others' opinions and won't insist on this.
>
> Best,
> Zakelly
>
> On Wed, Jan 24, 2024 at 10:26 PM Jinzhong Li 
> wrote:
>
> > Hi devs,
> >
> > I’m opening this thread to discuss about FLIP-420: Add API annotations
> for
> > RocksDB StateBackend user-facing classes[1].
> >
> > As described in FLINK-18255[2] , several user-facing classes in
> > flink-statebackend-rocksdb module don't have any API annotations, not
> even
> > @PublicEvolving. This FLIP will add annotations for them to clarify their
> > usage.
> >
> > Looking forward to hearing from you, thanks!
> >
> >
> > Best regards,
> > Jinzhong Li
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-420%3A+Add+API+annotations+for+RocksDB+StateBackend+user-facing+classes
> > [2] https://issues.apache.org/jira/browse/FLINK-18255
> >
>


-- 
Best,
Hangxiang.

Re: [VOTE] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

[Hangxiang Yu]

+1 (binding)

On Thu, Jan 25, 2024 at 8:49 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1(binding)
>
> Best,
> Rui
>
> On Wed, 24 Jan 2024 at 21:50, Zakelly Lan  wrote:
>
> > Hi everyone,
> >
> > I'd like to start a vote on the FLIP-406: Reorganize State &
> Checkpointing
> > & Recovery Configuration [1]. The discussion thread is here [2].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> or
> > insufficient votes.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > [2] https://lists.apache.org/thread/0oc10cr2q2ms855dbo29s7v08xs3bvqg
> >
> >
> > Best,
> > Zakelly
> >
>


-- 
Best,
Hangxiang.

Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

[Hangxiang Yu]

+1 (binding)

On Fri, Jan 19, 2024 at 12:10 PM Zakelly Lan  wrote:

> Hi everyone,
>
> I'd like to start a vote on the FLIP-416: Deprecate and remove the
> RestoreMode#LEGACY [1]. The discussion thread is here [2].
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
> [1] https://cwiki.apache.org/confluence/x/ookkEQ
> [2] https://lists.apache.org/thread/ho77fx13lw4ds52t0fs1xqz2vtn50n2o
>
>
> Best,
> Zakelly
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-34119) Improve description about changelog in document

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34119:


 Summary: Improve description about changelog in document
 Key: FLINK-34119
 URL: https://issues.apache.org/jira/browse/FLINK-34119
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / State Backends
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


Since we have resolved some issues and marked as prodution-ready in [release 
note,|#generalized-incremental-checkpoint]]

we could update some description about it in doc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

[Hangxiang Yu]

Hi, Zakelly.
Thanks for the quick feedback and driving this.
+1 for removing LEGACY mode in Flink 2.0.

On Mon, Jan 15, 2024 at 3:23 AM Danny Cranmer 
wrote:

> +1 to removing LEGACY mode in Flink 2.0. Thanks for driving.
>
> Danny,
>
> On Sat, 13 Jan 2024, 08:20 Yanfei Lei,  wrote:
>
> > Thanks Zakelly for starting this discussion.
> >
> > Regardless of whether it is for users or developers, deprecating
> > RestoreMode#LEGACY makes the semantics clearer and lower maintenance
> > costs, and Flink 2.0 is a good time point to do this.
> > So +1 for the overall idea.
> >
> > Best,
> > Yanfei
> >
> > Zakelly Lan  于2024年1月11日周四 14:57写道：
> >
> > >
> > > Hi devs,
> > >
> > > I'd like to start a discussion on FLIP-416: Deprecate and remove the
> > > RestoreMode#LEGACY[1].
> > >
> > > The FLIP-193[2] introduced two modes of state file ownership during
> > > checkpoint restoration: RestoreMode#CLAIM and RestoreMode#NO_CLAIM. The
> > > LEGACY mode, which was how Flink worked until 1.15, has been superseded
> > by
> > > NO_CLAIM as the default mode. The main drawback of LEGACY mode is that
> > the
> > > new job relies on artifacts from the old job without cleaning them up,
> > > leaving users uncertain about when it is safe to delete the old
> > checkpoint
> > > directories. This leads to the accumulation of unnecessary checkpoint
> > files
> > > that are never cleaned up. Considering cluster availability and job
> > > maintenance, it is not recommended to use LEGACY mode. Users could
> choose
> > > the other two modes to get a clear semantic for the state file
> ownership.
> > >
> > > This FLIP proposes to deprecate the LEGACY mode and remove it
> completely
> > in
> > > the upcoming Flink 2.0. This will make the semantic clear as well as
> > > eliminate many bugs caused by mode transitions involving LEGACY mode
> > (e.g.
> > > FLINK-27114 [3]) and enhance code maintainability.
> > >
> > > Looking forward to hearing from you!
> > >
> > > [1] https://cwiki.apache.org/confluence/x/ookkEQ
> > > [2] https://cwiki.apache.org/confluence/x/bIyqCw
> > > [3] https://issues.apache.org/jira/browse/FLINK-27114
> > >
> > > Best,
> > > Zakelly
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-414: Support Retry Mechanism in RocksDBStateDataTransfer

[Hangxiang Yu]

Thanks for driving this.
Retry mechanism is common when we want to get or put data by network.
So I think it will help when checkpoint failure due to temporary network
problems, of course it may increase a bit overhead for some other reasons.

Some comments and suggestions:
1. Since Flink has a checkpoint mechanism to retry failed checkpoint
coarsely, I think it looks good to me if this fine-grained retry could be
configurable and don't change the current default mechanism.
2. This should work with the checkpoint procedure of all state backends,
Could we make this config unrelated to a specific state backend (maybe
execution.checkpointing.xxx)?  Then it could be supported by below state
backends.
3. We may not need to re-implement it. There are some tools supporting the
Retry mechanism (see RetryingExecutor and RetryPolicy in changelog dstl
module), it's better to make them become more common tools and reuse them.

On Thu, Jan 11, 2024 at 3:09 PM yue ma  wrote:

> Thanks for driving this effort, xiangyu!
> The proposal overall LGTM.
> I just have a small question. There are other places in Flink that interact
> with external storage. Should we consider adding a general retry mechanism
> to them?
>
> xiangyu feng  于2024年1月8日周一 11:31写道：
>
> > Hi devs,
> >
> > I'm opening this thread to discuss FLIP-414: Support Retry Mechanism in
> > RocksDBStateDataTransfer[1].
> >
> > Currently, there is no retry mechanism for downloading and uploading
> > RocksDB state files. Any jittering of remote filesystem might lead to a
> > checkpoint failure. By supporting retry mechanism in
> > `RocksDBStateDataTransfer`, we can significantly reduce the failure rate
> of
> > checkpoint during asynchronous phrase.
> >
> > To make this retry mechanism configurable, we have introduced two options
> > in this FLIP: `state.backend.rocksdb.checkpoint.transfer.retry.times`
> and `
> > state.backend.rocksdb.checkpoint.transfer.retry.interval`. The default
> > behavior remains to be no retry will be performed in order to be
> consistent
> > with the original behavior.
> >
> > Looking forward to your feedback, thanks.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-414%3A+Support+Retry+Mechanism+in+RocksDBStateDataTransfer
> >
> > Best regards,
> > Xiangyu Feng
> >
>
>
> --
> Best,
> Yue
>


-- 
Best,
Hangxiang.

Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink

[Hangxiang Yu]

+1 (non-binding)

On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su  wrote:

> +1 (non-binding)
>
> Best,
> Xuannan
>
> On Thu, Jan 11, 2024 at 10:28 AM Xuyang  wrote:
> >
> > +1 (non-binding)--
> >
> > Best！
> > Xuyang
> >
> >
> >
> >
> >
> > 在 2024-01-11 10:00:11，"Yang Wang"  写道：
> > >+1 (binding)
> > >
> > >
> > >Best,
> > >Yang
> > >
> > >On Thu, Jan 11, 2024 at 9:53 AM liu ron  wrote:
> > >
> > >> +1 non-binding
> > >>
> > >> Best
> > >> Ron
> > >>
> > >> Matthias Pohl  于2024年1月10日周三 23:05写道：
> > >>
> > >> > +1 (binding)
> > >> >
> > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam 
> wrote:
> > >> >
> > >> > > +1 non-binding
> > >> > >
> > >> > > Dawid Wysakowicz  于2024年1月10日周三 21:06写道：
> > >> > >
> > >> > > > +1 (binding)
> > >> > > > Best,
> > >> > > > Dawid
> > >> > > >
> > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr Nowojski <
> pnowoj...@apache.org>
> > >> > > wrote:
> > >> > > >
> > >> > > > > +1 (binding)
> > >> > > > >
> > >> > > > > śr., 10 sty 2024 o 11:25 Martijn Visser <
> martijnvis...@apache.org>
> > >> > > > > napisał(a):
> > >> > > > >
> > >> > > > > > +1 (binding)
> > >> > > > > >
> > >> > > > > > On Wed, Jan 10, 2024 at 4:43 AM Xingbo Huang <
> hxbks...@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > > > > >
> > >> > > > > > > +1 (binding)
> > >> > > > > > >
> > >> > > > > > > Best,
> > >> > > > > > > Xingbo
> > >> > > > > > >
> > >> > > > > > > Dian Fu  于2024年1月10日周三 11:35写道：
> > >> > > > > > >
> > >> > > > > > > > +1 (binding)
> > >> > > > > > > >
> > >> > > > > > > > Regards,
> > >> > > > > > > > Dian
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Jan 10, 2024 at 5:09 AM Sharath <
> > >> dsaishar...@gmail.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > +1 (non-binding)
> > >> > > > > > > > >
> > >> > > > > > > > > Best,
> > >> > > > > > > > > Sharath
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Jan 9, 2024 at 1:02 PM Venkata Sanath
> Muppalla <
> > >> > > > > > > > sanath...@gmail.com>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > +1 (non-binding)
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Sanath
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Tue, Jan 9, 2024 at 11:16 AM Peter Huang <
> > >> > > > > > > > huangzhenqiu0...@gmail.com>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > +1 (non-binding)
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Best Regards
> > >> > > > > > > > > > > Peter Huang
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Tue, Jan 9, 2024 at 5:26 AM Jane Chan <
> > >> > > > > qingyue@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > +1 (non-binding)
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > Jane
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Tue, Jan 9, 2024 at 8:41 PM Lijie Wang <
> > >> > > > > > > > wangdachui9...@gmail.com>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > +1 (non-binding)
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > Lijie
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Jiabao Sun 
> > >> > > > 于2024年1月9日周二
> > >> > > > > > > > 19:28写道：
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > +1 (non-binding)
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > > Jiabao
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > On 2024/01/09 09:58:04 xiangyu feng wrote:
> > >> > > > > > > > > > > > > > > +1 (non-binding)
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Regards,
> > >> > > > > > > > > > > > > > > Xiangyu Feng
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Danny Cranmer 
> 于2024年1月9日周二
> > >> > > > 17:50写道：
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > +1 (binding)
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > > > > Danny
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > On Tue, Jan 9, 2024 at 9:31 AM Feng Jin
> <
> > >> > > > > > ji...@gmail.com>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > +1 (non-binding)
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > > > > > Feng Jin
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > On Tue, Jan 9, 2024 at 5:29 PM Yuxin
> Tan <
> > >> > > > > > > > ta...@gmail.com>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > +1 (non-binding)
> > >> > > > > > > > > > > > > 

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

[Hangxiang Yu]

>
> That's a very good point. I realize that the word 'recovery' means way too
> many things. So I suggest picking a more specific word here, how about
> 'execution.state-recovery.*' ? Checkpointing and state recovery are
> corresponding terms and won't make ambiguity.
>

This makes the configuration clearer to me. We could focus on the
`state-recovery` at first.

I think we could create another FLIP for the deprecation of LEGACY mode.
>

LGTM, Let's create a new FLIP to do this.

IIUC, there is no clear ownership of the local copy files from the previous
> job and it's better to define one. This needs more discussion so we could
> create another thread for this. WDYT?
>

Yeah, I have created a new ticket FLINK-34032 to track and discuss this.

On Wed, Jan 10, 2024 at 6:31 PM Zakelly Lan  wrote:

> Hi everyone,
>
> It seems we still don't have a consensus on the rules for boolean type
> options. Let me recap the alternatives we have:
>
> Option 1: Use enumeration options instead if possible. But this may cause
> some name collisions or confusion as we discussed and we should unify the
> statement everywhere.
> Option 2: Use boolean options and add 'enabled' as the suffix.
> Option 3: Use boolean options and ONLY add 'enabled' when there are more
> detailed configurations under the same prefix, to prevent one name from
> serving as a prefix to another.
>
> I am inclined to Option 3, since it is more in line with current practice
> and friendly for existing users. Also It reduces the length of
> configuration names as much as possible.
>
> Looking forward to your opinions! Thanks!
>
>
> Best,
> Zakelly
>
> On Wed, Jan 10, 2024 at 3:30 PM Zakelly Lan  wrote:
>
> > Hi Hangxiang,
> >
> > Thanks for your suggestions!
> >
> > 1. Could execution.recovery also contain some other behaviors about
> >> recovery ? e.g. restart-strategy.
> >
> >
> > That's a very good point. I realize that the word 'recovery' means way
> too
> > many things. So I suggest picking a more specific word here, how about
> > 'execution.state-recovery.*' ? Checkpointing and state recovery are
> > corresponding terms and won't make ambiguity.
> >
> > 2. Could we also remove some legacy configuration value ? e.g. LEGACY
> Mode
> >> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
> >
> >
> > I think we could create another FLIP for the deprecation of LEGACY mode.
> >
> >
> >> 3. Could the local checkpoint be cleaned
> >> if execution.checkpointing.local-copy.enabled is true and
> >> execution.recovery.from-local is false ? I found it's also an issue if
> >> current local-recovery from enabled to disabled. Maybe another ticket is
> >> needed.
> >
> >
> > IIUC, there is no clear ownership of the local copy files from the
> > previous job and it's better to define one. This needs more discussion so
> > we could create another thread for this. WDYT?
> >
> >
> > Best,
> > Zakelly
> >
> > On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu 
> wrote:
> >
> >> Hi, Zakelly.
> >> Thanks for driving this. Overall LGTM as we discussed offline.
> >>
> >> Some comments/suggestions just came to mind:
> >> 1. Could execution.recovery also contain some other behaviors about
> >> recovery ? e.g. restart-strategy.
> >> 2. Could we also remove some legacy configuration value ? e.g. LEGACY
> Mode
> >> for execution.savepoint-restore-mode/execution.recovery.claim-mode.
> >> 3. Could the local checkpoint be cleaned
> >> if execution.checkpointing.local-copy.enabled is true and
> >> execution.recovery.from-local is false ? I found it's also an issue if
> >> current local-recovery from enabled to disabled. Maybe another ticket is
> >> needed.
> >> 4. +1 for enabling execution.checkpointing.incremental by default which
> is
> >> basically default configuration in our production environment.
> >>
> >>
> >> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan 
> wrote:
> >>
> >> > Hi Yun,
> >> >
> >> > Thanks for your comments!
> >> >
> >> >  1.  We shall not describe the configuration with its implementation
> for
> >> > > 'execution.checkpointing.local-copy.*' options, for hashmap
> >> > state-backend,
> >> > > it would write two streams and for Rocksdb state-backend, it would
> use
> >> > > hard-link for backup. Thus, I think
> >> > > 'execution.checkpointing.local-backup.*' looks better.
> >

[jira] [Created] (FLINK-34051) Fix equals/hashCode/toString for SavepointRestoreSettings

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34051:


 Summary: Fix equals/hashCode/toString for SavepointRestoreSettings
 Key: FLINK-34051
 URL: https://issues.apache.org/jira/browse/FLINK-34051
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.19.0
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


SavepointRestoreSettings#equals/hashCode/toString missed restoreMode property



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34032) Cleanup local=recovery dir when switching local-recovery from enabled to disabled

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34032:


 Summary: Cleanup local=recovery dir when switching local-recovery 
from enabled to disabled
 Key: FLINK-34032
 URL: https://issues.apache.org/jira/browse/FLINK-34032
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Hangxiang Yu


When switching local-recovery from enabled to disabled, the local-recovery dir 
could not be cleaned.

In particular, for a job that switched multiple times, lots of historical local 
checkpoints will be retained forever.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

[Hangxiang Yu]

Hi, Zakelly.
Thanks for driving this. Overall LGTM as we discussed offline.

Some comments/suggestions just came to mind:
1. Could execution.recovery also contain some other behaviors about
recovery ? e.g. restart-strategy.
2. Could we also remove some legacy configuration value ? e.g. LEGACY Mode
for execution.savepoint-restore-mode/execution.recovery.claim-mode.
3. Could the local checkpoint be cleaned
if execution.checkpointing.local-copy.enabled is true and
execution.recovery.from-local is false ? I found it's also an issue if
current local-recovery from enabled to disabled. Maybe another ticket is
needed.
4. +1 for enabling execution.checkpointing.incremental by default which is
basically default configuration in our production environment.


On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan  wrote:

> Hi Yun,
>
> Thanks for your comments!
>
>  1.  We shall not describe the configuration with its implementation for
> > 'execution.checkpointing.local-copy.*' options, for hashmap
> state-backend,
> > it would write two streams and for Rocksdb state-backend, it would use
> > hard-link for backup. Thus, I think
> > 'execution.checkpointing.local-backup.*' looks better.
>
> I agreed that we'd better name the option in user's perspective instead of
> the implementation, thus I name it as a copy of the checkpoint in the
> local disk, regardless of the way of generating it. The word 'backup' is
> also suitable for this case, so I agree to change to
> 'execution.checkpointing.local-backup.*' if no one objects.
>
>  2.  What does the 'execution.checkpointing.data-inline-threshold' mean? It
> > seems not so easy to understand.
>
> The 'execution.checkpointing.data-inline-threshold' (original one as
> 'state.storage.fs.memory-threshold') stands for the size threshold below
> which state chunks will store inline with the metadata, thus I call it
> 'data-inline-threshold'.
>
>
> Best,
> Zakelly
>
> On Mon, Jan 8, 2024 at 10:09 AM Yun Tang  wrote:
>
> > Hi Zakelly,
> >
> > Thanks for driving this topic. I have two concerns here:
> >
> >   1.  We shall not describe the configuration with its implementation for
> > ​'execution.checkpointing.local-copy.*' options, for hashmap
> state-backend,
> > it would write two streams and for Rocksdb state-backend, it would use
> > hard-link for backup​. Thus, I think
> > 'execution.checkpointing.local-backup.*' looks better.
> >   2.  What does the 'execution.checkpointing.data-inline-threshold' mean?
> > It seems not so easy to understand.
> >
> > Best
> > Yun Tang
> > 
> > From: Piotr Nowojski 
> > Sent: Thursday, January 4, 2024 22:37
> > To: dev@flink.apache.org 
> > Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing &
> > Recovery Configuration
> >
> > Hi,
> >
> > Thanks for trying to clean this up! I don't have strong opinions on the
> > topics discussed here, so generally speaking +1 from my side!
> >
> > Best,
> > Piotrek
> >
> > śr., 3 sty 2024 o 04:16 Rui Fan <1996fan...@gmail.com> napisał(a):
> >
> > > Thanks for the feedback!
> > >
> > > Using the `execution.checkpointing.incremental.enabled`,
> > > and enabling it by default sounds good to me.
> > >
> > > Best,
> > > Rui
> > >
> > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan 
> > wrote:
> > >
> > > > Hi Rui,
> > > >
> > > > Thanks for your comments!
> > > >
> > > > IMO, given that the state backend can be plugably loaded (as you can
> > > > specify a state backend factory), I prefer not providing state
> backend
> > > > specified options in the framework.
> > > >
> > > > Secondly, the incremental checkpoint is actually a sharing file
> > strategy
> > > > across checkpoints, which means the state backend *could* reuse files
> > > from
> > > > previous cp but not *must* do so. When the state backend could not
> > reuse
> > > > the files, it is reasonable to fallback to a full checkpoint.
> > > >
> > > > Thus, I suggest we make it `execution.checkpointing.incremental` and
> > > enable
> > > > it by default. For those state backends not supporting this, they
> > perform
> > > > full checkpoints and print a warning to inform users. Users do not
> need
> > > to
> > > > pay special attention to different options to control this across
> > > different
> > > > state backends. This is more user-friendly in my opinion. WDYT?
> > > >
> > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <1996fan...@gmail.com>
> wrote:
> > > >
> > > > > Hi Zakelly,
> > > > >
> > > > > I'm not sure whether we could add the state backend type in the
> > > > > new key name of state.backend.incremental. It means we use
> > > > > `execution.checkpointing.rocksdb-incremental` or
> > > > > `execution.checkpointing.rocksdb-incremental.enabled`.
> > > > >
> > > > > So far, state.backend.incremental only works for rocksdb state
> > backend.
> > > > > And this feature or optimization is very valuable and huge for
> large
> > > > > state flink jobs. I believe it's enabled for most production flink
> > jobs
> > > > > with 

[jira] [Created] (FLINK-34030) Avoid using negative value for periodic-materialize.interval

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34030:


 Summary: Avoid using negative value for 
periodic-materialize.interval
 Key: FLINK-34030
 URL: https://issues.apache.org/jira/browse/FLINK-34030
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


Similar to FLINK-32023, a nagative value doesn't work for Duration Type.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-385: Add OpenTelemetryTraceReporter and OpenTelemetryMetricReporter

[Hangxiang Yu]

+1(binding)

On Wed, Nov 22, 2023 at 10:29 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1(binding)
>
> Best,
> Rui
>
> On Wed, Nov 22, 2023 at 1:20 AM Piotr Nowojski 
> wrote:
>
> > Hi All,
> >
> > I'd like to start a vote on the FLIP-385: Add OpenTelemetryTraceReporter
> > and OpenTelemetryMetricReporter [1]. The discussion thread is here [2].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> or
> > not enough votes.
> >
> > [1] https://cwiki.apache.org/confluence/x/UAuZE
> > [2] https://lists.apache.org/thread/1rqp8czz8wnplpzgn8m4qmzvf14lyx0k
> >
> >
> > Best,
> > Piotrek
> >
>


-- 
Best,
Hangxiang.

Re: [VOTE] FLIP-390: Support System out and err to be redirected to LOG or discarded

[Hangxiang Yu]

+1 (binding)
Thanks for your efforts!

On Mon, Nov 20, 2023 at 11:53 AM Rui Fan <1996fan...@gmail.com> wrote:

> Hi everyone,
>
> Thank you to everyone for the feedback on FLIP-390: Support
> System out and err to be redirected to LOG or discarded[1]
> which has been discussed in this thread [2].
>
> I would like to start a vote for it. The vote will be open for at least 72
> hours unless there is an objection or not enough votes.
>
> [1] https://cwiki.apache.org/confluence/x/4guZE
> [2] https://lists.apache.org/thread/47pdjggh0q0tdkq0cwt6y5o2o8wrl9jl
>
> Best,
> Rui
>


-- 
Best,
Hangxiang.

Re: [VOTE] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

[Hangxiang Yu]

+1 (binding)
Thanks for driving this again!

On Wed, Nov 22, 2023 at 10:30 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1(binding)
>
> Best,
> Rui
>
> On Wed, Nov 22, 2023 at 6:43 AM Jing Ge 
> wrote:
>
> > +1(binding) Thanks!
> >
> > Best regards,
> > Jing
> >
> > On Tue, Nov 21, 2023 at 6:17 PM Piotr Nowojski 
> > wrote:
> >
> > > Hi All,
> > >
> > > I'd like to start a vote on the FLIP-384: Introduce TraceReporter and
> use
> > > it to create checkpointing and recovery traces [1]. The discussion
> thread
> > > is here [2].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection
> > or
> > > not enough votes.
> > >
> > > [1] https://cwiki.apache.org/confluence/x/TguZE
> > > [2] https://lists.apache.org/thread/7lql5f5q1np68fw1wc9trq3d9l2ox8f4
> > >
> > >
> > > Best,
> > > Piotrek
> > >
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

[Hangxiang Yu]

Hi, Rui.
Overall LGTM now.
Thanks for driving this again!

On Wed, Nov 15, 2023 at 2:07 PM Rui Fan <1996fan...@gmail.com> wrote:

> Hi Hangxiang,
>
> Thanks for your feedback!
>
> > I saw some users may debug operator correctness using print() or
> System.out
> > which may bring lots of info.
> > It's fine if files are separate, otherwise these will make
> taskMnanager.log
> > roll quickly.
> > But I think it's fine if we don't use it as default first.
>
> Got it, we will support redirect System.out to LOG, and keep
> original behaviour(DEFAULT enum) as default value.
>
>
> > > Do you mean like this: LOG.info("taskName {} : {}", taskName,
> > > userLogContext)?
> >
> > Yes, That's what I mean.
>
> Sounds make sense, I added a new option to FLIP[1],
> it's taskmanager.system-out.log.thread-name.enabled: false.
>
> Why log the thread name here?
>
> Flink's RichFunction can get subtaskId, but redirecting
> System.out to LOG is public code, so it is difficult to get subtaskId.
> After consideration, a workaround solution is to log thread name,
> because the thread name of the default Task Thread contains the
> task name and subtask id. And this solution can support all threads,
> because some non-task threads may also call println.
>
> [1] https://cwiki.apache.org/confluence/x/4guZE
>
> Best,
> Rui
>
> On Wed, Nov 15, 2023 at 10:26 AM Hangxiang Yu  wrote:
>
> > Hi, Rui.
> >
> > As I mentioned before: the user didn't really want
> > > to log into taskmanager.out, it just happened by accident.
> > > So, if users change the System.out to LOG.info, it still happen.
> >
> > Thanks for the feedback.
> > I saw some users may debug operator correctness using print() or
> System.out
> > which may bring lots of info.
> > It's fine if files are separate, otherwise these will make
> taskMnanager.log
> > roll quickly.
> > But I think it's fine if we don't use it as default first.
> >
> > Do you mean like this: LOG.info("taskName {} : {}", taskName,
> > > userLogContext)?
> > >
> >
> > Yes, That's what I mean.
> >
> > On Sat, Nov 11, 2023 at 12:54 AM Piotr Nowojski <
> piotr.nowoj...@gmail.com>
> > wrote:
> >
> > > Thanks! :)
> > >
> > > Best, Piotrek
> > >
> > > czw., 9 lis 2023 o 16:15 Rui Fan <1996fan...@gmail.com> napisał(a):
> > >
> > > > Hi Piotr,
> > > >
> > > > Thanks for your feedback!
> > > >
> > > > > Or implement your own loop? It shouldn't be more than a couple of
> > > lines.
> > > >
> > > > Implementing it directly is fine, I have updated the FLIP.
> > > > And this logic can be found in the  `isLineEnded` method.
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Thu, Nov 9, 2023 at 11:00 PM Piotr Nowojski <
> > piotr.nowoj...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Rui,
> > > > >
> > > > > > I see java8 doesn't have `Arrays.equals(int[] a, int aFromIndex,
> > int
> > > > > > aToIndex, int[] b, int bFromIndex, int bToIndex)`,
> > > > > > and java11 has it. Do you have any other suggestions for java8?
> > > > >
> > > > > Maybe use `ByteBuffer.wrap`?
> > > > >
> > > > > ByteBuffer.wrap(array, ..., ...).equals(ByteBuffer.wrap(array2,
> ...,
> > > > ...))
> > > > >
> > > > > This shouldn't have overheads as far as I remember.
> > > > >
> > > > > Or implement your own loop? It shouldn't be more than a couple of
> > > lines.
> > > > >
> > > > > Best,
> > > > > Piotrek
> > > > >
> > > > > czw., 9 lis 2023 o 06:43 Rui Fan <1996fan...@gmail.com>
> napisał(a):
> > > > >
> > > > > > Hi Piotr, Archit, Feng and Hangxiang:
> > > > > >
> > > > > > Thanks a lot for your feedback!
> > > > > >
> > > > > > Following is my comment, please correct me if I misunderstood
> > > anything!
> > > > > >
> > > > > > To Piotr:
> > > > > >
> > > > > > > Is there a reason why you are suggesting to copy out bytes from
> > > `buf`
> > > > > to
> > > > > > `bytes`,
> > > >

Re: [DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

[Hangxiang Yu]

find the logs directly in taskmanager.log.
> > > > 2. I'm not sure whether the rolling strategy is easy to implement.
> > > >   If we do it, it's necessary to define a series of flink options
> > similar
> > > >   to log options, such as: fileMax(how many files should be
> retained),
> > > >   fileSize(The max size each file), fileNamePatten (The suffix of
> file
> > > > name),
> > > > 3. Check the file size periodically: all logs are written by log
> > plugin,
> > > >   they can check the log file size after writing. However, System.out
> > > >   are written directly. And flink must start a thread to check the
> > latest
> > > >   taskmanager.out size periodically. If it's too quick, most of job
> > > aren't
> > > >   necessary. If it's too slow, the file size cannot be controlled
> > > properly.
> > > >
> > > > Redirect it to LOG.info may be a reasonable and easy choice.
> > > > The user didn't really want to log into taskmanager.out, it just
> > > > happened by accident.
> > > >
> > > >
> > > >
> > > > To Hangxiang:
> > > >
> > > > > 1. I have a similar concern as Feng. Will we redirect to another
> log
> > > file
> > > > > not taskManager.log ?
> > > >
> > > > Please see my last comment, thanks!
> > > >
> > > > > taskManager.log contains lots of important information like init
> log.
> > > It
> > > > > will be rolled quickly if we redirect out and error here.
> > > >
> > > > IIUC, this issue isn't caused by System.out, and it can happen if
> user
> > > > call a lot of LOG.info. As I mentioned before: the user didn't really
> > > want
> > > > to log into taskmanager.out, it just happened by accident.
> > > > So, if users change the System.out to LOG.info, it still happen.
> > > >
> > > > > 2. Since we have redirected to LOG mode, Could we also log the
> > subtask
> > > > info
> > > > > ? It may help us to debug granularly.
> > > >
> > > > I'm not sure what `log the subtask info` means. Let me confirm with
> you
> > > > first.
> > > > Do you mean like this: LOG.info("taskName {} : {}", taskName,
> > > > userLogContext)?
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Thu, Nov 9, 2023 at 11:47 AM Hangxiang Yu 
> > > wrote:
> > > >
> > > > > Hi, Rui.
> > > > > Thanks for the proposal. It sounds reasonable.
> > > > > I have some questions, PTAL:
> > > > > 1. I have a similar concern as Feng. Will we redirect to another
> log
> > > file
> > > > > not taskManager.log ?
> > > > > taskManager.log contains lots of important information like init
> log.
> > > It
> > > > > will be rolled quickly if we redirect out and error here.
> > > > > 2. Since we have redirected to LOG mode, Could we also log the
> > subtask
> > > > info
> > > > > ? It may help us to debug granularly.
> > > > >
> > > > > On Thu, Nov 9, 2023 at 9:47 AM Feng Jin 
> > wrote:
> > > > >
> > > > > > Hi, Rui.
> > > > > >
> > > > > > Thank you for initiating this proposal.
> > > > > >
> > > > > > I have a question regarding redirecting stdout and stderr to LOG:
> > > > > >
> > > > > > Will they be written to the taskManager.log file by default or
> the
> > > > > > taskManager.out file?
> > > > > > If we can make taskmanager.out splittable and rolling, would it
> be
> > > > easier
> > > > > > for users to use this feature?
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > > On Thu, Nov 9, 2023 at 3:15 AM Archit Goyal
> > > >  > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Rui,
> > > > > > >
> > > > > > > Thanks for the proposal.
> > > > > > >
> > > > > > > The proposed solution of supporting System out and err to be
> > > > redirected
> >

Re: [DISCUSS] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

[Hangxiang Yu]

Hi, Piotr.
Thanks for the proposal.
Just as we discussed in FLINK-23411, +1 for supporting trace/span to
monitor metrics like checkpoint and recovery.

We could also do many things based on this mechanism:
1. more fine-grained metrics about checkpoint and recovery. For example,
some stage info about unaligned checkpoint or generic increment checkpoint
(changelog) which is a bit difficult to report in the current metric system.
2. users could also report their own-defined operator metrics to their own
distributed tracing system which may be traced together with other jobs or
systems.

Look forward to this feature!


On Thu, Nov 9, 2023 at 11:40 AM Zakelly Lan  wrote:

> Hi Piotr,
>
> Thanks for your detailed explanation! I could see the challenge of
> implementing traces with multiple spans and agree to put it in the future
> work. I personally prefer the idea of generating multi span traces for
> checkpoints on the JM only.
>
> > I'm not sure if I understand the proposal - I don't know how traces could
> > be used for this purpose?
> > Traces are perfect for one of events (like checkpointing, recovery, etc),
> > not for continuous monitoring
> > (like processing records). That's what metrics are. Creating trace and
> > span(s) per each record would
> > be prohibitively expensive.
>
> My original thought was to show how much time a sampled record is processed
> within each operator in stream processing. By saying 'sampled', I mean we
> won't generate a trace for every record due to the high cost involved.
> Instead, we could only trace ONE record from source when the user requests
> it (via REST API or Web UI) or when triggered periodically at a very low
> frequency. However after re-thinking my idea, I realized it's hard to
> define the whole lifecycle of a record since it is transformed into
> different forms among operators. We could discuss this in future after the
> basic trace is implemented in Flink.
>
> > Unless you mean in batch/bounded jobs? Then yes, we could create a
> bounded
> > job trace, with spans
> > for every stage/task/subtask.
>
> Oh yes, batch jobs could definitely leverage the trace.
>
> Best,
> Zakelly
>
>
> On Wed, Nov 8, 2023 at 9:18 PM Jinzhong Li 
> wrote:
>
> > Hi Piotr,
> >
> > Thanks for driving this proposal!   I strongly agree that the existing
> > metric APIs are not suitable for monitoring restore/checkpoint behavior!
> >
> > I think the TM-level recovery/checkpointing traces are necessary in the
> > future. In our production environment, we sometimes encounter that job
> > recovery time is very long (30min+), due to several subTask heavy disk
> > traffic. The TM-level recovery trace is helpful for troubleshooting such
> > issues.
> >
> > Best
> > Jinzhong
> >
> > On Wed, Nov 8, 2023 at 5:09 PM Piotr Nowojski 
> > wrote:
> >
> > > Hi Zakelly,
> > >
> > > Thanks for the comments. Quick answer for both of your questions would
> be
> > > that it probably should be
> > > left as a future work. For more detailed answers please take a look
> below
> > > :)
> > >
> > > > Does it mean the inclusion and subdivision relationships of spans
> > defined
> > > > by "parent_id" are not supported? I think it is a very necessary
> > feature
> > > > for the trace.
> > >
> > > Yes exactly, that is the current limitation. This could be solved
> somehow
> > > one way or another in the future.
> > >
> > > Support for reporting multi span traces all at once - for example
> > > `CheckpointStatsTracker` running JM,
> > > could upon checkpoint completion create in one place the whole
> structure
> > of
> > > parent spans, to have for
> > > example one span per each subtask. This would be a relatively easy
> follow
> > > up.
> > >
> > > However, if we would like to create true distributed traces, with spans
> > > reported from many different
> > > components, potentially both on JM and TM, the problem is a bit deeper.
> > The
> > > issue in that case is how
> > > to actually fill out `parrent_id` and `trace_id`? Passing some context
> > > entity as a java object would be
> > > unfeasible. That would require too many changes in too many places. I
> > think
> > > the only realistic way
> > > to do it, would be to have a deterministic generator of `parten_id` and
> > > `trace_id` values.
> > >
> > > For example we could create the parent trace/span of the checkpoint on
> > JM,
> > > and set those ids to
> > > something like: `jobId#attemptId#checkpointId`. Each subtask then could
> > > re-generate those ids
> > > and subtasks' checkpoint span would have an id of
> > > `jobId#attemptId#checkpointId#subTaskId`.
> > > Note that this is just an example, as most likely distributed spans for
> > > checkpointing do not make
> > > sense, as we can generate them much easier on the JM anyway.
> > >
> > > > In addition to checkpoint and recovery, I believe the trace would
> also
> > be
> > > > valuable for performance tuning. If Flink can trace and visualize the
> > > time
> > > > cost of each operator and stage 

Re: [DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

[Hangxiang Yu]

Hi, Rui.
Thanks for the proposal. It sounds reasonable.
I have some questions, PTAL:
1. I have a similar concern as Feng. Will we redirect to another log file
not taskManager.log ?
taskManager.log contains lots of important information like init log. It
will be rolled quickly if we redirect out and error here.
2. Since we have redirected to LOG mode, Could we also log the subtask info
? It may help us to debug granularly.

On Thu, Nov 9, 2023 at 9:47 AM Feng Jin  wrote:

> Hi, Rui.
>
> Thank you for initiating this proposal.
>
> I have a question regarding redirecting stdout and stderr to LOG:
>
> Will they be written to the taskManager.log file by default or the
> taskManager.out file?
> If we can make taskmanager.out splittable and rolling, would it be easier
> for users to use this feature?
>
> Best,
> Feng
>
> On Thu, Nov 9, 2023 at 3:15 AM Archit Goyal 
> wrote:
>
> > Hi Rui,
> >
> > Thanks for the proposal.
> >
> > The proposed solution of supporting System out and err to be redirected
> to
> > LOG or discarded and introducing an enum and two options to manage this,
> > seems reasonable.
> >
> > +1
> >
> > Thanks,
> > Archit Goyal
> >
> >
> > From: Piotr Nowojski 
> > Date: Wednesday, November 8, 2023 at 7:38 AM
> > To: dev@flink.apache.org 
> > Subject: Re: [DISCUSS] FLIP-390: Support System out and err to be
> > redirected to LOG or discarded
> > Hi Rui,
> >
> > Thanks for the proposal.
> >
> > +1 I don't have any major comments :)
> >
> > One nit. In `SystemOutRedirectToLog` in this code:
> >
> >System.arraycopy(buf, count - LINE_SEPARATOR_LENGTH, bytes, 0,
> > LINE_SEPARATOR_LENGTH);
> > return Arrays.equals(LINE_SEPARATOR_BYTES, bytes)
> >
> > Is there a reason why you are suggesting to copy out bytes from `buf` to
> > `bytes`,
> > instead of using `Arrays.equals(int[] a, int aFromIndex, int aToIndex,
> > int[] b, int bFromIndex, int bToIndex)`?
> >
> > Best,
> > Piotrek
> >
> > śr., 8 lis 2023 o 11:53 Rui Fan <1996fan...@gmail.com> napisał(a):
> >
> > > Hi all!
> > >
> > > I would like to start a discussion of FLIP-390: Support System out and
> > err
> > > to be redirected to LOG or discarded[1].
> > >
> > > In various production environments, either cloud native or physical
> > > machines, the disk space that Flink TaskManager can use is limited.
> > >
> > > In general, the flink users shouldn't use the `System.out.println` in
> > > production,
> > > however this may happen when the number of Flink jobs and job
> developers
> > > is very large. Flink job may use System.out to output a large amount of
> > > data
> > > to the taskmanager.out file. This file will not roll, it will always
> > > increment.
> > > Eventually the upper limit of what the TM can be used for is reached.
> > >
> > > We can support System out and err to be redirected to LOG or discarded,
> > > the LOG can roll and won't increment forever.
> > >
> > > This feature is useful for SREs who maintain Flink environments, they
> can
> > > redirect System.out to LOG by default. Although the cause of this
> problem
> > > is
> > > that the user's code is not standardized, for SRE, pushing users to
> > modify
> > > the code one by one is usually a very time-consuming operation. It's
> also
> > > useful for job stability where System.out is accidentally used.
> > >
> > > Looking forward to your feedback, thanks~
> > >
> > > [1]
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fx%2F4guZE=05%7C01%7Cargoyal%40linkedin.com%7C937821de7bd846e6b97408dbe070beae%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C638350547072823674%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=zEv6B0Xiq2SNuU6Fm%2BAXnH%2BRILbm6Q0uDRbN7h6iaPM%3D=0
> > 
> > >
> > > Best,
> > > Rui
> > >
> >
>


-- 
Best,
Hangxiang.

Re: [ANNOUNCE] The Flink Speed Center and benchmark daily run are back online

[Hangxiang Yu]

Thanks Zakelly for the great work!


On Fri, Oct 20, 2023 at 5:05 PM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks for your effort! It's very useful when some new commits affect
> performance.
>
> Best,
> Rui
>
> On Fri, Oct 20, 2023 at 4:42 PM Yanfei Lei  wrote:
>
> > Thanks for your hard work!
> > Looking forward to the daily monitoring being available again soon.
> >
> > Best,
> > Yanfei
> >
> > Yuan Mei  于2023年10月20日周五 16:19写道：
> > >
> > > Thank you for your great efforts!
> > >
> > > Best
> > > Yuan
> > >
> > > On Fri, Oct 20, 2023 at 4:08 PM Sergey Nuyanzin 
> > wrote:
> > >
> > > > Thanks a lot for working on this!
> > > >
> > > > On Fri, Oct 20, 2023 at 9:27 AM Yangze Guo 
> wrote:
> > > >
> > > > > Thanks for the effort, Zhaoqian!
> > > > >
> > > > > Best,
> > > > > Yangze Guo
> > > > >
> > > > > On Fri, Oct 20, 2023 at 2:55 PM Leonard Xu 
> > wrote:
> > > > > >
> > > > > > Thanks Zakelly for the great work.
> > > > > >
> > > > > > Best,
> > > > > > Leonard
> > > > > >
> > > > > > > 2023年10月19日 下午7:39，Jing Ge  写道：
> > > > > > >
> > > > > > > Hi Zakelly,
> > > > > > >
> > > > > > > Thank you for your effort! Really appreciate it!
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > On Thu, Oct 19, 2023 at 12:02 PM Yun Tang 
> > wrote:
> > > > > > >
> > > > > > >> Thanks for Zakelly's great work!
> > > > > > >>
> > > > > > >>
> > > > > > >> Best
> > > > > > >> Yun Tang
> > > > > > >> 
> > > > > > >> From: Piotr Nowojski 
> > > > > > >> Sent: Thursday, October 19, 2023 17:56
> > > > > > >> To: dev@flink.apache.org 
> > > > > > >> Subject: Re: [ANNOUNCE] The Flink Speed Center and benchmark
> > daily
> > > > > run are
> > > > > > >> back online
> > > > > > >>
> > > > > > >> Thank you!
> > > > > > >>
> > > > > > >> czw., 19 paź 2023 o 11:31 Konstantin Knauf  >
> > > > > napisał(a):
> > > > > > >>
> > > > > > >>> Thanks a lot for working on this!
> > > > > > >>>
> > > > > > >>> Am Do., 19. Okt. 2023 um 10:24 Uhr schrieb Zakelly Lan <
> > > > > > >>> zakelly@gmail.com>:
> > > > > > >>>
> > > > > >  Hi everyone,
> > > > > > 
> > > > > >  Flink benchmarks [1] generate daily performance reports in
> the
> > > > > Apache
> > > > > >  Flink slack channel (#flink-dev-benchmarks) to detect
> > performance
> > > > > >  regression [2]. Those benchmarks previously were running on
> > > > several
> > > > > >  machines donated and maintained by Ververica. Unfortunately,
> > those
> > > > > >  machines were gone due to account issues [3] and the
> > benchmarks
> > > > > daily
> > > > > >  run stopped since August 24th delaying the release of Flink
> > 1.18 a
> > > > > >  bit. [4].
> > > > > > 
> > > > > >  Ververica donated several new machines! After several weeks
> of
> > > > > work, I
> > > > > >  have successfully re-established the codespeed panel and
> > benchmark
> > > > > >  daily run pipelines on them. At this time, we are pleased to
> > > > > announce
> > > > > >  that the Flink Speed Center and benchmark pipelines are back
> > > > online.
> > > > > >  These new machines have a more formal management to ensure
> > that
> > > > > >  previous accidents will not occur in the future.
> > > > > > 
> > > > > >  What's more, I successfully recovered historical data backed
> > up by
> > > > > >  Yanfei Lei [5]. So with the old domain [6] redirected to the
> > new
> > > > > >  machines, the old links that existed in previous records
> will
> > > > still
> > > > > be
> > > > > >  valid. Besides the benchmarks with Java8 and Java11, I also
> > added
> > > > a
> > > > > >  pipeline for Java17 running daily.
> > > > > > 
> > > > > >  How to use it:
> > > > > >  We also registered a new domain name 'flink-speed.xyz' for
> > the
> > > > > Flink
> > > > > >  Speed Center [7]. It is recommended to use the new domain in
> > the
> > > > > >  future. Currently, the self-service method of triggering
> > > > benchmarks
> > > > > is
> > > > > >  unavailable considering the lack of resources and potential
> > > > > >  vulnerabilities of Jenkins. Please contact one of Apache
> Flink
> > > > PMCs
> > > > > to
> > > > > >  submit a benchmark. More info is updated on the wiki[8].
> > > > > > 
> > > > > >  Daily Monitoring:
> > > > > >  The performance daily monitoring on the Apache Flink slack
> > channel
> > > > > [2]
> > > > > >  is still unavailable as the benchmark results need more time
> > to
> > > > > >  stabilize in the new environment. Once the baseline results
> > become
> > > > > >  available for regression detection, I will enable the daily
> > > > > >  monitoring.
> > > > > > 
> > > > > >  Please feel free to reach out to me if you have any
> > suggestions or
> > > > > >  questions. Thanks Ververica again for denoting machines!
> > > > > > 
> > > > > > 
> > > > > >  Best,
> > > > 

Re: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

[Hangxiang Yu]

Hi, Zakelly.

Thanks for the proposal.

+1 for reorganizing exceptions of state interfaces which indeed confuses me
currently.

>From my experience, users usually omit these exceptions because they cannot
do much even if they catch the exceptions.

I have some problems and suggestions, PTAL:

   1. Could we also reorganize or add more state exceptions (may be related
   to other state interfaces/classes e.g. KeyedStateBackend) into the
   exception class diagrams ? Although these state-related classes may not
   be public, it could be better to consider them together to make all
   state-related exceptions clear. For example, we could reorganize some
   existing exceptions such as StateMigrationException, add some exceptions
   such as StateNotFoundException.
   2. Could you clarify or give an example about the extended relation
   "StateAccessException -- StateIOException" ? When do we just return
   StateAccessException instead of StateIOException or others ?
   3. Which version do you want to implement it in ? Since it has to break
   changes for users who have catched the IOException, If we want to implement
   it in 1.19, we must mark it very clearly in the release note, or we should
   make it in 2.0.


On Tue, Sep 19, 2023 at 5:08 PM Zakelly Lan  wrote:

> Hi everyone,
>
> I would like to initiate a discussion on FLIP-368, which focuses on
> reorganizing the exceptions thrown in state interfaces [1].
>
> Currently, we have identified several problems with the exceptions
> thrown by state-related interfaces:
>   1. The exception types thrown by each interface are inconsistent.
> While most of the interfaces claim to throw `Exception`, the
> interfaces of `ValueState` throw `IOException`. Additionally, the
> `State#clear()` never throws an exception. This can be confusing for
> users.
>   2. The use of `Exception` or `IOException` as the thrown exception
> type is too generic and lacks specificity.
>   3. Users may not be able to handle these exceptions. In cases where
> an exception occurs while accessing state, the job should fail. This
> aligns more with the characteristic of *unchecked exceptions* instead
> of checked exceptions.
>
> To address these issues, we borrow the idea of throwing unchecked
> exceptions in Java Collection API and propose the following changes in
> state-related exceptions:
>   1. Introduction of specific unchecked exception types for different
> reasons, providing users with more precise information about the cause
> of the exception.
>   2. Removal of all checked exceptions from interface signatures and
> instead, throwing newly introduced unchecked exceptions in the
> implementations.
>
> Please share your thoughts and suggestions regarding the proposed
> changes. Thank you for your attention and support.
>
>
> Best,
> Zakelly
>
> [1] FLIP-368: Reorganize the exceptions thrown in state interfaces,
> https://cwiki.apache.org/confluence/x/eZ2zDw
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-33055) Correct the error value about 'state.backend.type' in the document

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-33055:


 Summary: Correct the error value about 'state.backend.type' in the 
document
 Key: FLINK-33055
 URL: https://issues.apache.org/jira/browse/FLINK-33055
 Project: Flink
  Issue Type: Bug
  Components: Documentation, Runtime / State Backends
Reporter: Hangxiang Yu


{{}}
{code:java}
state.backend.type: The state backend to use. This defines the data structure 
mechanism for taking snapshots. Common values are filesystem or rocksdb{code}
filesystem should be replaced with hashmap after FLINK-16444.

{{}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Add config to enable job stop with savepoint on exceeding tolerable checkpoint Failures

[Hangxiang Yu]

Hi, Dongwoo.
IIUC, you mean using savepoint to store a snapshot to other storage if
checkpoints fail multiple times due to some long lasting exceptions of
external storage, right ?
I think it's better to achieve this by an external tool instead of
introducing a config like that:
1. it's not so easy to judge whether an exception occurs due to external
storage or not sometimes, and it's not so reasonable that we just trigger a
savepoint if checkpoints fail multiple times.
2. It's better to let some logic about triggering savepoint, e.g. periodic
savepoint, triggering stop-with-savepoint, done by external tools or
platform. As you could see from [1], we intend to make their scopes clear.

Maybe you could check the status and failure message by [2] periodically in
your external tool or platform and then trigger savepoint or
stop-with-savepoint by REST API or CLI.

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/checkpoints_vs_savepoints/
[2]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/#jobs-jobid-checkpoints

On Wed, Sep 6, 2023 at 11:05 AM Yanfei Lei  wrote:

> Hi Dongwoo,
>
> If the checkpoint has failed
> `execution.checkpointing.tolerable-failed-checkpoints` times, then
> stopWithSavepoint is likely to fail as well.
> If stopWithSavepoint succeeds or fails, will the job just stop?  I am
> more curious about how this option works with the restart strategy?
>
> Best,
> Yanfei
>
>
> Dongwoo Kim  于2023年9月4日周一 22:17写道：
> >
> > Hi all,
> > I have a proposal that aims to enhance the flink application's
> resilience in cases of unexpected failures in checkpoint storages like S3
> or HDFS,
> >
> > [Background]
> > When using self managed S3-compatible object storage, we faced
> checkpoint async failures lasting for an extended period more than 30
> minutes,
> > leading to multiple job restarts and causing lags in our streaming
> application.
> >
> > [Current Behavior]
> > Currently, when the number of checkpoint failures exceeds a predefined
> tolerable limit, flink will either restart or fail the job based on how
> it's configured.
> > In my opinion this does not handle scenarios where the checkpoint
> storage itself may be unreliable or experiencing downtime.
> >
> > [Proposed Feature]
> > I propose a config that allows for a graceful job stop with a savepoint
> when the tolerable checkpoint failure limit is reached.
> > Instead of restarting/failing the job when tolerable checkpoint failure
> exceeds, when this new config is set to true just trigger stopWithSavepoint.
> >
> > This could offer the following benefits.
> > - Indication of Checkpoint Storage State: Exceeding tolerable checkpoint
> failures could indicate unstable checkpoint storage.
> > - Automated Fallback Strategy: When combined with a monitoring cron job,
> this feature could act as an automated fallback strategy for handling
> unstable checkpoint storage.
> >   The job would stop safely, take a savepoint, and then you could
> automatically restart with different checkpoint storage configured like
> switching from S3 to HDFS.
> >
> > For example let's say checkpoint path is configured to s3 and savepoint
> path is configured to hdfs.
> > When the new config is set to true the job stops with savepoint like
> below when tolerable checkpoint failure exceeds.
> > And we can restart the job from that savepoint while the checkpoint
> configured as hdfs.
> >
> >
> >
> > Looking forward to hearing the community's thoughts on this proposal.
> > And also want to ask how the community is handling long lasting unstable
> checkpoint storage issues.
> >
> > Thanks in advance.
> >
> > Best dongwoo,
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-32601) Unstable RemoteChannelThroughputBenchmark_remoteRebalance_jmhTest

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-32601:


 Summary: Unstable 
RemoteChannelThroughputBenchmark_remoteRebalance_jmhTest
 Key: FLINK-32601
 URL: https://issues.apache.org/jira/browse/FLINK-32601
 Project: Flink
  Issue Type: Bug
  Components: Benchmarks
Reporter: Hangxiang Yu


It's an exising exception which may occur accidentally, see workflow 
[#74|https://github.com/apache/flink-benchmarks/actions/runs/5219158886/jobs/9450453549#logs],
 
[#40,|https://github.com/apache/flink-benchmarks/actions/runs/4915916989/jobs/8779014239],
 
[#75,|https://github.com/apache/flink-benchmarks/actions/runs/5527523559/jobs/10171425495]

Exception stack as below:
{code:java}

2310ERROR: org.openjdk.jmh.runner.RunnerException: Benchmark caught the 
exception
2311Benchmark had encountered error, and fail on error was requested
2312at org.openjdk.jmh.runner.Runner.runBenchmarks(Runner.java:570)
2313at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:313)
2314at org.openjdk.jmh.runner.Runner.run(Runner.java:206)
2315at org.openjdk.jmh.Main.main(Main.java:71)
2316Caused by: org.openjdk.jmh.runner.BenchmarkException: Benchmark error 
during the run
2317at 
org.openjdk.jmh.runner.BenchmarkHandler.runIteration(BenchmarkHandler.java:428)
2318at org.openjdk.jmh.runner.BaseRunner.runBenchmark(BaseRunner.java:282)
2319at org.openjdk.jmh.runner.BaseRunner.runBenchmark(BaseRunner.java:234)
2320at org.openjdk.jmh.runner.BaseRunner.doSingle(BaseRunner.java:139)
2321at 
org.openjdk.jmh.runner.BaseRunner.runBenchmarksForked(BaseRunner.java:76)
2322at org.openjdk.jmh.runner.ForkedRunner.run(ForkedRunner.java:72)
2323at org.openjdk.jmh.runner.ForkedMain.main(ForkedMain.java:84)
2324Suppressed: org.apache.flink.runtime.client.JobExecutionException: Job 
execution failed.
2325at 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2326at 
org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:1010)
2327at 
org.apache.flink.benchmark.RemoteChannelThroughputBenchmark.remoteRebalance(RemoteChannelThroughputBenchmark.java:76)
2328at 
org.apache.flink.benchmark.generated.RemoteChannelThroughputBenchmark_remoteRebalance_jmhTest.remoteRebalance_thrpt_jmhStub(RemoteChannelThroughputBenchmark_remoteRebalance_jmhTest.java:123)
2329at 
org.apache.flink.benchmark.generated.RemoteChannelThroughputBenchmark_remoteRebalance_jmhTest.remoteRebalance_Throughput(RemoteChannelThroughputBenchmark_remoteRebalance_jmhTest.java:85)
2330at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2331at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2332at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2333at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2334at 
org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
2335at 
org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
2336at 
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2337at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2338at 
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2339at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2340at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2341at java.base/java.lang.Thread.run(Thread.java:829)
2342Caused by: org.apache.flink.runtime.JobException: Recovery is 
suppressed by NoRestartBackoffTimeStrategy
2343at 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:176)
2344at 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getGlobalFailureHandlingResult(ExecutionFailureHandler.java:126)
2345at 
org.apache.flink.runtime.scheduler.DefaultScheduler.handleGlobalFailure(DefaultScheduler.java:328)
2346at 
org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyGlobalFailure(UpdateSchedulerNgOnInternalFailuresListener.java:57)
2347at 
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.failGlobal(DefaultExecutionGraph.java:1073)
2348at 
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.failGlobalIfExecutionIsStillRunning(DefaultExecutionGraph.java:1061)
2349at 
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph$1.lambda

[jira] [Created] (FLINK-32364) Add Rescaling benchmark for ChangelogStateBackend

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-32364:


 Summary: Add Rescaling benchmark for ChangelogStateBackend
 Key: FLINK-32364
 URL: https://issues.apache.org/jira/browse/FLINK-32364
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


After FLINK-23484, we could Supports rescaling benchmark just like HEAP and 
ROCKSDB.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints

[Hangxiang Yu]

Hi Zakelly.
Thanks for driving this.
+1 (no-binding)

On Wed, May 10, 2023 at 10:52 AM Yuan Mei  wrote:

> Thanks for driving this, Zakelly.
>
> As discussed in the thread,
>
> +1 for the proposal (binding)
>
> Best,
>
> Yuan
>
>
>
> On Wed, May 10, 2023 at 10:39 AM Zakelly Lan 
> wrote:
>
> > Hi everyone,
> >
> > Sorry for the 4 duplicate emails. There was a problem with the dev
> > mailing list blocking the mails from Gmail. I thought it was a network
> > problem so I tried several times. The issue is addressed by
> > INFRA-24572[1] and the piled up mails are delivered at one time.
> >
> > Based on the sending time, the vote will be open until May 12th at
> > 11:00PM GMT. Please discuss and vote in the last thread (this one).
> > Thanks!
> >
> >
> > Best regards,
> > Zakelly
> >
> > [1] https://issues.apache.org/jira/browse/INFRA-24572
> >
> > On Wed, May 10, 2023 at 10:30 AM Yanfei Lei  wrote:
> > >
> > > +1 (no-binding)
> > >
> > > Best,
> > > Yanfei
> > >
> > >
> > > Jing Ge  于2023年5月10日周三 07:03写道：
> > >
> > > >
> > > > Hi Zakelly,
> > > >
> > > > I saw you sent at least 4 same emails for voting FLIP-306. I guess
> > this one
> > > > should be the last one and the right one for us to vote right? BTW,
> > based
> > > > on the sending time, 72 hours means to open the discussion until May
> > 12th.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Tue, May 9, 2023 at 8:24 PM Zakelly Lan 
> > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Thanks for all the feedback for FLIP-306: Unified File Merging
> > > > > Mechanism for Checkpoints[1] on the discussion thread[2].
> > > > >
> > > > > I'd like to start a vote for it. The vote will be open for at least
> > 72
> > > > > hours (until May 11th, 12:00AM GMT) unless there is an objection or
> > an
> > > > > insufficient number of votes.
> > > > >
> > > > >
> > > > > Best,
> > > > > Zakelly
> > > > >
> > > > > [1]
> > > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints
> > > > > [2]
> https://lists.apache.org/thread/56px3kvn3tlwpc7sl12kx6notfmk9g7q
> > > > >
> >
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-31875) OSS throwns NoClassDefFoundError due to old hadoop-common version

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-31875:


 Summary: OSS throwns NoClassDefFoundError due to old hadoop-common 
version
 Key: FLINK-31875
 URL: https://issues.apache.org/jira/browse/FLINK-31875
 Project: Flink
  Issue Type: Bug
  Components: FileSystems
Affects Versions: 1.17.0
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu
 Fix For: 1.17.1


h2. Problem

When using OSS in 1.17, an exception will be thrown:
{code:java}
java.lang.NoClassDefFoundError: 
org/apache/hadoop/thirdparty/com/google/common/base/Preconditions

at 
org.apache.hadoop.fs.aliyun.oss.AliyunOSSUtils.longOption(AliyunOSSUtils.java:221)
at 
org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:343)
at 
org.apache.flink.fs.osshadoop.OSSFileSystemFactory.create(OSSFileSystemFactory.java:147)
at 
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:508)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274){code}
h2. Why


After https://issues.apache.org/jira/browse/FLINK-27308 and  
https://issues.apache.org/jira/browse/FLINK-29502 ，hadoop-aliyun has also be 
upgraded to 3.3.4 which relys on the newest version of hadoop-common.

OSS still uses the old version (2.10.2) extended from flink-parent so that some 
classes cannot be found.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31366) Exception is thrown when s3a and s3p are used together

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-31366:


 Summary: Exception is thrown when s3a and s3p are used together
 Key: FLINK-31366
 URL: https://issues.apache.org/jira/browse/FLINK-31366
 Project: Flink
  Issue Type: Bug
  Components: FileSystems
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


h3. Exception

When s3a and s3p plugins exist at the same time, an exception will be thrown as 
below:
{code:java}
Caused by:java.lang.NoSuchMethodError: 
org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory. 
(Ljava/lang/String;Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader;)
{code}
h3. Why

In the construction method of AbstractS3FileSystemFactory, s3a shades 
HadoopConfigLoader into org.apache.flink.fs.s3hadoop.common.HadoopConfigLoader, 
and s3p shades HadoopConfigLoader into org. 
apache.flink.fs.s3presto.common.HadoopConfigLoader when package.

(see shade-plugin)

 

But the AbstractS3FileSystemFactory class will only be loaded once when loading 
even if there are two plugins.

So it may first uses s3a plugin to load AbstractS3FileSystemFactory, and the 
construction method is loaded into AbstractS3FileSystemFactory(String name, 
org.apache. flink.fs.s3hadoop.common.HadoopConfigLoader hadoopConfigLoader), at 
this time s3p will be abnormal when it is constructed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30854) Expose periodic compaction to RocksdbCompactFilterCleanupStrategy

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-30854:


 Summary: Expose periodic compaction to 
RocksdbCompactFilterCleanupStrategy
 Key: FLINK-30854
 URL: https://issues.apache.org/jira/browse/FLINK-30854
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / State Backends
Reporter: Hangxiang Yu


In 6.20.3-ververica-2.0 rocksdb version flink uses from 1.17[1], we introduce 
periodic compaction option[2].

it deserves to expose this into RocksdbCompactFilterCleanupStrategy or some 
confs to make users use it conveniently.

cc [~yunta] 

[1] [https://github.com/apache/flink/pull/21747]
[2] https://github.com/ververica/frocksdb/pull/57



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30656) Provide more logs for schema compatibility check

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-30656:


 Summary: Provide more logs for schema compatibility check
 Key: FLINK-30656
 URL: https://issues.apache.org/jira/browse/FLINK-30656
 Project: Flink
  Issue Type: Improvement
  Components: API / Type Serialization System
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


Currently, we have very few logs and exception info when checking schema 
compatibility.

It's difficult to see why the compatibility is not compatible, especially for 
some complicated nested serializers.

For example, for map serializer, when it's not compatible, we may only see 
below without other information:
{code:java}
Caused by: org.apache.flink.util.StateMigrationException: The new state 
serializer (org.apache.flink.api.common.typeutils.base.MapSerializer@e95e076a) 
must not be incompatible with the old state serializer 
(org.apache.flink.api.common.typeutils.base.MapSerializer@c33b100f). {code}
So I think we could add more infos when checking the compatibility.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee

[Hangxiang Yu]

Congratulations, Lincoln!

On Tue, Jan 10, 2023 at 1:34 PM Dian Fu  wrote:

> Congratulations, Lincoln!
>
> Regards,
> Dian
>
> On Tue, Jan 10, 2023 at 1:31 PM weijie guo 
> wrote:
>
> > Congratulations, Lincoln!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Lijie Wang  于2023年1月10日周二 12:24写道：
> >
> > > Congratulations, Lincoln!
> > >
> > > Best,
> > > Lijie
> > >
> > > Jingsong Li  于2023年1月10日周二 12:07写道：
> > >
> > > > Congratulations, Lincoln!
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Tue, Jan 10, 2023 at 11:56 AM Leonard Xu 
> wrote:
> > > > >
> > > > > Congratulations, Lincoln!
> > > > >
> > > > > Impressive work in streaming semantics, well deserved!
> > > > >
> > > > >
> > > > > Best,
> > > > > Leonard
> > > > >
> > > > >
> > > > > > On Jan 10, 2023, at 11:52 AM, Jark Wu  wrote:
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a
> > new
> > > > Flink
> > > > > > committer.
> > > > > >
> > > > > > Lincoln Lee has been a long-term Flink contributor since 2017. He
> > > > mainly
> > > > > > works on Flink
> > > > > > SQL parts and drives several important FLIPs, e.g., FLIP-232
> (Retry
> > > > Async
> > > > > > I/O), FLIP-234 (
> > > > > > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides,
> > He
> > > > also
> > > > > > contributed
> > > > > > much to Streaming Semantics, including the non-determinism
> problem
> > > and
> > > > the
> > > > > > message
> > > > > > ordering problem.
> > > > > >
> > > > > > Please join me in congratulating Lincoln for becoming a Flink
> > > > committer!
> > > > > >
> > > > > > Cheers,
> > > > > > Jark Wu
> > > > >
> > > >
> > >
> >
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-30614) Improve resolving schema compatibility -- Milestone two

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-30614:


 Summary: Improve resolving schema compatibility -- Milestone two
 Key: FLINK-30614
 URL: https://issues.apache.org/jira/browse/FLINK-30614
 Project: Flink
  Issue Type: Sub-task
  Components: API / Type Serialization System
Reporter: Hangxiang Yu


In the milestone two, we should:
 # Remove TypeSerializerSnapshot#resolveSchemaCompatibility(TypeSerializer 
newSerializer) and related implementation.
 # Make all places where use 
TypeSerializerSnapshot#resolveSchemaCompatibility(TypeSerializer 
newSerializer) to check the compatibility call 
Typeserializer#resolveSchemaCompatibility(TypeSerializerSnapshot 
oldSerializerSnapshot). 
 # Remove the default implementation of the new method.

It will be done after several stable version.

See FLIP-263 for more details.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30613) Improve resolving schema compatibility -- Milestone one

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-30613:


 Summary: Improve resolving schema compatibility -- Milestone one
 Key: FLINK-30613
 URL: https://issues.apache.org/jira/browse/FLINK-30613
 Project: Flink
  Issue Type: Sub-task
  Components: API / Type Serialization System
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu
 Fix For: 1.17.0


In the milestone one, we should:
 # Add an extra method 
(TypeserializeSnapshotr#resolveSchemaCompatibility(TypeSerializerSnapshot 
oldSerializerSnapshot)) in TypeSerializerSnapshot.java as above, and return 
INCOMPATIBLE as default.
 # Mark the original method as deprecated and it will use new method to resolve 
as default.
 # Implement the new method for all built-in TypeserializerSnapshots.

See FLIP-263 for more details.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30450) FileSystem supports exporting client-side metrics

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-30450:


 Summary: FileSystem supports exporting client-side metrics
 Key: FLINK-30450
 URL: https://issues.apache.org/jira/browse/FLINK-30450
 Project: Flink
  Issue Type: New Feature
  Components: FileSystems
Reporter: Hangxiang Yu


Client-side metrics, or job level metrics for filesystem could help us to 
monitor filesystem more precisely.

Some metrics (like request rate , throughput, latency, retry count, etc) are 
useful to monitor the network or client problem of checkpointing or or other 
access cases for a job.  
Some filesystems like s3, s3-presto, gs have supported enabling some metrics, 
these could be exported in filesystem.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30345) Improve the serializer performace of state change of changelog

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-30345:


 Summary: Improve the serializer performace of state change of 
changelog
 Key: FLINK-30345
 URL: https://issues.apache.org/jira/browse/FLINK-30345
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / State Backends
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu
 Fix For: 1.17.0, 1.16.1


Currently, AbstractStateChangeLogger use sync 

DataOutputViewStreamWrapper to serialize state change which is unnecessary 
because it will always be executed in single thread.

So replace it with a unsync one could improve the performance of serialization.

In my simple stateful WordCount case, it could improve TPS by 10% at least.

Furthermore, because the serialization and deserialization of key and value 
have been executed in some delegaed state backend, maybe we could avoid double 
serialization. It may improve the performance if the serialization logic is 
complex and even is the bottleneck.

This ticket focuses on the sync serializer problem.
The second problem about double serialization could also be disscussed, and I 
will create a new ticket if necessary.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He, Xingbo Huang

[Hangxiang Yu]

Congratulations, Godfrey and Xingbo!

On Thu, Nov 24, 2022 at 4:12 PM Jingsong Li  wrote:

> Congrats, Godfrey and Xingbo!
>
> Best,
> Jingsong
>
> On Thu, Nov 24, 2022 at 4:03 PM Sergey Nuyanzin 
> wrote:
> >
> > Congrats to both!
> >
> > On Thu, Nov 24, 2022 at 8:59 AM Yun Gao 
> > wrote:
> >
> > > Congratulations Godfrey and Xingbo!
> > > Best,
> > > Yun Gao
> > > --
> > > From:Matthias Pohl 
> > > Send Time:2022 Nov. 24 (Thu.) 12:25
> > > To:dev 
> > > Subject:Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He,
> Xingbo
> > > Huang
> > > Congratulations to the two of you!
> > > Best,
> > > Matthias
> > > On Thu, Nov 24, 2022 at 11:29 AM Shengkai Fang 
> wrote:
> > > > Congratulations, Godfrey and Xingbo!
> > > >
> > > > Best,
> > > > Shengkai
> > > >
> > > > Shuo Cheng  于2022年11月24日周四 10:56写道：
> > > >
> > > > > Congratulations, Godfrey and Xingbo!
> > > > >
> > > > > Best,
> > > > > Shuo
> > > > >
> > > > > On Wed, Nov 23, 2022 at 12:19 PM Dian Fu 
> > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > On behalf of the Apache Flink PMC, I'm very happy to announce
> that
> > > > > Godfrey
> > > > > > He and Xingbo Huang have joined the Flink PMC!
> > > > > >
> > > > > > Godfrey He becomes a Flink committer in Sep 2020. His
> contributions
> > > are
> > > > > > mainly focused on the Flink table module, covering almost all
> > > important
> > > > > > parts such as Client(SQL Client, SQL gateway, JDBC driver, etc),
> API,
> > > > SQL
> > > > > > parser, query optimization, query execution, etc. Especially in
> the
> > > > query
> > > > > > optimization part, he built the query optimization framework and
> the
> > > > cost
> > > > > > model, improved the statistics information and made a lot of
> > > > > optimizations,
> > > > > > e.g. dynamic partition pruning, join hint, multiple input
> rewrite,
> > > etc.
> > > > > In
> > > > > > addition, he has done a lot of groundwork, such as refactoring
> the
> > > > > > ExecNode(which is the basis for further DAG optimizations) and
> SQL
> > > Plan
> > > > > > JSON serialization (which is a big step to support SQL job
> version
> > > > > > upgrade). Besides that, he's also helping the projects in other
> ways,
> > > > > e.g.
> > > > > > managing releases, giving talks, etc.
> > > > > >
> > > > > > Xingbo Huang becomes a Flink committer in Feb 2021. His
> contributions
> > > > are
> > > > > > mainly focused on the PyFlink module and he's the author of many
> > > > > important
> > > > > > features in PyFlink, e.g. Cython support, Python thread execution
> > > mode,
> > > > > > Python UDTF support, Python UDAF support in windowing, etc. He is
> > > also
> > > > > one
> > > > > > of the main contributors in the Flink ML project. Besides that,
> he's
> > > > also
> > > > > > helping to manage releases, taking care of the build stabilites,
> etc.
> > > > > >
> > > > > > Congratulations and welcome them as Apache Flink PMC!
> > > > > >
> > > > > > Regards,
> > > > > > Dian
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Sergey
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-29844) FLIP-263: Improve resolving schema compatibility

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-29844:


 Summary: FLIP-263: Improve resolving schema compatibility
 Key: FLINK-29844
 URL: https://issues.apache.org/jira/browse/FLINK-29844
 Project: Flink
  Issue Type: Improvement
  Components: API / Type Serialization System
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-263: Improve resolving schema compatibility

[Hangxiang Yu]

Hi everyone,

Happy to announce that FLIP-263 [1] has been accepted.


>From vote thread [2], There are 9 approving votes, 7 of which are binding:

- Yanfei Lei (non-binding)
- Yuan Mei (binding)

- Zakelly Lan (non-binding)
- Yun Gao (binding)
- Godfrey He (binding)

- Dawid Wysakowicz (binding)

- Yu Li (binding)

- Tzu-Li (Gordon) Tai (binding)

- Yun Tang (binding)

There are no disapproving votes.

Thanks everyone for joining the discussion, giving feedback and voting !

Best regards,
Hangxiang.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility
[2] https://lists.apache.org/thread/0bh530j5ob11lzj48vpm883sqwgmstp8

[jira] [Created] (FLINK-29802) ChangelogStateBackend supports native savepoint

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-29802:


 Summary: ChangelogStateBackend supports native savepoint
 Key: FLINK-29802
 URL: https://issues.apache.org/jira/browse/FLINK-29802
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / State Backends
Reporter: Hangxiang Yu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Apache Flink 1.16.0 released

[Hangxiang Yu]

Congratulations!
Thanks Chesnay, Martijn, Godfrey & Xingbo for managing the release.

On Fri, Oct 28, 2022 at 7:35 PM Jing Ge  wrote:

> Congrats!
>
> On Fri, Oct 28, 2022 at 1:22 PM 任庆盛  wrote:
>
>> Congratulations and a big thanks to Chesnay, Martijn, Godfrey and Xingbo
>> for the awesome work for 1.16!
>>
>> Best regards,
>> Qingsheng Ren
>>
>> > On Oct 28, 2022, at 14:46, Xingbo Huang  wrote:
>> >
>> > The Apache Flink community is very happy to announce the release of
>> Apache
>> > Flink 1.16.0, which is the first release for the Apache Flink 1.16
>> series.
>> >
>> > Apache Flink® is an open-source stream processing framework for
>> > distributed, high-performing, always-available, and accurate data
>> streaming
>> > applications.
>> >
>> > The release is available for download at:
>> > https://flink.apache.org/downloads.html
>> >
>> > Please check out the release blog post for an overview of the
>> > improvements for this release:
>> > https://flink.apache.org/news/2022/10/28/1.16-announcement.html
>> >
>> > The full release notes are available in Jira:
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351275
>> >
>> > We would like to thank all contributors of the Apache Flink community
>> > who made this release possible!
>> >
>> > Regards,
>> > Chesnay, Martijn, Godfrey & Xingbo
>>
>

-- 
Best,
Hangxiang.

Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0

[Hangxiang Yu]

+1 (non-binding)

On Sat, Oct 29, 2022 at 2:21 AM Tzu-Li (Gordon) Tai 
wrote:

> +1
>
> On Fri, Oct 28, 2022 at 10:21 AM Konstantin Knauf 
> wrote:
>
> > +1 (binding)
> >
> > Am Fr., 28. Okt. 2022 um 16:58 Uhr schrieb Piotr Nowojski <
> > pnowoj...@apache.org>:
> >
> > > Hi,
> > >
> > > As discussed on the dev mailing list [0] I would like to start a vote
> to
> > > drop support of older savepoint formats (for Flink versions older than
> > > 1.8). You can find the original explanation from the aforementioned dev
> > > mailing list thread at the bottom of this message.
> > >
> > > Draft PR containing the proposed change you can find here:
> > > https://github.com/apache/flink/pull/21056
> > >
> > > Vote will be open at least until Wednesday, November 2nd 18:00 CET.
> > >
> > > Best,
> > > Piotrek
> > >
> > > [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w
> > >
> > > I would like to open a discussion to remove the long deprecated
> > > (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the
> related
> > > code.
> > >
> > > The motivation behind this move is two fold. One reason is that it
> > > complicates our code base unnecessarily and creates confusion on how to
> > > actually implement custom serializers. The immediate reason is that I
> > > wanted to clean up Flink's configuration stack a bit and refactor the
> > > ExecutionConfig class [2]. This refactor would keep the API
> compatibility
> > > of the ExecutionConfig, but it would break savepoint compatibility with
> > > snapshots written with some of the old serializers, which had
> > > ExecutionConfig as a field and were serialized in the snapshot. This
> > issue
> > > has been resolved by the introduction of TypeSerializerSnapshot in
> Flink
> > > 1.7 [3], where serializers are no longer part of the snapshot.
> > >
> > > TypeSerializerConfigSnapshot has been deprecated and no longer used by
> > > built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged
> > to
> > > migrate to TypeSerializerSnapshot since then with their own custom
> > > serializers. That has been plenty of time for the migration.
> > >
> > > This proposal would have the following impact for the users:
> > > 1. we would drop support for recovery from savepoints taken with Flink
> <
> > > 1.7.0 for all built in types serializers
> > > 2. we would drop support for recovery from savepoints taken with Flink
> <
> > > 1.8.0 for built in kryo serializers
> > > 3. we would drop support for recovery from savepoints taken with Flink
> <
> > > 1.17 for custom serializers using deprecated
> TypeSerializerConfigSnapshot
> > >
> > > 1. and 2. would have a simple migration path. Users migrating from
> those
> > > old savepoints would have to first start his job using a Flink version
> > from
> > > the [1.8, 1.16] range, and take a new savepoint that would be
> compatible
> > > with Flink 1.17.
> > > 3. This is a bit more problematic, because users would have to first
> > > migrate their own custom serializers to use TypeSerializerSnapshot
> > (using a
> > > Flink version from the [1.8, 1.16]), take a savepoint, and only then
> > > migrate to Flink 1.17. However users had already 4 years to migrate,
> > which
> > > in my opinion has been plenty of time to do so.
> > >
> > > As a side effect, we could also drop support for some of the legacy
> > > metadata serializers from LegacyStateMetaInfoReaders and potentially
> > other
> > > places that we are keeping for the sake of compatibility with old
> > > savepoints.
> > >
> > > [1]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html
> > > [2] https://issues.apache.org/jira/browse/FLINK-29379
> > > [3] https://issues.apache.org/jira/browse/FLINK-9377
> > > [4] https://issues.apache.org/jira/browse/FLINK-9376
> > > [5] https://issues.apache.org/jira/browse/FLINK-11323
> > >
> >
> >
> > --
> > https://twitter.com/snntrable
> > https://github.com/knaufk
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

[Hangxiang Yu]

Sure, Thanks a lot for the reminder !

On Fri, Oct 28, 2022 at 3:59 PM Dawid Wysakowicz 
wrote:

> I see the majority of the community prefers the not immediate breaking
> of the API. Even though I have different preference I am more than fine
> to commit to the others choice. Just please make sure it is prominently
> documented that one or the other methods MUST be implemented in any new
> serializer.
>
> Best,
>
> Dawid
>
> On 27/10/2022 08:33, Yun Gao wrote:
> > Hi,
> > I have discussed offline with Hangxiang and Yuan, and the current
> > proposal also looks good to me.
> > Thanks Hangxiang for driving this.
> > Best,
> > Yun Gao
> > --
> > From:Yuan Mei 
> > Send Time:2022 Oct. 26 (Wed.) 12:20
> > To:dev 
> > Subject:Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility
> > Hey Huangxiang,
> > The section of `Rejected Alternatives` may also need an update.
> > Current plan sounds like a reasonable one. I am fine with it.
> > Thanks for driving this.
> > Best
> > Yuan
> > On Tue, Oct 25, 2022 at 5:11 PM Hangxiang Yu 
> wrote:
> >> （Resend the mail to fix the format issue）
> >> Hi, everyone.
> >>
> >> Thanks for your suggestions!
> >>
> >> Let me summarize the remaining questions in the thread and share my
> ideas
> >> based on your suggestions:
> >>
> >>
> >> 1. Should we put the new opposite interface in TypeSerializer or
> >> TypeSerializerSnapshot ?
> >>
> >> Just as I replied to Dawid, I'd like to put it in
> >> TypeSerializerSnapshot so that we could still follow the contract
> between
> >> two classes and make later code migration easier based on current tools.
> >>
> >> Thanks Dawid for the initial suggestion and Godfrey for the additional
> >> supplement.
> >>
> >>
> >>
> >> 2. Do we just break changes and make user codes incompatible or make
> sure
> >> compatibility using a more suitable migration plan ?
> >>
> >> I agree with Yuan that we should make sure that user jobs still work
> >> without modifying any codes before removing the deprecated method.
> >>
> >> Thanks Yuan for the migration plan. Let me try to add something to the
> >> suggestion of Yuan:
> >>
> >>
> >> a. In Step 1, I prefer to make the new interface like:
> >>
> >>
> >> default TypeSerializerSchemaCompatibility
> >> resolveSchemaCompatibility(
> >> // Use 'oldSnapshot' not 'oldSerializer'
> >> TypeSerializerSnapshot oldSnapshot) {
> >> return INCOMPATIBLE;
> >> }
> >> }
> >>
> >> I think using 'oldSnapshot' as the parameter will make the
> >> logic clear --- TypeSerializerSnapshot will take all responsibility for
> >> compatibility checks.
> >> BTW, It's also easy to migrate original check logic to this
> >> interface.
> >>
> >> b. In Step 1, In addition to introducing default implementations
> >> for both interfaces, we also need to implement the new interface in all
> >> inner TypeSerializerSnapshots.
> >> Users may implement their own serializers based on inner
> >> serializers, we should make sure that the new interface of inner
> >> TypeSerializerSnapshots is usable.
> >>
> >> Then I think it could work for both old custom serializers or new
> >> custom serializers.
> >> No matter which interface the user implements, it could always work.
> >> Of course, we will deprecate the old interface and encourage users to
> >> use the new one.
> >>
> >>
> >> 3. Do we need to squash this with
> >> https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w <
> https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w > ?
> >> We will not break the compatibility based on 2, so it's not necessary
> >> to squash them together.
> >>
> >>
> >> Do you have any other suggestions ? Look forward to your reply!
> >>
> >> On Tue, Oct 25, 2022 at 1:31 PM Hangxiang Yu 
> wrote:
> >>
> >>> Hi, everyone.
> >>>
> >>> Thanks for your suggestions!
> >>>
> >>> Let me summarize the remaining questions in the thread and share my
> ideas
> >>> based on your suggestions:
> >>>
> >>> 1. Should we put the new opposite interface in TypeSerializer or
> >>

[VOTE] FLIP-263: Improve resolving schema compatibility

[Hangxiang Yu]

Hi everyone,

I'd like to start the vote for FLIP-263 [1].

Thanks for your feedback and the discussion in [2][3].

The vote will be open for at least 72 hours.

Best regards,
Hangxiang.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility
[2] https://lists.apache.org/thread/4w36oof8dh28b9f593sgtk21o8qh8qx4

[3] https://lists.apache.org/thread/t0bdkx1161rlbnsf06x0kswb05mch164

[jira] [Created] (FLINK-29777) [JUnit5 Migration] Module: flink-dstl

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-29777:


 Summary: [JUnit5 Migration] Module: flink-dstl
 Key: FLINK-29777
 URL: https://issues.apache.org/jira/browse/FLINK-29777
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends, Tests
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29776) [JUnit5 Migration] Module: flink-statebackend-changelog

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-29776:


 Summary: [JUnit5 Migration] Module: flink-statebackend-changelog
 Key: FLINK-29776
 URL: https://issues.apache.org/jira/browse/FLINK-29776
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends, Tests
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29775) [JUnit5 Migration] Module: flink-statebackend-rocksdb

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-29775:


 Summary: [JUnit5 Migration] Module: flink-statebackend-rocksdb
 Key: FLINK-29775
 URL: https://issues.apache.org/jira/browse/FLINK-29775
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends, Tests
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Hangxiang Yu]

Hi, Yanfei.
Thanks for driving this.
It could help us to detect and resolve the regression problem quickly and
officially.
I'd like to join as a maintainer.
Looking forward to the workflow.

On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei  wrote:

> Thanks, Yanfei, to drive this and make the performance monitoring publicly
> available.
>
> Looking forward to seeing the workflow, and more details as Martijn
> mentioned.
>
> Best
> Yuan
>
> On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser 
> wrote:
>
> > Hi Yanfei Lei,
> >
> > Thanks for setting this up! It would be interesting to also know which
> > aspects of Flink are monitored for "performance". I'm assuming there are
> > specific pieces of functionality that are performance tested, but it
> would
> > be great if this would be written down somewhere (next to a procedure how
> > to detect a regression and what should be next steps).
> >
> > Best regards,
> >
> > Martijn
> >
> > On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan 
> wrote:
> >
> > > Hi yanfei,
> > >
> > > Thanks for driving this! It's a great help.
> > >
> > > I would like to join as a maintainer.
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Wed, Oct 26, 2022 at 11:32 AM yanfei lei 
> wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > As discussed earlier, we plan to create a benchmark channel in Apache
> > > Flink
> > > > slack[1], but the plan was shelved for a while[2]. So I went on with
> > this
> > > > work, and created the #flink-dev-benchmarks channel for performance
> > > > regression notifications.
> > > >
> > > > We have a regression report script[3] that runs daily, and a
> > notification
> > > > would be sent to the slack channel when the last few benchmark
> results
> > > are
> > > > significantly worse than the baseline.
> > > > Note, regressions are detected by a simple script which may have
> false
> > > > positives and false negatives. And all benchmarks are executed on one
> > > > physical machine[4] which is provided by Ververica(Alibaba)[5], it
> > might
> > > > happen that hardware issues affect performance, like "[FLINK-18614
> > > > ] Performance
> > > regression
> > > > 2020.07.13"[6].
> > > >
> > > > After the migration, we need a procedure to watch over the entire
> > > > performance of Flink code together. For example, if a regression
> > > > occurs, investigating the cause and resolving the problem are needed.
> > In
> > > > the past, this procedure is maintained internally within Ververica,
> but
> > > we
> > > > think making the procedure public would benefit all. I volunteer to
> > serve
> > > > as one of the initial maintainers, and would be glad if more
> > contributors
> > > > can join me. I'd also prepare some guidelines to help others get
> > familiar
> > > > with the workflow. I will start a new thread to discuss the workflow
> > > soon.
> > > >
> > > >
> > > > [1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> > > > [2] https://issues.apache.org/jira/browse/FLINK-28468
> > > > [3]
> > > >
> > >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
> > > > [4] http://codespeed.dak8s.net:8080
> > > > [5] https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro
> > > >
> > > > [6] https://issues.apache.org/jira/browse/FLINK-18614
> > >
> >
>


-- 
Best,
Hangxiang.

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

[Hangxiang Yu]

（Resend the mail to fix the format issue）
Hi, everyone.

Thanks for your suggestions!

Let me summarize the remaining questions in the thread and share my ideas
based on your suggestions:


1. Should we put the new opposite interface in TypeSerializer or
TypeSerializerSnapshot ?

Just as I replied to Dawid, I'd like to put it in
TypeSerializerSnapshot so that we could still follow the contract between
two classes and make later code migration easier based on current tools.

Thanks Dawid for the initial suggestion and Godfrey for the additional
supplement.



2. Do we just break changes and make user codes incompatible or make sure
compatibility using a more suitable migration plan ?

I agree with Yuan that we should make sure that user jobs still work
without modifying any codes before removing the deprecated method.

Thanks Yuan for the migration plan. Let me try to add something to the
suggestion of Yuan:


  a. In Step 1, I prefer to make the new interface like:


  default TypeSerializerSchemaCompatibility
resolveSchemaCompatibility(
  // Use 'oldSnapshot' not 'oldSerializer'
 TypeSerializerSnapshot oldSnapshot) {
 return INCOMPATIBLE;
   }
  }

  I think using 'oldSnapshot' as the parameter will make the
logic clear --- TypeSerializerSnapshot will take all responsibility for
compatibility checks.
  BTW, It's also easy to migrate original check logic to this
interface.

   b. In Step 1, In addition to introducing default implementations
for both interfaces, we also need to implement the new interface in all
inner TypeSerializerSnapshots.
   Users may implement their own serializers based on inner
serializers, we should make sure that the new interface of inner
TypeSerializerSnapshots is usable.

 Then I think it could work for both old custom serializers or new
custom serializers.
 No matter which interface the user implements, it could always work.
 Of course, we will deprecate the old interface and encourage users to
use the new one.


 3. Do we need to squash this with
https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w ?
 We will not break the compatibility based on 2, so it's not necessary
to squash them together.


Do you have any other suggestions ? Look forward to your reply!

On Tue, Oct 25, 2022 at 1:31 PM Hangxiang Yu  wrote:

> Hi, everyone.
>
> Thanks for your suggestions!
>
> Let me summarize the remaining questions in the thread and share my ideas
> based on your suggestions:
>
>1. Should we put the new opposite interface in TypeSerializer or
>TypeSerializerSnapshot ?
>
> Just as I replied to Dawid, I'd like to put it in TypeSerializerSnapshot
> so that we could still follow the contract between two classes and make
> later code migration easier based on current tools.
>
> Thanks Dawid for the initial suggestion and Godfrey for the additional
> supplement.
>
>
>
>1. Do we just break changes and make user codes incompatible or make
>sure compatibility using a more suitable migration plan ?
>
> I agree with Yuan that we should make sure that user jobs still work
> without modifying any codes before removing the deprecated method.
>
> Thanks Yuan for the migration plan. Let me try to add something to the
> suggestion of Yuan:
>
>1. In Step 1, I prefer to make the new interface like:
>
> default TypeSerializerSchemaCompatibility resolveSchemaCompatibility(
>
> // Use 'oldSnapshot' not 'oldSerializer'
> TypeSerializerSnapshot oldSnapshot) {
>
> return INCOMPATIBLE;
>
> }
>
> I think using 'oldSnapshot' as the parameter will make the logic clear ---
> TypeSerializerSnapshot will take all responsibility for compatibility
> checks.
>
> BTW, It's also easy to migrate original check logic to this interface.
>
>1. In Step 1, In addition to introducing default implementations for
>   both interfaces, we also need to implement the new interface in all 
> inner
>   TypeSerializerSnapshots.
>
> Users may implement their own serializers based on inner serializers, we
> should make sure that the new interface of inner TypeSerializerSnapshots is
> usable.
>
>
> Then I think it could work for both old custom serializers or new custom
> serializers.
>
> No matter which interface the user implements, it could always work.
>
> Of course, we will deprecate the old interface and encourage users to use
> the new one.
>
>
>
>1. Do we need to squash this with
>https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w ?
>
> We will not break the compatibility based on 2, so it's not necessary to
> squash them together.
>
>
> Do you have

Re: Limiting backpressure during checkpoints

[Hangxiang Yu]

Hi Robin.
Could you share how you got the metric of CPU usage ?
By summing all used CPU cores of TMs or evaluating it by the busy metric in
Flink UI ?
I think it's the first thing we need to align.

> network (async) part of the checkpoint should in theory not cause
backpressure since resources would be used for the main stream during async
waits, but I might be wrong
I also believe it works as you said when the job is running normally.
So I think there are other bottlenecks causing backpressure when checkpoint.
Maybe you could check other resources usage like io utilization (as Zakelly
suggested), heap usage, gc when checkpoint ?

On Tue, Oct 25, 2022 at 2:01 PM Zakelly Lan  wrote:

> Hi Robin,
>
> You said that during the checkpoint async phase the CPU is stable at
> 100%, which is pretty strange to me. Normally the cpu usage of the
> taskmanager process could exceed 100%, depending on what all the
> threads are doing. I'm wondering if there is any scheduling mechanism
> controlling the CPU usage of a process in your setup, such as
> leveraging CGroup in yarn or Kubernetes. In this case, the uploading
> thread may preempt cpu resources from the task processing thread.
> The second thing that might help is, you may check the io utilization
> during the checkpoint. The uploading thread keeps reading from the
> local disk and writing to the remote, which may affect the io and
> state access latency, especially when the state size is large.
>
> Best,
> Zakelly
>
> On Tue, Oct 25, 2022 at 12:10 AM Robin Cassan via user
>  wrote:
> >
> > Hello Yuan Mei! Thanks a lot for your answer :)
> >
> > About the CPU usage, it is pretty stable at 80% normally. Every 15
> minutes we trigger a checkpoint, and during this time it is stable at 100%
> > I am starting to wonder if CPU is the real limiting factor, because when
> checking the Flink UI I see that most of the checkpoint duration is async.
> I do not know how the async phase affects backpressure, but it does look
> like the upload to S3 phase is causing the backpressure. The sync phase is
> quite short as well.
> > Looking at this article
> https://flink.apache.org/2022/05/23/latency-part2.html it seems we
> already are in the most efficient configuration (at-least-once,
> non-concurrent checkpointing, rocksdb on local NVME SSDs...), I don't see
> an obvious quick-win apart from scaling up the full cluster.
> >
> > Reducing the state size will be a big challenge but even then it would
> not guarantee consistent latency, same for less frequent checkpoints.
> > For now it looks like our only option to achieve real-time computation
> would be to not use Flink (or at least, not include these computations
> inside a job with a big state that is checkpointed). Thanks again for the
> insight, and if you happen to have any information on how we could prevent
> the async phase of checkpoints to add backpressure on our stream I would be
> very interested!
> >
> > Le mer. 19 oct. 2022 à 10:57, Yuan Mei  a écrit
> :
> >>
> >> Hey Robin,
> >>
> >> Thanks for sharing the detailed information.  May I ask, when you are
> saying "CPU usage is around 80% when checkpoints aren't running, and capped
> at 100% when they are", do you see zigzag patterns of CPU usage, or is it
> kept capped at 100% of CPU?
> >>
> >> I think one possibility is that the sync phase of cp (the writebuffer
> flush during the sync phase) triggers a rocksdb compaction, and we saw this
> happens on Ververica services as well.
> >>
> >> At this moment, maybe you can try to make the checkpoint less frequent
> (increase the checkpoint interval) to reduce the frequency of compaction.
> Please let me know whether this helps.
> >>
> >> In long term, I think we probably need to separate the compaction
> process from the internal db and control/schedule the compaction process
> ourselves (compaction takes a good amount of CPU and reduces TPS).
> >>
> >> Best.
> >> Yuan
> >>
> >>
> >>
> >> On Thu, Oct 13, 2022 at 11:39 PM Robin Cassan via user <
> u...@flink.apache.org> wrote:
> >>>
> >>> Hello all, hope you're well :)
> >>> We are attempting to build a Flink job with minimal and stable latency
> (as much as possible) that consumes data from Kafka. Currently our main
> limitation happens when our job checkpoints the RocksDB state: backpressure
> is applied on the stream, causing latency. I am wondering if there are ways
> to configure Flink so that the checkpointing process affects the flow of
> data as little as possible?
> >>>
> >>> In our case, backpressure seems to arise from CPU consumption, because:
> >>> - CPU usage is around 80% when checkpoints aren't running, and capped
> at 100% when they are
> >>> - checkpoint alignment time is very low, using unaligned checkpoints
> doesn't appear to help with backpressure
> >>> - network (async) part of the checkpoint should in theory not cause
> backpressure since resources would be used for the main stream during async
> waits, but I might be wrong
> >>>
> >>> What we