Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

2024-06-05 Thread Yanfei Lei
Congratulations, Rui! Best, Yanfei Luke Chen 于2024年6月5日周三 20:08写道: > > Congrats, Rui! > > Luke > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun wrote: > > > Congrats, Rui. Well-deserved! > > > > Best, > > Jiabao > > > > Zhanghao Chen 于2024年6月5日周三 19:29写道: > > > > > Congrats, Rui! > > > > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

2024-06-04 Thread Yanfei Lei
Congratulations! Best, Yanfei Leonard Xu 于2024年6月4日周二 16:20写道: > > Congratulations! > > Best, > Leonard > > > 2024年6月4日 下午4:02,Yangze Guo 写道: > > > > Congratulations! > > > > Best, > > Yangze Guo > > > > On Tue, Jun 4, 2024 at 4:00 PM Weihua Hu wrote: > >> > >> Congratulations, Weijie! > >> >

[jira] [Created] (FLINK-35495) The native metrics for column family are not reported

2024-05-30 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35495: -- Summary: The native metrics for column family are not reported Key: FLINK-35495 URL: https://issues.apache.org/jira/browse/FLINK-35495 Project: Flink Issue Type

Re: [VOTE] FLIP-443: Interruptible watermark processing

2024-05-23 Thread Yanfei Lei
Thanks for driving this! +1 (binding) Best, Yanfei Zakelly Lan 于2024年5月24日周五 10:13写道: > > +1 (binding) > > Best, > Zakelly > > On Thu, May 23, 2024 at 8:21 PM Piotr Nowojski wrote: > > > Hi all, > > > > After reaching what looks like a consensus in the discussion thread [1], I > > would like

[jira] [Created] (FLINK-35405) Add buffer size/in-flight records metrics for AsyncExecutionController

2024-05-20 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35405: -- Summary: Add buffer size/in-flight records metrics for AsyncExecutionController Key: FLINK-35405 URL: https://issues.apache.org/jira/browse/FLINK-35405 Project: Flink

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-05 Thread Yanfei Lei
+1 (binding) Best, Yanfei Zakelly Lan 于2024年5月6日周一 11:00写道: > > +1 (binding) > > Thanks for driving this! > > > Best, > Zakelly > > On Mon, May 6, 2024 at 10:54 AM yue ma wrote: > > > Hi everyone, > > > > Thanks for all the feedback, I'd like to start a vote on the FLIP-447: > > Upgrade

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-01 Thread Yanfei Lei
capsulated into mail and > >>> executed in mailbox. Is processing-timer allowed to be interrupted? > >> > >> Yes, both firing processing and even time timers share the same code and > >> both will > >> support interruptions in the same way. Actually I've renamed the

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-04-29 Thread Yanfei Lei
Hi Piotrek, Thanks for this proposal. It looks like it will shorten the checkpoint duration, especially in the case of back pressure. +1 for it! I'd like to ask some questions to understand your thoughts more precisely. 1. Currently when AbstractStreamOperator or AbstractStreamOperatorV2

Re: [DISCUSS] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-04-22 Thread Yanfei Lei
Hi Yue & Roman, Thanks for initiating this FLIP and all the efforts for the upgrade. 8.10.0 introduces some new features, making it possible for Flink to implement some new exciting features, and the upgrade also makes FRocksDB easier to maintain, +1 for upgrading. I read the FLIP and have a

[jira] [Created] (FLINK-35158) Error handling in StateFuture's callback

2024-04-18 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35158: -- Summary: Error handling in StateFuture's callback Key: FLINK-35158 URL: https://issues.apache.org/jira/browse/FLINK-35158 Project: Flink Issue Type: Sub-task

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-14 Thread Yanfei Lei
Congratulations, Zakelly! Best regards, Yanfei Junrui Lee 于2024年4月15日周一 12:06写道: > > Congratulations, Zakelly! > > Best, > Junrui > > yue ma 于2024年4月15日周一 12:00写道: > > > Congratulations Zakelly! > > > > Best, > > Yue > >

Re: [VOTE] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

2024-04-11 Thread Yanfei Lei
Hi Rui, Thanks for driving it! +1 (binding) Hangxiang Yu 于2024年4月12日周五 10:26写道: > > +1 (binding) > > On Fri, Apr 12, 2024 at 10:22 AM Jinzhong Li > wrote: > > > +1 (non binding) > > > > Bests, > > Jinzhong > > > > On Thu, Apr 11, 2024 at 7:26 AM Muhammet Orazov > > wrote: > > > > > Hey

[jira] [Created] (FLINK-35031) Event timer firing under async execution model

2024-04-07 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35031: -- Summary: Event timer firing under async execution model Key: FLINK-35031 URL: https://issues.apache.org/jira/browse/FLINK-35031 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-35030) Introduce Epoch Manager for watermark under async execution

2024-04-07 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35030: -- Summary: Introduce Epoch Manager for watermark under async execution Key: FLINK-35030 URL: https://issues.apache.org/jira/browse/FLINK-35030 Project: Flink

[jira] [Created] (FLINK-35029) Store timer in JVM heap when async execution enabled

2024-04-07 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35029: -- Summary: Store timer in JVM heap when async execution enabled Key: FLINK-35029 URL: https://issues.apache.org/jira/browse/FLINK-35029 Project: Flink Issue Type

[jira] [Created] (FLINK-35028) Processing timer firing under async execution model

2024-04-07 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35028: -- Summary: Processing timer firing under async execution model Key: FLINK-35028 URL: https://issues.apache.org/jira/browse/FLINK-35028 Project: Flink Issue Type

[jira] [Created] (FLINK-35027) Implement checkpoint drain in AsyncExecutionController

2024-04-07 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35027: -- Summary: Implement checkpoint drain in AsyncExecutionController Key: FLINK-35027 URL: https://issues.apache.org/jira/browse/FLINK-35027 Project: Flink Issue

[jira] [Created] (FLINK-35026) Introduce async-state configurations

2024-04-06 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35026: -- Summary: Introduce async-state configurations Key: FLINK-35026 URL: https://issues.apache.org/jira/browse/FLINK-35026 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35025) Wire AsyncExecutionController to AbstractStreamOperator

2024-04-06 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35025: -- Summary: Wire AsyncExecutionController to AbstractStreamOperator Key: FLINK-35025 URL: https://issues.apache.org/jira/browse/FLINK-35025 Project: Flink Issue

[jira] [Created] (FLINK-35024) Implement record order preservation and buffering of AsyncExecutionController

2024-04-06 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35024: -- Summary: Implement record order preservation and buffering of AsyncExecutionController Key: FLINK-35024 URL: https://issues.apache.org/jira/browse/FLINK-35024 Project

[RESULT][VOTE] FLIP-425: Asynchronous Execution Model

2024-03-31 Thread Yanfei Lei
I'm happy to announce that FLIP-425: Asynchronous Execution Model[1] has been accepted with 10 approving votes (6 binding) [2]: - Xuannan Su (non-binding) - Yuan Mei (binding) - Piotr Nowojski (binding) - Feifan Wang (non-binding) - Jing Ge (binding) - Rui Fan (binding) - Yunfeng Zhou

Re: [VOTE] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State

2024-03-31 Thread Yanfei Lei
+1 (binding) Best, Yanfei yue ma 于2024年3月29日周五 17:29写道: > > +1(non-binding) > > Jinzhong Li 于2024年3月27日周三 19:31写道: > > > Hi devs, > > > > > > I'd like to start a vote on the FLIP-428: Fault Tolerance/Rescale > > Integration for Disaggregated State [1]. The discussion thread is here [2]. > > >

Re: [VOTE] FLIP-426: Grouping Remote State Access

2024-03-31 Thread Yanfei Lei
+1 (binding) Best, Yanfei yue ma 于2024年3月29日周五 16:10写道: > > +1 (non-binding) > > Jinzhong Li 于2024年3月27日周三 18:57写道: > > > Hi devs, > > > > I'd like to start a vote on the FLIP-426: Grouping Remote State Access [1]. > > The discussion thread is here [2]. > > > > The vote will be open for at

Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

2024-03-28 Thread Yanfei Lei
Congratulations! Best, Yanfei Zhanghao Chen 于2024年3月28日周四 19:59写道: > > Congratulations! > > Best, > Zhanghao Chen > > From: Yu Li > Sent: Thursday, March 28, 2024 15:55 > To: d...@paimon.apache.org > Cc: dev ; user > Subject: Re: [ANNOUNCE] Apache Paimon is

Re: [DISCUSS] FLIP-XXX: Introduce Flink SQL variables

2024-03-27 Thread Yanfei Lei
Hi Ferenc, Thanks for the proposal, using SQLvariables to exclude environment-specific configuration from code sounds like a good idea. I'm new to Flink SQL and I'm curious if these variables can be calculated from statements or expression [1]? In FLIP, it seems that the values are in the form

[VOTE] FLIP-425: Asynchronous Execution Model

2024-03-27 Thread Yanfei Lei
Hi everyone, Thanks for all the feedback about the FLIP-425: Asynchronous Execution Model [1]. The discussion thread is here [2]. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1] https://cwiki.apache.org/confluence/x/S4p3EQ [2]

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-25 Thread Yanfei Lei
uot; would execute before "the subsequent > > records of watermark", but in out-of-order mode(async API), the > > execution order between them is not guaranteed > > > > Got it, what I don't get exactly is what type of inconsistency/issue the > > user co

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-21 Thread Yanfei Lei
Congratulations! Best regards, Yanfei Xuannan Su 于2024年3月22日周五 09:21写道: > > Congratulations! > > Best regards, > Xuannan > > On Fri, Mar 22, 2024 at 9:17 AM Charles Zhang wrote: > > > > Congratulations! > > > > Best wishes, > > Charles Zhang > > from Apache InLong > > > > > > Jeyhun Karimov

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-21 Thread Yanfei Lei
ps://lists.apache.org/thread/986zxq1k9rv3vkbk39yw16g24o6h83mz 于2024年3月21日周四 19:29写道: > > Thank you everybody for the questions and answers (especially Yanfei Lei), it > was very instructive to go over the discussion. > I am gonna add some questions on top of what happened and add so

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-19 Thread Yanfei Lei
Hi everyone, Thanks for your valuable discussion and feedback! Our discussions have been going on for a while and there have been no new comments for several days. So I would like to start a vote after 72 hours. Please let me know if you have any concerns, thanks! Yanfei Lei 于2024年3月13日周三 12

Re: [ANNOUNCE] Apache Flink 1.19.0 released

2024-03-18 Thread Yanfei Lei
Congrats, thanks for the great work! Sergey Nuyanzin 于2024年3月18日周一 19:30写道: > > Congratulations, thanks release managers and everyone involved for the great > work! > > On Mon, Mar 18, 2024 at 12:15 PM Benchao Li wrote: >> >> Congratulations! And thanks to all release managers and everyone >>

[jira] [Created] (FLINK-34667) Changelog state backend support local rescaling

2024-03-14 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-34667: -- Summary: Changelog state backend support local rescaling Key: FLINK-34667 URL: https://issues.apache.org/jira/browse/FLINK-34667 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-12 Thread Yanfei Lei
; > > My question was not about how exceptions will be defined. I am not sure how > unchecked exceptions handling will be implemented. I was wondering if > exceptions in the mail chain would have an impact on the reference > counting? E.g. in Fig 5, if an exception happened in the value

Re: [VOTE] Release 1.19.0, release candidate #2

2024-03-11 Thread Yanfei Lei
+1 (non-binding) - Signature and checksum are OK - Source dist doesn't contain unexpected binaries - Build from source successfully, `SocketWindowWordCount` worked well Hangxiang Yu 于2024年3月12日周二 09:59写道: > > +1 (non-binding) > > - Verified signatures and checksums > - Reviewed Web PR > -

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-11 Thread Yanfei Lei
va#L226 > <https://github.com/apache/flink/blob/fcf61b9108942a892aa93765178a81f2c0e787c5/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/stream/MiniBatchStreamingJoinOperator.java#L226> > > Best, > Jane > > > On Mon, M

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-11 Thread Yanfei Lei
uot; or the > context is switched in the background? > > One example use-case would be that you get something from the state and > based on the value you register a timer. > > Thanks, > Gyula > > On Mon, Mar 11, 2024 at 3:58 AM Yanfei Lei wrote: &

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-10 Thread Yanfei Lei
aking of draining, is it correct to understand that AEC is stateless? > E.g. AEC could be easily scaled out if it became a bottleneck. > 2. There are Pseudo-code for the inFilghtReocordNum increment, would you > like to add Pseudo-code for the inFilghtReocordNum decrement to help us > understand the logic bett

[jira] [Created] (FLINK-34624) Enable local recovery in ChangelogRescalingITCase

2024-03-08 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-34624: -- Summary: Enable local recovery in ChangelogRescalingITCase Key: FLINK-34624 URL: https://issues.apache.org/jira/browse/FLINK-34624 Project: Flink Issue Type

[DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-07 Thread Yanfei Lei
Hi devs, I'd like to start a discussion on FLIP-425: Asynchronous Execution Model[1], which is a sub-FLIP of FLIP-423: Disaggregated State Storage and Management[2]. FLIP-425 introduces a non-blocking execution model leveraging the asynchronous APIs introduced in FLIP-424[3]. For the whole story

Re: [VOTE] FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

2024-03-06 Thread Yanfei Lei
+1(binding) for this vote. Hangxiang Yu 于2024年3月7日周四 09:54写道: > > +1 (binding) > > On Thu, Mar 7, 2024 at 9:34 AM Yun Tang wrote: > > > > +1 for this FLIP. > > Sorry for not being clear in my previous reply, it's a binding vote. > > > > Best > > Yun Tang > > > >

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-05 Thread Yanfei Lei
r other results > >> >> > of our > >> >> > PoC[1], you can follow the instructions here[2] to reproduce. Since > >> >> > the > >> >> > compilation may take some effort, we will directly provide the jar for > >> >> > testing

Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

2024-02-25 Thread Yanfei Lei
rocksdb-memory > > > Best > Yun Tang > > From: Yanfei Lei > Sent: Thursday, February 22, 2024 15:39 > To: dev@flink.apache.org > Subject: Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB StateBackend > user-facing classe

Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

2024-02-21 Thread Yanfei Lei
Hi Jinzhong, Thanks for driving this! 1. I'm wondering if `ConfigurableRocksDBOptionsFactory` will be used by users, currently it looks like only developers use it in rocksdb state backend module. And Its only non-testing subclass "DefaultConfigurableOptionsFactory" is marked @Deprecated. 2.

Re: [VOTE] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-24 Thread Yanfei Lei
+1 (binding) Hangxiang Yu 于2024年1月25日周四 10:00写道: > > +1 (binding) > > On Thu, Jan 25, 2024 at 8:49 AM Rui Fan <1996fan...@gmail.com> wrote: > > > +1(binding) > > > > Best, > > Rui > > > > On Wed, 24 Jan 2024 at 21:50, Zakelly Lan wrote: > > > > > Hi everyone, > > > > > > I'd like to start a

Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-18 Thread Yanfei Lei
+1 (binding) Best, Yanfei Hangxiang Yu 于2024年1月19日周五 12:13写道: > > +1 (binding) > > On Fri, Jan 19, 2024 at 12:10 PM Zakelly Lan wrote: > > > Hi everyone, > > > > I'd like to start a vote on the FLIP-416: Deprecate and remove the > > RestoreMode#LEGACY [1]. The discussion thread is here [2]. >

Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-13 Thread Yanfei Lei
Thanks Zakelly for starting this discussion. Regardless of whether it is for users or developers, deprecating RestoreMode#LEGACY makes the semantics clearer and lower maintenance costs, and Flink 2.0 is a good time point to do this. So +1 for the overall idea. Best, Yanfei Zakelly Lan

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2023-12-27 Thread Yanfei Lei
>> But I'm not so sure since there is only one savepoint-related > > option. > > > > Maybe someone else could share some thoughts here. > > > > > > > > How about > > > > state.savepoints.dir -> execution.checkpointing.savepoint.dir

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2023-12-25 Thread Yanfei Lei
Hi Zakelly, Thank you for creating the FLIP and starting the discussion. The current arrangement of these options is indeed somewhat haphazard, and the new arrangement looks much better. I have some questions about the arrangement of some new configuration options: 1. For some state backends

Re: [ANNOUNCE] The Flink Speed Center and benchmark daily run are back online

2023-10-20 Thread Yanfei Lei
y were running on > > several > > > > >>>> machines donated and maintained by Ververica. Unfortunately, those > > > > >>>> machines were gone due to account issues [3] and the benchmarks > > > daily > > > > >>>&g

Re: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

2023-09-19 Thread Yanfei Lei
Hi Zakelly, Thanks for bringing this up. +1 for reorganizing. IIUC, this proposal aims to change all state-related exceptions to unchecked exceptions. If users have caught checked exceptions (such as IOException ) in their code, leaving the code as is would also work. Is it possible not to put

Re: [DISCUSS] Add config to enable job stop with savepoint on exceeding tolerable checkpoint Failures

2023-09-05 Thread Yanfei Lei
Hi Dongwoo, If the checkpoint has failed `execution.checkpointing.tolerable-failed-checkpoints` times, then stopWithSavepoint is likely to fail as well. If stopWithSavepoint succeeds or fails, will the job just stop? I am more curious about how this option works with the restart strategy? Best,

Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl

2023-08-05 Thread Yanfei Lei
Congratulations! Best regards, Yanfei Hong Liang 于2023年8月4日周五 20:02写道: > > Congratulations Matthias! > > Very well deserved! > > Regards, > Hong > > On Fri, Aug 4, 2023 at 10:38 AM Markos Sfikas > wrote: > > > Congrats Matthias! > > Best > > Markos > > > > On Fri, Aug 4, 2023 at 11:25 AM

[jira] [Created] (FLINK-32677) flink-benchmarks-regression-check failed to send slack messages since 2023.07.17

2023-07-26 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-32677: -- Summary: flink-benchmarks-regression-check failed to send slack messages since 2023.07.17 Key: FLINK-32677 URL: https://issues.apache.org/jira/browse/FLINK-32677 Project

Re: [DISCUSS][2.0] FLIP-349: Move RocksDB statebackend classes to o.a.f.state.rocksdb package

2023-07-25 Thread Yanfei Lei
+1 for moving all classes in the state-backend-rocksdb module under the classes to o.a.f.state.rocksdb package. I have always been curious about the relationship between o.a.f.contrib.xx and the flink-contrib module. :) Best, Yanfei Jing Ge 于2023年7月25日周二 17:50写道: > > make sense. > > Best

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

2023-07-25 Thread Yanfei Lei
Congratulations! Best regards, Yanfei weijie guo 于2023年7月26日周三 10:10写道: > > Congrats, Yong Fang! > > Best regards, > > Weijie > > > Danny Cranmer 于2023年7月26日周三 03:34写道: > > > Congrats and welcome! > > > > Danny. > > > > On Tue, 25 Jul 2023, 16:48 Matthias Pohl, > > wrote: > > > > >

Re: Rocksdb info

2023-07-20 Thread Yanfei Lei
Hi Taher, > is flink using rocks-cloud too? Or flink is moving rocks files after checkpoints? Flink isn't using rocks-cloud. flink would upload local rocks files to s3/DFS during checkpointing. Here is the related code: - RocksNativeFullSnapshotStrategy:

[jira] [Created] (FLINK-32443) Translate "State Processor API" page into Chinese

2023-06-26 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-32443: -- Summary: Translate "State Processor API" page into Chinese Key: FLINK-32443 URL: https://issues.apache.org/jira/browse/FLINK-32443 Project: Flink

[jira] [Created] (FLINK-32440) Introduce file merging configuration

2023-06-26 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-32440: -- Summary: Introduce file merging configuration Key: FLINK-32440 URL: https://issues.apache.org/jira/browse/FLINK-32440 Project: Flink Issue Type: Sub-task

Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints

2023-05-09 Thread Yanfei Lei
+1 (no-binding) Best, Yanfei Jing Ge 于2023年5月10日周三 07:03写道: > > Hi Zakelly, > > I saw you sent at least 4 same emails for voting FLIP-306. I guess this one > should be the last one and the right one for us to vote right? BTW, based > on the sending time, 72 hours means to open the discussion

[jira] [Created] (FLINK-32018) Many builds of benchmark have been interrupted since 20230428

2023-05-05 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-32018: -- Summary: Many builds of benchmark have been interrupted since 20230428 Key: FLINK-32018 URL: https://issues.apache.org/jira/browse/FLINK-32018 Project: Flink

[jira] [Created] (FLINK-31969) Remove all local recovery files immediately when local recovery fail

2023-04-28 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-31969: -- Summary: Remove all local recovery files immediately when local recovery fail Key: FLINK-31969 URL: https://issues.apache.org/jira/browse/FLINK-31969 Project: Flink

Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu

2023-04-23 Thread Yanfei Lei
Congratulations, Leonard! Best, Yanfei liu ron 于2023年4月23日周日 17:45写道: > > Congratulations, Leonard. > > Best, > Ron > > Zhanghao Chen 于2023年4月23日周日 17:33写道: > > > Congratulations, Leonard! > > > > > > Best, > > Zhanghao Chen > > > > From: Shammon FY > > Sent:

Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren

2023-04-23 Thread Yanfei Lei
Congratulations, Qingsheng! Best, Yanfei liu ron 于2023年4月23日周日 17:47写道: > > Congratulations, Qingsheng. > > Best, > Ron > > Zhanghao Chen 于2023年4月23日周日 17:32写道: > > > Congratulations, Qingsheng! > > > > Best, > > Zhanghao Chen > > > > From: Shammon FY > >

[jira] [Created] (FLINK-31876) [JUnit5 Migration] Module: flink-queryable-state

2023-04-20 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-31876: -- Summary: [JUnit5 Migration] Module: flink-queryable-state Key: FLINK-31876 URL: https://issues.apache.org/jira/browse/FLINK-31876 Project: Flink Issue Type: Sub

Re: [DISCUSS] FLIP-306: Unified File Merging Mechanism for Checkpoints

2023-04-07 Thread Yanfei Lei
s per TM as much as possible. However, for shared state, > > > > merging at the subtask level is the best choice to prevent significant > > > > data transfer over the network after restoring. I think it is better > > > > to keep these merging granularities for different

Re: [DISCUSS] FLIP-306: Unified File Merging Mechanism for Checkpoints

2023-04-03 Thread Yanfei Lei
Hi Zakelly, Thanks for driving this, this proposal enables the files merging of different types of states to be grouped under a unified framework. I think it has the added benefit of lightening the load on JM. As FLINK-26590[1] described, triggered checkpoints can be delayed by discarding

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Yanfei Lei
Congratulations! Best Regards, Yanfei ramkrishna vasudevan 于2023年3月27日周一 21:46写道: > > Congratulations !!! > > On Mon, Mar 27, 2023 at 2:54 PM Yu Li wrote: >> >> Dear Flinkers, >> >> >> As you may have noticed, we are pleased to announce that Flink Table Store >> has joined the Apache

Re: RIC Incremental and GIC Incremental checkpoint use question

2023-03-26 Thread Yanfei Lei
Hi ConradJam, > If Generic Incremental Checkpoint (GIC) enable, rocksdb Incremental > Checkpoint can be disable or enable, Do they both have conflicting switches, > does my turning on (GIC) mean I no longer need enable rocksdb Incremental > Checkpoint ? The GIC and rocksdb incremental

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

2023-03-12 Thread Yanfei Lei
Congratulations, Yuxia! Best, Yanfei Samrat Deb 于2023年3月13日周一 10:41写道: > > congratulations Yuxia > > Bests, > Samrat > > On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan wrote: > > > Congratulations, Yuxia! > > > > Best, > > Yuxin > > > > > > Jark Wu 于2023年3月13日周一 10:26写道: > > > > > Hi, everyone >

Re: [VOTE] Release 1.15.4, release candidate #1

2023-02-23 Thread Yanfei Lei
Thanks Danny. +1 (non-binding) - Downloaded artifacts & built Flink from sources - Verified GPG signatures of bin and source. - Verified version in poms - Ran WordCount example in streaming and batch mode(standalone cluster) - Went over flink-web PR, looks good except for Sergey's remark. Danny

Re: [ANNOUNCE] New Apache Flink PMC Member - Dong Lin

2023-02-16 Thread Yanfei Lei
Congratulations, Dong! Best regards, Yanfei Sergey Nuyanzin 于2023年2月16日周四 17:21写道: > > Congratulations, Dong! > > On Thu, Feb 16, 2023 at 10:01 AM Wencong Liu wrote: > > > Congratulations Dong! > > > > > > Bast, > > Wencong Liu > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

Re: [ANNOUNCE] New Apache Flink Committer - Jing Ge

2023-02-14 Thread Yanfei Lei
Congratulations, Jing Ge ! Best regards, Yanfei yuxia 于2023年2月15日周三 09:34写道: > > Congratulations, Jing Ge ! > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Jark Wu" > 收件人: "dev" > 发送时间: 星期二, 2023年 2 月 14日 下午 11:32:30 > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Jing Ge > > Welcome

Re: [ANNOUNCE] New Apache Flink Committer - Weijie Guo

2023-02-12 Thread Yanfei Lei
Congratulations, Weijie! Best regards, Yanfei Junrui Lee 于2023年2月13日周一 12:12写道: > > Congratulations, Weijie! > > Best, > Junrui

Re: How to find the root cause of e2e ci failure?

2023-02-07 Thread Yanfei Lei
Hi Rui, In this case I would search the log for `Run Kubernetes test` in https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_apis/build/builds/45813/logs/216 first, and then we can see: 2023-02-07T10:25:38.6193935Z X Exiting due to RUNTIME_ENABLE: Temporary Error: sudo

[jira] [Created] (FLINK-30890) Backup codespeed‘s data regularly

2023-02-02 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-30890: -- Summary: Backup codespeed‘s data regularly Key: FLINK-30890 URL: https://issues.apache.org/jira/browse/FLINK-30890 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-30863) Do not delete the local changelog file of aborted checkpoint

2023-02-01 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-30863: -- Summary: Do not delete the local changelog file of aborted checkpoint Key: FLINK-30863 URL: https://issues.apache.org/jira/browse/FLINK-30863 Project: Flink

Re: [DISCUSS] Incorporate performance regression monitoring into routine process

2023-01-31 Thread Yanfei Lei
> > this > > > process by e.g. exporting the nightly benchmark results to a database and > > > using scripts to detect regression based on pre-defined rules? > > > > > > This approach is probably more scalable and accurate in the long term. > > And > > > I

[jira] [Created] (FLINK-30836) Release FRocksDB 6.20.3-ververica-2.0

2023-01-30 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-30836: -- Summary: Release FRocksDB 6.20.3-ververica-2.0 Key: FLINK-30836 URL: https://issues.apache.org/jira/browse/FLINK-30836 Project: Flink Issue Type: Improvement

[ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

2023-01-30 Thread Yanfei Lei
It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0. Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86, MacOS arm, and Windows are included in FRocksDB 6.20.3-ververica-2.0 jar, and the FRocksDB in Flink 1.17 would be updated to 6.20.3-ververica-2.0. Release

Re: [DISCUSS] Release new FRocksDB

2023-01-29 Thread Yanfei Lei
to > > > pass on Mac M1s. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi > > > > > wrote: > > > > > > > Hi Yanfei, > > > > > > > > Makes sen

[DISCUSS] Incorporate performance regression monitoring into routine process

2023-01-19 Thread Yanfei Lei
Hi devs, I'd like to start a discussion about incorporating performance regression monitoring into the routine process. Flink benchmarks are periodically executed on http://codespeed.dak8s.net:8080 to monitor Flink performance. In late Oct'22, a new slack channel #flink-dev-benchmarks was created

Re: [Discuss] Checkout the CURRENT_SNAPSHOT_VERSION branch of the flink-benchmarks repo during each major release

2023-01-16 Thread Yanfei Lei
Hi Yun, Thanks for kicking off this discussion, +1 for the proposal. In the past, we occasionally encountered flink-benchmarks compilation failures caused by the code updates in flink repo, like FLINK-28931[1]. Managing the flink-benchmarks repository by release branch can help us handle these

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

2023-01-15 Thread Yanfei Lei
rg/jira/browse/FLINK-30625 [8] https://github.com/apache/flink-benchmarks [9] https://issues.apache.org/jira/browse/FLINK-18614 [10] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847 Best regards, Yanfei Ververica(Alibaba) Yanfei Lei 于2023年1月12日周四 17:46写道: > > Hi

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

2023-01-12 Thread Yanfei Lei
; > > > Alternatively, we could leave everything as is and just respond if > there's > > some complaint. I'm curious about your (and other's) opinions. > > > > Matthias > > > > On Tue, Nov 29, 2022 at 2:13 PM Yanfei Lei wrote: > > > > > Hi

Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee

2023-01-09 Thread Yanfei Lei
Congratulations, well deserved! Best, Yanfei Yuan Mei 于2023年1月10日周二 13:16写道: > Congratulations, Lincoln! > > Best, > Yuan > > On Tue, Jan 10, 2023 at 12:23 PM Lijie Wang > wrote: > > > Congratulations, Lincoln! > > > > Best, > > Lijie > > > > Jingsong Li 于2023年1月10日周二 12:07写道: > > > > >

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

2022-12-25 Thread Yanfei Lei
Hi Yuxin, Thanks for the proposal! After reading the FLIP, I have some questions about the default value. This FLIP seems to introduce a *new* config option(taskmanager.memory.network.required-buffer-per-gate.max) to control the network memory usage. 1. Is this configuration at the job level or

[DISCUSS] Release new FRocksDB

2022-12-12 Thread Yanfei Lei
Hi devs, I'd like to bring up a discussion about releasing the new frocksdbjni version, we are planning to adapt frocksdbjni to Apple M1 machines in Flink 1.17[1], a new frocksdbjni jar needs to be released. Thanks to the efforts of the community, FLINK-24932[1] has been merged into

[jira] [Created] (FLINK-30321) Upgrade ZLIB of FRocksDB to 1.2.13

2022-12-07 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-30321: -- Summary: Upgrade ZLIB of FRocksDB to 1.2.13 Key: FLINK-30321 URL: https://issues.apache.org/jira/browse/FLINK-30321 Project: Flink Issue Type: Bug

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

2022-11-29 Thread Yanfei Lei
gt; > > From: Hangxiang Yu > > > > > > Sent: Thursday, October 27, 2022 10:59 > > > > > > To: dev@flink.apache.org > > > > > > Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved from > > > > Ververica > > > > &

[jira] [Created] (FLINK-30181) MultiInputCheckpointingTimeBenchmark.checkpointMultiInput benchmark failed

2022-11-23 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-30181: -- Summary: MultiInputCheckpointingTimeBenchmark.checkpointMultiInput benchmark failed Key: FLINK-30181 URL: https://issues.apache.org/jira/browse/FLINK-30181 Project

Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He, Xingbo Huang

2022-11-23 Thread Yanfei Lei
Congrats,Godfrey and Xingbo! Best, Yanfei Yang Wang 于2022年11月24日周四 09:32写道: > Congratulations, Godfrey and Xingbo! > > Best, > Yang > > Jing Ge 于2022年11月24日周四 02:06写道: > > > Congrats, Godfrey! Congrats, Xingbo! > > > > Best regards, > > Jing > > > > On Wed, Nov 23, 2022 at 6:11 PM Maximilian

Re: FRocksDB Some Question

2022-11-21 Thread Yanfei Lei
Hi ConradJam, Thanks for bringing this up. In fact, @Sebastian has tried to update RocksDB to 6.29.5[1], but there are some regressions on MapState[2]. If you want to use the newest feature of RocksDB, I think you can refer to this release document[3] to release your own version. > I would like

Re: [ANNOUNCE] New Apache Flink Committer - Matyas Orhidi

2022-11-21 Thread Yanfei Lei
Congrats Matyas!  Zheng Yu Chen 于2022年11月22日周二 11:15写道: > Congratulations ~  > > Márton Balassi 于2022年11月21日周一 22:18写道: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce Matyas Orhidi as a new > > Flink > > committer. > > > > Matyas has over a decade of experience

Re: [DISCUSS] FLIP-268: Rack Awareness for Kafka Sources

2022-11-10 Thread Yanfei Lei
Hi Jeremy, Thanks for the proposal. I'm curious if this feature can adapt to the current kafka-client version in the connector. The rack awareness feature requires a version>=3.2.0 to use, while the kafka-client version of Kafka connector is 2.8.1. Will we upgrade the Kafka-client version first?

Re: [DISCUSS] Allow sharing (RocksDB) memory between slots

2022-11-08 Thread Yanfei Lei
Hi Roman, Thanks for the proposal, this allows State Backend to make better use of memory. After reading the ticket, I'm curious about some points: 1. Is shared-memory only for the state backend? If both "taskmanager.memory.managed.shared-fraction: >0" and "state.backend.rocksdb.memory.managed:

[jira] [Created] (FLINK-29913) Shared state would be discarded by mistake when maxConcurrentCheckpoint>1

2022-11-06 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-29913: -- Summary: Shared state would be discarded by mistake when maxConcurrentCheckpoint>1 Key: FLINK-29913 URL: https://issues.apache.org/jira/browse/FLINK-29913 Project: Fl

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-06 Thread Yanfei Lei
Hi Max, Thanks for the proposal. This proposal makes Flink better adapted to cloud-native applications! After reading the FLIP, I'm curious about some points: 1) It's said that "The first step is collecting metrics for all JobVertices by combining metrics from all the runtime subtasks and

[jira] [Created] (FLINK-29825) Improve benchmark stablility

2022-10-31 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-29825: -- Summary: Improve benchmark stablility Key: FLINK-29825 URL: https://issues.apache.org/jira/browse/FLINK-29825 Project: Flink Issue Type: Improvement

Re: [VOTE] FLIP-263: Improve resolving schema compatibility

2022-10-27 Thread yanfei lei
+1(non-binding) and thanks for Hangxiang's driving. Hangxiang Yu 于2022年10月28日周五 09:24写道: > Hi everyone, > > I'd like to start the vote for FLIP-263 [1]. > > Thanks for your feedback and the discussion in [2][3]. > > The vote will be open for at least 72 hours. > > Best regards, > Hangxiang. >

Re: [DISCUSS] Delete useless branches from Flink repository

2022-10-27 Thread yanfei lei
Hi Leonard, Thanks for driving this, +1 for removing useless branches, this would make the git tree cleaner. And first we need to identify which branches are really useless. Leonard Xu 于2022年10月27日周四 16:12写道: > Hi, all > > Recently I checked our repo and founded there are some useless >

  1   2   >