Re: [VOTE] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-26 Thread Lijie Wang
+1 (binding) Best, Lijie Zhu Zhu 于2024年4月26日周五 14:47写道: > +1 (binding) > > Thanks, > Zhu > > Ron Liu 于2024年4月26日周五 13:11写道: > > > +1(binding) > > > > Best, > > Ron > > > > Rui Fan <1996fan...@gmail.com> 于2024年4月26日周五 12:55写道: > > > > > +1(binding) > > > > > > Best, > > > Rui > > > > > > On

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-17 Thread Lijie Wang
Thanks for driving the discussion. +1 for the proposal and +1 for the `InferMode.NONE` option. Best, Lijie Ron liu 于2024年4月18日周四 11:36写道: > Hi, Xia > > Thanks for driving this FLIP. > > This proposal looks good to me overall. However, I have the following minor > questions: > > 1. FLIP

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-12 Thread Lijie Wang
Congratulations Lincoln ! Best, Lijie 于2024年4月12日周五 18:16写道: > Congratulations Lincoln! > On Apr 12, 2024 at 12:08 +0200, yuxia , > wrote: > > Congratulations, Lincoln! > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "Feng Jin" > > 收件人: "dev" > > 发送时间: 星期五, 2024年 4 月 12日 下午

Re: [VOTE] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

2024-04-11 Thread Lijie Wang
+1 (binding) Thanks for driving. Best, Lijie Zakelly Lan 于2024年4月12日周五 11:08写道: > +1 non-binding > > > Best, > Zakelly > > On Fri, Apr 12, 2024 at 11:05 AM Yuepeng Pan > wrote: > > > Hi Rui, > > > > Thanks for driving it! > > +1 (non-binding) > > Best, > > Yuepeng Pan > > > > At 2024-04-12

Re: Question around Flink's AdaptiveBatchScheduler

2024-02-29 Thread Lijie Wang
Hi Venkat, >> default-source-parallelism config should be independent from the max-parallelism Actually, it's not. Firstly, it's obvious that the parallelism should be less than or equal to the max parallelism(both literally and execution). The

Re: [VOTE] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-24 Thread Lijie Wang
+1 (binding) Best, Lijie Yanfei Lei 于2024年1月25日周四 10:06写道: > +1 (binding) > > Hangxiang Yu 于2024年1月25日周四 10:00写道: > > > > +1 (binding) > > > > On Thu, Jan 25, 2024 at 8:49 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > +1(binding) > > > > > > Best, > > > Rui > > > > > > On Wed, 24 Jan

Re: [DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-01-17 Thread Lijie Wang
Hi Mason, Thanks for driving the discussion. +1 for the proposal. How about we return all operator metrics in a vertex? (the response is a map of operatorId/operatorName -> operator-metrics). Correspondingly, the url may be changed to /jobs//vertices//operator-metrics In this way, users can

Re: Re: [VOTE] Accept Flink CDC into Apache Flink

2024-01-09 Thread Lijie Wang
+1 (non-binding) Best, Lijie Jiabao Sun 于2024年1月9日周二 19:28写道: > +1 (non-binding) > > Best, > Jiabao > > > On 2024/01/09 09:58:04 xiangyu feng wrote: > > +1 (non-binding) > > > > Regards, > > Xiangyu Feng > > > > Danny Cranmer 于2024年1月9日周二 17:50写道: > > > > > +1 (binding) > > > > > > Thanks, >

Re: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov

2024-01-02 Thread Lijie Wang
Congratulations Alex ! Best, Lijie Romit Mahanta 于2024年1月3日周三 13:41写道: > Happy New Year & congratulations Alex! > > Best, > > R > > On Tue, 2 Jan, 2024, 5:45 pm Maximilian Michels, wrote: > > > Happy New Year everyone, > > > > I'd like to start the year off by announcing Alexander Fedulov as

[jira] [Created] (FLINK-33968) Compute the number of subpartitions when initializing executon job vertices

2024-01-02 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-33968: -- Summary: Compute the number of subpartitions when initializing executon job vertices Key: FLINK-33968 URL: https://issues.apache.org/jira/browse/FLINK-33968 Project

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2023-12-26 Thread Lijie Wang
; checkpoint instead of enabling a function. So I think an enumeration option > `execution.checkpointing.mode` which can be 'full' (default) or > 'incremental' would be better, WDYT? > And @Rui Fan @Yanfei What do you think about this? > > > On Tue, Dec 26, 2023 at 5:1

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2023-12-26 Thread Lijie Wang
Hi Zakelly, Thanks for driving the discussion. 1. >> But I'm not so sure since there is only one savepoint-related option. Maybe someone else could share some thoughts here. How about state.savepoints.dir -> execution.checkpointing.savepoint.dir state.checkpoints.dir ->

[jira] [Created] (FLINK-33892) FLIP-383: Support Job Recovery for Batch Jobs

2023-12-19 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-33892: -- Summary: FLIP-383: Support Job Recovery for Batch Jobs Key: FLINK-33892 URL: https://issues.apache.org/jira/browse/FLINK-33892 Project: Flink Issue Type: New

[RESULT][VOTE] FLIP-383: Support Job Recovery for Batch Jobs

2023-12-19 Thread Lijie Wang
Hi devs, I'm happy to announce that FLIP-383 [1] has been approved unanimously. According to the vote thread[2], there are 5 approving votes, 4 of which are binding: Xintong Song (binding) Zhu Zhu (binding) Weijie Guo (binding) Rui Fan (binding) Yuepeng Pan (non-binding) [1]

Re: [VOTE] FLIP-382: Unify the Provision of Diverse Metadata for Context-like APIs

2023-12-17 Thread Lijie Wang
+1 (binding) Best, Lijie Yuxin Tan 于2023年12月15日周五 17:14写道: > +1 (non-binding) > > Best, > Yuxin > > > weijie guo 于2023年12月15日周五 10:05写道: > > > +1(binding) > > > > Best regards, > > > > Weijie > > > > > > Wencong Liu 于2023年12月15日周五 09:13写道: > > > > > Hi dev, > > > > > > I'd like to start a

[VOTE] FLIP-383: Support Job Recovery for Batch Jobs

2023-12-13 Thread Lijie Wang
Hi devs, Thanks for all feedback about the FLIP-383: Support Job Recovery for Batch Jobs[1]. This FLIP was discussed in [2]. I'd like to start a vote for it. The vote will be open for at least 72 hours (until December 19th 12:00 GMT) unless there is an objection or insufficient votes. [1]

Re: [DISCUSS][2.0] FLIP-382: Unify the Provision of Diverse Metadata for Context-like APIs

2023-12-11 Thread Lijie Wang
Hi Wencong Thanks for driving the discussion, +1 for the proposal. I left two minor questions/suggestions: 1. Is the getTaskNameWithIndexAndAttemptNumber method a bit redundant? It can be replaced by getTaskName + getTaskIndex + getAttemptNumber. 2. I think it would be better if we can

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

2023-12-11 Thread Lijie Wang
n the internal state is too large, to make the snapshot operations faster and occupy less storage. See FLIP "JobEvent" section for details. By the way, if there are no more questions, we will start voting tomorrow. Best, Lijie Lijie Wang 于2023年12月5日周二 22:57写道: > Hi Paul, > &

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

2023-12-05 Thread Lijie Wang
mission to upload and run jars which may produce multiple > >> Flink jobs. However, these jobs may not be submitted at once and run in > >> parallel. Instead, they could be dependent on other jobs like a DAG. The > >> schedule of the jobs is controlled by the user's main me

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

2023-12-04 Thread Lijie Wang
from Shuffle > Master once a while, and provide such snapshots to Shuffle Master upon > recovery? > > Best, > > Xintong > > > > On Thu, Nov 30, 2023 at 5:48 PM Lijie Wang > wrote: > > > Hi Guowei, > > > > Thanks for your feedback. > > &g

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

2023-11-30 Thread Lijie Wang
use a normal > checkpoint ID greater than 0 and record it in the event store? > If the issues raised in point 2 cannot be resolved in the short term, would > it be possible to consider not supporting failover with a source job > manager? > > Best, > Guowei > > > On

Re: [VOTE] FLIP-379: Dynamic source parallelism inference for batch jobs

2023-11-29 Thread Lijie Wang
+1 (binding) Best, Lijie Zhu Zhu 于2023年11月30日周四 13:13写道: > +1 > > Thanks, > Zhu > > Xia Sun 于2023年11月30日周四 11:41写道: > > > Hi everyone, > > > > I'd like to start a vote on FLIP-379: Dynamic source parallelism > inference > > for batch jobs[1] which has been discussed in this thread [2]. > > >

Re: [VOTE] FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects

2023-11-12 Thread Lijie Wang
+1 (binding) Best, Lijie Yuepeng Pan 于2023年11月10日周五 18:32写道: > +1(non-binding) > > Best, > Roc > > On 2023/11/10 03:58:10 Junrui Lee wrote: > > Hi everyone, > > > > Thank you to everyone for the feedback on FLIP-381: Deprecate > configuration > > getters/setters that return/set complex Java

Re: [DISCUSS] FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects

2023-11-02 Thread Lijie Wang
Thanks Junrui for driving this. Making configurations simple and consistent has great benefits for both users and devs. +1 for the proposal. Best, Lijie weijie guo 于2023年11月2日周四 16:49写道: > Thanks Junrui for driving this proposal! > > I believe this is helpful for the new Process Function API.

[DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

2023-11-02 Thread Lijie Wang
Hi devs, Zhu Zhu and I would like to start a discussion about FLIP-383: Support Job Recovery for Batch Jobs[1] Currently, when Flink’s job manager crashes or gets killed, possibly due to unexpected errors or planned nodes decommission, it will cause the following two situations: 1. Failed, if

Re: [DISCUSS] FLIP-379: Dynamic source parallelism inference for batch jobs

2023-11-01 Thread Lijie Wang
t; adaptive batch scheduler to dynamically infer the source parallelism at > > runtime. Please refer to the FLIP[1] document for more details about the > > proposed design and implementation. > > > > I also thank Zhu Zhu and LiJie Wang for their suggestions during the > &g

Re: [VOTE] Release 1.18.0, release candidate #2

2023-10-16 Thread Lijie Wang
+1 (non-binding) - Verified the signature and checksum - Built from the source code - Ran an example job on yarn cluster - Checked the website PR Best, Lijie Jing Ge 于2023年10月16日周一 18:43写道: > Hi everyone, > > Please review and vote on the release candidate #2 for the version > 1.18.0,

Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan

2023-10-15 Thread Lijie Wang
Congratulations Jane ! Best, Lijie Samrat Deb 于2023年10月16日周一 10:04写道: > Congratulations Jane Chan :) > > On Mon, 16 Oct 2023 at 7:30 AM, tison wrote: > > > Congrats! I noticed Jane has been around for a while; well-deserved. > > > > Best, > > tison. > > > > > > Jark Wu 于2023年10月16日周一

Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

2023-10-15 Thread Lijie Wang
Congratulations Ron ! Best, Lijie Samrat Deb 于2023年10月16日周一 10:03写道: > Congratulations Ron Liu :) > > On Mon, 16 Oct 2023 at 7:29 AM, tison wrote: > > > Congrats! Glad to see more and more committers on board :D > > > > Enjoy your journey ;-) > > > > Best, > > tison. > > > > > > Jark Wu

Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

2023-10-12 Thread Lijie Wang
+1 (binding) Best, Lijie Zhanghao Chen 于2023年10月13日周五 10:56写道: > +1 (non-binding) > > Best, > Zhanghao Chen > > From: Junrui Lee > Sent: Friday, October 13, 2023 10:12 > To: dev@flink.apache.org > Subject: [VOTE] FLIP-366: Support standard YAML for FLINK

Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

2023-09-21 Thread Lijie Wang
Hi Junrui, +1 for this proposal, thanks for driving. Best, Lijie ConradJam 于2023年9月22日周五 10:07写道: > +1 Support for standard YAML format facilitates specification > > Jing Ge 于2023年9月22日周五 02:23写道: > > > Hi Junrui, > > > > +1 for following the standard. Thanks for your effort! > > > > Best

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

2023-09-20 Thread Lijie Wang
Hi community and release managers: We found a critical bug[1] of the rest client a few days ago, which may cause the inode to be used up. Now the fix-PR[2] is ready for merging, I hope to backport it to release-1.18. Please let me know if you have any concerns. Thanks. [1]

[jira] [Created] (FLINK-32974) RestClusterClient leaks flink-rest-client-jobgraphs* directories

2023-08-28 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-32974: -- Summary: RestClusterClient leaks flink-rest-client-jobgraphs* directories Key: FLINK-32974 URL: https://issues.apache.org/jira/browse/FLINK-32974 Project: Flink

[jira] [Created] (FLINK-32844) Runtime Filter should not be applied if the field is already filtered by DPP

2023-08-11 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-32844: -- Summary: Runtime Filter should not be applied if the field is already filtered by DPP Key: FLINK-32844 URL: https://issues.apache.org/jira/browse/FLINK-32844 Project

[jira] [Created] (FLINK-32831) RuntimeFilterProgram should aware join type when looking for the build side

2023-08-10 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-32831: -- Summary: RuntimeFilterProgram should aware join type when looking for the build side Key: FLINK-32831 URL: https://issues.apache.org/jira/browse/FLINK-32831 Project

[jira] [Created] (FLINK-32827) Operator fusion codegen may not take effect when enable runtime filter

2023-08-10 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-32827: -- Summary: Operator fusion codegen may not take effect when enable runtime filter Key: FLINK-32827 URL: https://issues.apache.org/jira/browse/FLINK-32827 Project: Flink

Re: [ANNOUNCE] New Apache Flink Committer - Hong Teoh

2023-08-04 Thread Lijie Wang
Congratulations, Hong! Best, Lijie Paul Lam 于2023年8月4日周五 15:18写道: > Congrats, Hong! > > Best, > Paul Lam > > > 2023年8月4日 14:59,Jark Wu 写道: > > > > Congratulations, Hong! > > > > Best, > > Jark > > > > On Fri, 4 Aug 2023 at 14:24, Sergey Nuyanzin > wrote: > > > >> Congratulations, Hong! > >>

Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl

2023-08-04 Thread Lijie Wang
Congratulation, Matthias! Best, Lijie Konstantin Knauf 于2023年8月4日周五 15:27写道: > Congrats, Matthias! > > Am Fr., 4. Aug. 2023 um 09:15 Uhr schrieb Paul Lam >: > > > Congratulation, Matthias! > > > > Best, > > Paul Lam > > > > > 2023年8月4日 15:09,yuxia 写道: > > > > > > Congratulation, Matthias! >

Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu

2023-08-04 Thread Lijie Wang
Congratulations, Weihua! Best, Lijie yuxia 于2023年8月4日周五 15:14写道: > Congratulations, Weihua! > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Yun Tang" > 收件人: "dev" > 发送时间: 星期五, 2023年 8 月 04日 下午 3:05:30 > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu > > Congratulations,

[jira] [Created] (FLINK-32680) Job vertex names get messed up once there is a source vertex chained with a MultipleInput vertex in job graph

2023-07-26 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-32680: -- Summary: Job vertex names get messed up once there is a source vertex chained with a MultipleInput vertex in job graph Key: FLINK-32680 URL: https://issues.apache.org/jira/browse

[jira] [Created] (FLINK-32492) Introduce FlinkRuntimeFilterProgram to inject runtime filter

2023-06-29 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-32492: -- Summary: Introduce FlinkRuntimeFilterProgram to inject runtime filter Key: FLINK-32492 URL: https://issues.apache.org/jira/browse/FLINK-32492 Project: Flink

[jira] [Created] (FLINK-32486) FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-29 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-32486: -- Summary: FLIP-324: Introduce Runtime Filter for Flink Batch Jobs Key: FLINK-32486 URL: https://issues.apache.org/jira/browse/FLINK-32486 Project: Flink Issue

[RESULT][VOTE] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-29 Thread Lijie Wang
Hi, all Happy to announce that FLIP-324 [1] has been approved unanimously. According to the vote thread[2], there are 10 approving votes, out of which 7 are binding: - Jing Ge (binding) - Rui Fan (binding) - Zhu Zhu (binding) - Yuepeng Pan (non-binding) - Yuxia (binding) - Xia Sun (non-binding) -

Re: [VOTE] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-29 Thread Lijie Wang
t. > > > > >> > > > > >> Best regards, > > > > >> Yuxia > > > > >> > > > > >> - 原始邮件 - > > > > >> 发件人: "Yuepeng Pan" > > > > >> 收件人: "dev" > > > > >> 发送时间: 星期六, 2023年 6 月 24日 下午 9:06:53 > > > >

[VOTE] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-23 Thread Lijie Wang
Hi all, Thanks for all the feedback about the FLIP-324: Introduce Runtime Filter for Flink Batch Jobs[1]. This FLIP was discussed in [2]. I'd like to start a vote for it. The vote will be open for at least 72 hours (until June 29th 12:00 GMT) unless there is an objection or insufficient votes.

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-22 Thread Lijie Wang
; > this correct? > > > > > > > > > > Speaking of the "Conditions of injecting Runtime Filter" in the > FLIP, > > > > will > > > > > the value of max-build-data-size and min-prob-data-size depend on > the > > >

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-20 Thread Lijie Wang
that the large table might have more columns > > than the small table, the probeNdv should still be 100 or 10 times > > buildNdv, which ends up with a min-filter-ratio equals to 0.99 or 0.9. > Both > > are bigger than the default value 0.5 in the FLIP. If I am not mistaken, > > co

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-19 Thread Lijie Wang
s, I’m fine with excluding this optimization from the current FLIP > if you believe it would be hard to implement in Flink. > > Best, > Stefan > > > > On 19. Jun 2023, at 14:07, Lijie Wang wrote: > > > > Hi Stefan, > > > > If I understand correctly(I h

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-19 Thread Lijie Wang
sport the filter and coordination to tell the sources > that such a filter is available. I cannot answer how hard those would be to > implement, but the idea doesn’t seem impossible to me. > > Best, > Stefan > > > > On 19. Jun 2023, at 11:56, Lijie Wang wrote: >

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-19 Thread Lijie Wang
For example, if you have min/max > information about columns and partitions you could have a bit vector > represent equilibrium-sized ranges of the key space between min and max and > have the bits represent what part of the range is present and push that > information down to the scan.

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-19 Thread Lijie Wang
huffle" is a essential for OLAP/MPP computing, although this > >> has > >>> not been much exposed to users for now, I know a few companies that > uses > >>> Flink as a MPP computing engine, and there is an ongoing effort[1] to > >> make > >

Re: [VOTE] FLIP-287: Extend Sink#InitContext to expose TypeSerializer, ObjectReuse and JobID

2023-06-16 Thread Lijie Wang
+1 (binding) Thanks for driving it, Joao. Best, Lijie Joao Boto 于2023年6月16日周五 15:53写道: > Hi all, > > Thank you to everyone for the feedback on FLIP-287[1]. Based on the > discussion thread [2], we have come to a consensus on the design and are > ready to take a vote to contribute this to

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-15 Thread Lijie Wang
ill > > occupy resources", whether it benefits us depends on the scale of data, > if > > the RuntimeFIlterBuilder could be done quickly than RuntimeFilter > operator, > > it can still filter out additional data afterwards. Hence in my opinion, > we > > do not n

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-15 Thread Lijie Wang
, it will not process any data and will occupy resources. Best, Lijie Lijie Wang 于2023年6月15日周四 09:48写道: > Hi Yuxia, > > Thanks for your feedback. The answers of your questions are as follows: > > 1. Yes, the row count comes from statistic of underlying table(Or > estimated based on the stati

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-14 Thread Lijie Wang
e continuously optimizing the performance of batches. > Runtime filter is a general performance optimization technique that can > improve the performance of Flink batch jobs, so we are introducing it on > batch as well. > > Looking forward to all feedback. > > Be

[DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-14 Thread Lijie Wang
Hi devs Ron Liu, Gen Luo and I would like to start a discussion about FLIP-324: Introduce Runtime Filter for Flink Batch Jobs[1] Runtime Filter is a common optimization to improve join performance. It is designed to dynamically generate filter conditions for certain Join queries at runtime to

[jira] [Created] (FLINK-32244) flink-master-benchmarks-regression-check always fails since 2023.05.30

2023-06-01 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-32244: -- Summary: flink-master-benchmarks-regression-check always fails since 2023.05.30 Key: FLINK-32244 URL: https://issues.apache.org/jira/browse/FLINK-32244 Project: Flink

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

2023-05-23 Thread Lijie Wang
Hi Joao, I noticed the FLIP currently contains the following 2 methods about type serializer: (1) TypeSerializer createInputSerializer(); (2) TypeSerializer createSerializer(TypeInformation inType); Is the method (2) still needed now? Best, Lijie João Boto 于2023年5月19日周五 16:53写道: > Updated

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

2023-05-17 Thread Lijie Wang
Hi, +1 for `InitContext#createInputSerializer()` . I think we can get the serializer directly in InitContextImpl through `getOperatorConfig().getTypeSerializerIn(0, getUserCodeClassloader()).duplicate()`. Best, Lijie. João Boto 于2023年4月24日周一 19:04写道: > Hi @Gordon, > >

Re: [DISCUSS] FLIP-229: Prometheus Sink Connector

2023-05-15 Thread Lijie Wang
Hi Karthi, I think you are using a wrong FLIP id, the FLIP-229 has already be used[1]. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job Best, Lijie Martijn Visser 于2023年5月16日周二 04:44写道: > Hi Karthi, > > Thanks for the FLIP and

Re: [DISCUSS] Release Flink 1.17.1

2023-05-15 Thread Lijie Wang
+1 for the release. Best, Lijie Jing Ge 于2023年5月15日周一 17:07写道: > +1 for releasing 1.17.1 > > Best Regards, > Jing > > On Thu, May 11, 2023 at 10:03 AM Martijn Visser > wrote: > > > +1, thanks for volunteering! > > > > On Thu, May 11, 2023 at 9:23 AM Xintong Song > > wrote: > > > > > +1 > > >

[jira] [Created] (FLINK-32048) DecimalITCase.testAggMinGroupBy fails with "Insufficient number of network buffers"

2023-05-10 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-32048: -- Summary: DecimalITCase.testAggMinGroupBy fails with "Insufficient number of network buffers" Key: FLINK-32048 URL: https://issues.apache.org/jira/browse/F

Re: [DISCUSS] Preventing Mockito usage for the new code with Checkstyle

2023-04-25 Thread Lijie Wang
Thanks for driving this. +1 for the proposal. Can we also prevent Junit4 usage in new code by this way?Because currently we are aiming to migrate our codebase to JUnit 5. Best, Lijie Piotr Nowojski 于2023年4月25日周二 23:02写道: > Ok, thanks for the clarification. > > Piotrek > > wt., 25 kwi 2023 o

[jira] [Created] (FLINK-31651) Improve logging of granting/revoking leadership in JobMasterServiceLeadershipRunner to INFO level

2023-03-28 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-31651: -- Summary: Improve logging of granting/revoking leadership in JobMasterServiceLeadershipRunner to INFO level Key: FLINK-31651 URL: https://issues.apache.org/jira/browse/FLINK-31651

[jira] [Created] (FLINK-31650) Incorrect busyMsTimePerSecond metric value for FINISHED task

2023-03-28 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-31650: -- Summary: Incorrect busyMsTimePerSecond metric value for FINISHED task Key: FLINK-31650 URL: https://issues.apache.org/jira/browse/FLINK-31650 Project: Flink

Re: [DISCUSS] Use 1.15.4 as the final patch version of Flink 1.15 and drop 1.15 support

2023-03-24 Thread Lijie Wang
Thanks for driving this, +1 for dropping 1.15 support. Best, Lijie Leonard Xu 于2023年3月24日周五 17:58写道: > +1 as 1.17.0 is available. > > Best, > Leonard > > > On Mar 24, 2023, at 5:50 PM, Jing Ge wrote: > > > > Thanks Qingsheng for driving this. +1 since there are no changes after > 1.15.4 > > >

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

2023-03-20 Thread Lijie Wang
Hi Panagiotis, Thanks for driving this. +1 for supporting custom restart strategy, we did receive such requests from the user mailing list [1][2]. Besides, in current design, the plugin will only do some statistical and classification work, and will not affect the *FailureHandlingResult*. Just

Re: [VOTE] Release 1.17.0, release candidate #3

2023-03-18 Thread Lijie Wang
+1 (binding) - Built from the source code - Verified the signature and checksum - Ran both streaming/batch jobs on yarn cluster - Checked the website PR Best, Lijie Qingsheng Ren 于2023年3月17日周五 22:02写道: > Hi everyone, > > Please review and vote on the release candidate #3 for the

[jira] [Created] (FLINK-31114) Batch job fails with IllegalStateException when using adaptive batch scheduler

2023-02-16 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-31114: -- Summary: Batch job fails with IllegalStateException when using adaptive batch scheduler Key: FLINK-31114 URL: https://issues.apache.org/jira/browse/FLINK-31114 Project

[jira] [Created] (FLINK-31079) Release Testing: Verify FLINK-29663 Further improvements of adaptive batch scheduler

2023-02-14 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-31079: -- Summary: Release Testing: Verify FLINK-29663 Further improvements of adaptive batch scheduler Key: FLINK-31079 URL: https://issues.apache.org/jira/browse/FLINK-31079

[jira] [Created] (FLINK-31071) Release Testing: Verify FLIP-283 Use adaptive batch scheduler as default scheduler for batch jobs

2023-02-14 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-31071: -- Summary: Release Testing: Verify FLIP-283 Use adaptive batch scheduler as default scheduler for batch jobs Key: FLINK-31071 URL: https://issues.apache.org/jira/browse/FLINK-31071

Re: [ANNOUNCE] New Apache Flink Committer - Jing Ge

2023-02-14 Thread Lijie Wang
Congratulations, Jing Ge ! Best, Lijie Sergey Nuyanzin 于2023年2月14日周二 21:47写道: > Congratulations, Jing Ge! > > On Tue, Feb 14, 2023 at 2:47 PM Rui Fan wrote: > > > Congratulations, Jing! > > > > Best, > > Rui Fan > > > > On Tue, Feb 14, 2023 at 19:36 Yuepeng Pan wrote: > > > > >

[jira] [Created] (FLINK-31055) The dynamic flag of stream graph does not take effect when translating the transformations

2023-02-13 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-31055: -- Summary: The dynamic flag of stream graph does not take effect when translating the transformations Key: FLINK-31055 URL: https://issues.apache.org/jira/browse/FLINK-31055

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

2023-02-12 Thread Lijie Wang
ady use the Sink2 API. How were those implemented without > exposing the ExecutionConfig? > > Best, > > Konstantin > > > Am Mi., 1. Feb. 2023 um 12:28 Uhr schrieb Lijie Wang < > wangdachui9...@gmail.com>: > > > +1 for Option 2, if we can abstract an "Re

Re: [ANNOUNCE] New Apache Flink Committer - Weijie Guo

2023-02-12 Thread Lijie Wang
Congratulations, Weijie! Best, Lijie Xintong Song 于2023年2月13日周一 11:45写道: > Hi everyone, > > On behalf of the PMC, I'm very happy to announce Weijie Guo as a new Flink > committer. > > Weijie has been consistently contributing to the project for over 1 year. > He mainly works on Flink runtime

[jira] [Created] (FLINK-30972) E2e tests always fail in phase "Prepare E2E run"

2023-02-08 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-30972: -- Summary: E2e tests always fail in phase "Prepare E2E run" Key: FLINK-30972 URL: https://issues.apache.org/jira/browse/FLINK-30972 Project: Flink Issue

[jira] [Created] (FLINK-30942) Fix the bug that the decided parallelism by adaptive batch scheduler may be larger than the max parallelism

2023-02-07 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-30942: -- Summary: Fix the bug that the decided parallelism by adaptive batch scheduler may be larger than the max parallelism Key: FLINK-30942 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-30917) The user configured max parallelism does not take effect when using adaptive batch scheduler

2023-02-06 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-30917: -- Summary: The user configured max parallelism does not take effect when using adaptive batch scheduler Key: FLINK-30917 URL: https://issues.apache.org/jira/browse/FLINK-30917

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

2023-02-01 Thread Lijie Wang
be a pros) > > > What option you think is better? > > > > On 2023/01/13 14:15:04 Joao Boto wrote: > > Hi flink devs, > > > > I'd like to start a discussion thread for FLIP-287[1]. > > This comes from an offline discussion with @Lijie Wang, from FLIP-239[2]

Re: [SUMMARY] Flink 1.17 release sync 23rd of January, 2023

2023-01-26 Thread Lijie Wang
fix would take something like more than a week after the > 31st, then it makes more sense to revert the commit that introduced the > regression. > > Let me know what you think. > > Best regards, > > Martijn > > Op do 26 jan. 2023 om 01:11 schreef Lijie Wang : > &

Re: [SUMMARY] Flink 1.17 release sync 23rd of January, 2023

2023-01-25 Thread Lijie Wang
Hi Martijn, I'm working on FLINK-30624, and it may take a while to be resolved. Do you mean we should resolve it before the 26th? I used to think the deadline was the 31st(the date of feature freeze). Best, Lijie Martijn Visser 于2023年1月25日周三 18:07写道: > Hi everyone, > > A summary of the

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

2023-01-18 Thread Lijie Wang
Hi Martijn, Thanks for driving this. I have a only concern about the Sink.InitContext. Does the Sink.InitContext will also be changed to @Public ? As described in FLIP-287, currently the Sink.InitContext still lacks some necessary information to migrate existing connectors to new sinks. If it is

Re: [Discuss] Checkout the CURRENT_SNAPSHOT_VERSION branch of the flink-benchmarks repo during each major release

2023-01-17 Thread Lijie Wang
+1 for the proposal Best, Lijie Martijn Visser 于2023年1月17日周二 21:54写道: > Hi Yun Tang, > > +1 for the proposal. Thanks for driving this. > > Best regards, > > Martijn > > Op ma 16 jan. 2023 om 12:29 schreef Yanfei Lei : > > > Hi Yun, > > > > Thanks for kicking off this discussion, +1 for the

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

2023-01-16 Thread Lijie Wang
. Later > > > > on, other contracts which effectively shouldn't modify a config could > > > > use that one, too. > > > > > > > > Note I don't mean to stall your efforts here, but I thought it'd be a > > > > good idea to bring it up a

[jira] [Created] (FLINK-30670) Ignore broadcast bytes when computing parallelism and input infos

2023-01-12 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-30670: -- Summary: Ignore broadcast bytes when computing parallelism and input infos Key: FLINK-30670 URL: https://issues.apache.org/jira/browse/FLINK-30670 Project: Flink

Re: [VOTE] FLIP-281: Sink Supports Speculative Execution For Batch Job

2023-01-12 Thread Lijie Wang
+1 (binding) Best, Lijie Martijn Visser 于2023年1月12日周四 15:56写道: > +0 (binding) > > Op di 10 jan. 2023 om 13:11 schreef yuxia : > > > +1 (non-binding). > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "Zhu Zhu" > > 收件人: "dev" > > 发送时间: 星期二, 2023年 1 月 10日 下午 5:50:39 > > 主题: Re:

[jira] [Created] (FLINK-30641) docs_404_check fail

2023-01-11 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-30641: -- Summary: docs_404_check fail Key: FLINK-30641 URL: https://issues.apache.org/jira/browse/FLINK-30641 Project: Flink Issue Type: Bug Reporter: Lijie

[jira] [Created] (FLINK-30631) Limit the max number of subpartitons consumed by each downstream task

2023-01-11 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-30631: -- Summary: Limit the max number of subpartitons consumed by each downstream task Key: FLINK-30631 URL: https://issues.apache.org/jira/browse/FLINK-30631 Project: Flink

Re: [VOTE] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs

2023-01-10 Thread Lijie Wang
+1 (binding) Best, Lijie Junrui Lee 于2023年1月10日周二 11:31写道: > Hi all, > > Thanks for all the feedback so far. > Based on the discussion[1], we have come to a consensus, > so I would like to start a vote on FLIP-283: Use adaptive > batch scheduler as default scheduler for batch jobs[2]. > > The

Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee

2023-01-09 Thread Lijie Wang
Congratulations, Lincoln! Best, Lijie Jingsong Li 于2023年1月10日周二 12:07写道: > Congratulations, Lincoln! > > Best, > Jingsong > > On Tue, Jan 10, 2023 at 11:56 AM Leonard Xu wrote: > > > > Congratulations, Lincoln! > > > > Impressive work in streaming semantics, well deserved! > > > > > > Best, >

[jira] [Created] (FLINK-30604) Remove the limitation of that parallelism decided by adaptive batch scheduler must be power-of-two

2023-01-09 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-30604: -- Summary: Remove the limitation of that parallelism decided by adaptive batch scheduler must be power-of-two Key: FLINK-30604 URL: https://issues.apache.org/jira/browse/FLINK-30604

[jira] [Created] (FLINK-30601) Omit "setKeyContextElement" call for non-keyed stream/operators to improve performance

2023-01-08 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-30601: -- Summary: Omit "setKeyContextElement" call for non-keyed stream/operators to improve performance Key: FLINK-30601 URL: https://issues.apache.org/jira/browse/F

Re: [VOTE]FLIP-266: Simplify network memory configurations for TaskManager

2023-01-04 Thread Lijie Wang
+1 (binding) Best, Lijie 17610775726 <17610775...@163.com> 于2023年1月4日周三 13:03写道: > > > +1 (no binding) > > > Best > JasonLee > > > Replied Message > | From | Yuxin Tan | > | Date | 01/3/2023 17:56 | > | To | | > | Subject | [VOTE]FLIP-266: Simplify network memory configurations for >

Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job

2023-01-03 Thread Lijie Wang
nks for driving this FLIP. It's meaningful to support speculative > > execution > > of sinks is important. > > > > I have a question about this: Speculative execution of Committer will be > > disabled. > > > > I agree with your point and I saw the similar requirements

[jira] [Created] (FLINK-30544) Speed up finding minimum watermark across all channels by introducing heap-based algorithm

2023-01-02 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-30544: -- Summary: Speed up finding minimum watermark across all channels by introducing heap-based algorithm Key: FLINK-30544 URL: https://issues.apache.org/jira/browse/FLINK-30544

Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs

2023-01-02 Thread Lijie Wang
Hi Junrui, Thanks for driving this FLIP, + 1 for this proposal. I believe it will greatly improve the experiences of batch users. Best, Lijie Zhu Zhu 于2022年12月30日周五 12:40写道: > Hi Junrui, > > Thanks for creating this FLIP! > > AdaptiveBatchScheduler is more powerful than DefaultScheduler in

Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job

2022-12-27 Thread Lijie Wang
Hi Biao, Thanks for driving this FLIP. In this FLIP, it introduces "int getFinishedAttempt(int subtaskIndex)" for OutputFormat to know which subtask attempt is the one marked as finished by JM and commit the right data. I have a question about this: does SinkV2 need to do the same thing? Should

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

2022-12-22 Thread Lijie Wang
Hi, Thanks for driving this FLIP, +1 for the proposed changes. Limit the maximum value of shuffle read memory is very useful when using when using adaptive batch scheduler. Currently, the adaptive batch scheduler may cause a large number of input channels in a certain TM, so we generally

Re: [ANNOUNCE] New Apache Flink Committer - Matyas Orhidi

2022-11-24 Thread Lijie Wang
Congratulations Matyas! Best, Lijie Dong Lin 于2022年11月25日周五 08:31写道: > Congrats Matyas! > > On Mon, Nov 21, 2022 at 10:18 PM Márton Balassi > wrote: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce Matyas Orhidi as a new > > Flink > > committer. > > > > Matyas has

Re: [ANNOUNCE] Apache Flink 1.16.0 released

2022-10-28 Thread Lijie Wang
Congratulations! Thanks everyone involved! Best, LIjie Xingbo Huang 于2022年10月28日周五 14:46写道: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.16.0, which is the first release for the Apache Flink 1.16 series. > > Apache Flink® is an open-source stream

  1   2   3   >