Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-16 Thread Zhanghao Chen
Congrats, Zakelly! Best, Zhanghao Chen From: Shawn Huang Sent: Wednesday, April 17, 2024 9:59 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan Congratulations, Zakelly! Best, Shawn Huang Feng Jin 于2024年4月17日周三

[jira] [Created] (FLINK-35127) CDC ValuesDataSourceITCase crashed due to OutOfMemoryError

2024-04-16 Thread Jiabao Sun (Jira)
Jiabao Sun created FLINK-35127: -- Summary: CDC ValuesDataSourceITCase crashed due to OutOfMemoryError Key: FLINK-35127 URL: https://issues.apache.org/jira/browse/FLINK-35127 Project: Flink Issue

Re: [Vote] FLIP-438: Amazon SQS Sink Connector

2024-04-16 Thread Leonard Xu
+1 (binding) Best, Leonard > 2024年4月17日 上午2:25,Robert Metzger 写道: > > +1 binding > > On Tue, Apr 16, 2024 at 2:05 PM Jeyhun Karimov wrote: > >> Thanks Priya for driving the FLIP. >> >> +1 (non-binding) >> >> Regards, >> Jeyhun >> >> On Tue, Apr 16, 2024 at 12:37 PM Hong Liang wrote: >>

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-16 Thread Shawn Huang
Congratulations, Zakelly! Best, Shawn Huang Feng Jin 于2024年4月17日周三 00:16写道: > Congratulations! > > Best, > Feng > > On Tue, Apr 16, 2024 at 10:43 PM Ferenc Csaky > wrote: > > > Congratulations! > > > > Best, > > Ferenc > > > > > > > > > > On Tuesday, April 16th, 2024 at 16:28, Jeyhun Karimov

[jira] [Created] (FLINK-35128) Re-calculate the starting binlog offset after the new table added

2024-04-16 Thread Hongshun Wang (Jira)
Hongshun Wang created FLINK-35128: - Summary: Re-calculate the starting binlog offset after the new table added Key: FLINK-35128 URL: https://issues.apache.org/jira/browse/FLINK-35128 Project: Flink

[jira] [Created] (FLINK-35129) Postgres source commits the offset after every multiple checkpoint cycles.

2024-04-16 Thread Hongshun Wang (Jira)
Hongshun Wang created FLINK-35129: - Summary: Postgres source commits the offset after every multiple checkpoint cycles. Key: FLINK-35129 URL: https://issues.apache.org/jira/browse/FLINK-35129

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-16 Thread Zakelly Lan
Thanks everyone! I'm glad to be part of such an active and collaborative community! Best, Zakelly On Wed, Apr 17, 2024 at 10:14 AM Zhanghao Chen wrote: > Congrats, Zakelly! > > Best, > Zhanghao Chen > > From: Shawn Huang > Sent: Wednesday, April 17, 2024 9:59

[VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0

2024-04-16 Thread Xuannan Su
Hi everyone, Thanks for all the feedback about the FLIP-442: General Improvement to Configuration for Flink 2.0 [1] [2]. I'd like to start a vote for it. The vote will be open for at least 72 hours(excluding weekends,until APR 22, 12:00AM GMT) unless there is an objection or an insufficient

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-16 Thread Xia Sun
Hi Jeyhun, Muhammet, Thanks for all the feedback! > Could you please mention the default values for the new configurations > (e.g., table.exec.hive.infer-source-parallelism.mode, > table.exec.hive.infer-source-parallelism.enabled, > etc) ? Thanks for your suggestion. I have

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-16 Thread Muhammet Orazov
Hello Xia, Thanks for the FLIP! Since we are introducing the mode as a configuration option, could it make sense to have `InferMode.NONE` option also? The `NONE` option would disable the inference. This way we deprecate the `table.exec.hive.infer-source-parallelism` and no additional

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-16 Thread Jeyhun Karimov
Congratulations Zakelly! Regards, Jeyhun On Tue, Apr 16, 2024 at 6:35 AM Feifan Wang wrote: > Congratulations, Zakelly!—— > > Best regards, > > Feifan Wang > > > > > At 2024-04-15 10:50:06, "Yuan Mei" wrote: > >Hi everyone, > > > >On behalf of the PMC, I'm happy to let you know

Re: [DISCUSS] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-16 Thread Ferenc Csaky
Thank you Mate for initiating this discussion. +1 for this idea. Some Qs: Can you specify the newly introduced configurations in more details? Currently, it is not fully clear to me what are the possible values of `kubernetes.operator.periodic.savepoint.mode`, is it optional, has a default value?

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-16 Thread Ferenc Csaky
Congratulations! Best, Ferenc On Tuesday, April 16th, 2024 at 16:28, Jeyhun Karimov wrote: > > > Congratulations Zakelly! > > Regards, > Jeyhun > > On Tue, Apr 16, 2024 at 6:35 AM Feifan Wang zoltar9...@163.com wrote: > > > Congratulations, Zakelly!—— > > > > Best

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-16 Thread Jeyhun Karimov
Hi Xia, Thanks for driving this FLIP. +1 from my side. I have one comment. Could you please mention the default values for the new configurations (e.g., table.exec.hive.infer-source-parallelism.mode, table.exec.hive.infer-source-parallelism.enabled, etc) ? Regards, Jeyhun On Tue, Apr 16, 2024

Re: [DISCUSS] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-16 Thread Mate Czagany
Hi Ferenc, Thank you for your comments, I have updated the Google docs with a new section for the new configs. All of the newly added config keys will have defaults set, and by default all the savepoint/checkpoint operations will use the old system: write their results to the

Re: [DISCUSS] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-16 Thread Gyula Fóra
Thanks Mate, this is great stuff. Mate, I think the new configs should probably default to the new mode and they should only be useful for users to fall back to the old behaviour. We could by default use the new Snapshot CRD if the CRD is installed, otherwise use the old mode by default and log a

[jira] [Created] (FLINK-35126) Improve checkpoint progress health check config and enable by default

2024-04-16 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-35126: -- Summary: Improve checkpoint progress health check config and enable by default Key: FLINK-35126 URL: https://issues.apache.org/jira/browse/FLINK-35126 Project: Flink

[jira] [Created] (FLINK-35123) Flink Kubernetes Operator should not do deleteHAData

2024-04-16 Thread Fei Feng (Jira)
Fei Feng created FLINK-35123: Summary: Flink Kubernetes Operator should not do deleteHAData Key: FLINK-35123 URL: https://issues.apache.org/jira/browse/FLINK-35123 Project: Flink Issue Type:

[DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-16 Thread Xia Sun
Hi everyone, I would like to start a discussion on FLIP-445: Support dynamic parallelism inference for HiveSource[1]. FLIP-379[2] has introduced dynamic source parallelism inference for batch jobs, which can utilize runtime information to more accurately decide the source parallelism. As a

[jira] [Created] (FLINK-35121) CDC pipeline connector should verify requiredOptions and optionalOptions

2024-04-16 Thread Hongshun Wang (Jira)
Hongshun Wang created FLINK-35121: - Summary: CDC pipeline connector should verify requiredOptions and optionalOptions Key: FLINK-35121 URL: https://issues.apache.org/jira/browse/FLINK-35121 Project:

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-16 Thread Zhu Zhu
Thanks for creating this FLIP. @Xia +1 for this proposal. Dynamic parallelism inference can be helpful to decide a better parallelism. And it's good to unify the settings of static & dynamic parallelism inference. Thanks, Zhu Xia Sun 于2024年4月16日周二 15:12写道: > Hi everyone, > I would like to

[jira] [Created] (FLINK-35120) Add Doris Pipeline connector integration test cases

2024-04-16 Thread Xiqian YU (Jira)
Xiqian YU created FLINK-35120: - Summary: Add Doris Pipeline connector integration test cases Key: FLINK-35120 URL: https://issues.apache.org/jira/browse/FLINK-35120 Project: Flink Issue Type:

[jira] [Created] (FLINK-35122) Implement Watermark Alignment for DynamicKafkaSource

2024-04-16 Thread Mason Chen (Jira)
Mason Chen created FLINK-35122: -- Summary: Implement Watermark Alignment for DynamicKafkaSource Key: FLINK-35122 URL: https://issues.apache.org/jira/browse/FLINK-35122 Project: Flink Issue Type:

[jira] [Created] (FLINK-35124) Connector Release Fails to run Checkstyle

2024-04-16 Thread Danny Cranmer (Jira)
Danny Cranmer created FLINK-35124: - Summary: Connector Release Fails to run Checkstyle Key: FLINK-35124 URL: https://issues.apache.org/jira/browse/FLINK-35124 Project: Flink Issue Type: Bug

Re: [Vote] FLIP-438: Amazon SQS Sink Connector

2024-04-16 Thread Hong Liang
+1 (binding) Thanks Priya for driving this! This has been a requested feature for a while now, and will benefit the community :) Hong On Tue, Apr 16, 2024 at 3:23 AM Muhammet Orazov wrote: > +1 (non-binding) > > Thanks Priya for the FLIP and driving it! > > Best, > Muhammet > > On 2024-04-12

[VOTE] Release flink-connector-aws v4.3.0, release candidate #1

2024-04-16 Thread Danny Cranmer
Hi everyone, Please review and vote on the release candidate #1 for flink-connector-aws v4.3.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This release supports Apache Flink 1.18 and 1.19. The complete staging area is available

RE: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

2024-04-16 Thread David Radley
Hi Anupam, Thanks for your response. I was wondering around the schema id and had some thoughts: I assume that for Confluent Avro, specifying the schema is not normally done, but could be useful to force a particular shape. If you specify a schema id in the format configuration: - for

Re: [DISCUSS] Connector releases for Flink 1.19

2024-04-16 Thread Danny Cranmer
Hello, I have opened the VOTE thread for the AWS connectors release [1]. > If I'm not mistaking (please correct me if I'm wrong) this request is not about version update it is about new releases for connectors Yes, correct. If there are any other code changes required then help would be

[DISCUSS] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-16 Thread Mate Czagany
Hi Everyone, I would like to start a discussion on FLIP-446: Kubernetes Operator State Snapshot CRD. This FLIP adds a new custom resource for Operator users to create and manage their savepoints and checkpoints. I have also developed an initial POC to prove that this approach is feasible, you

Re: [VOTE] Release flink-connector-aws v4.3.0, release candidate #1

2024-04-16 Thread Jeyhun Karimov
+1 (non-binding) - Verified tags - Verified Lisence - Reviewed web pr Regards, Jeyhun On Tue, Apr 16, 2024 at 12:59 PM Danny Cranmer wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for flink-connector-aws > v4.3.0, as follows: > [ ] +1, Approve the release > [ ]

Re: [Vote] FLIP-438: Amazon SQS Sink Connector

2024-04-16 Thread Jeyhun Karimov
Thanks Priya for driving the FLIP. +1 (non-binding) Regards, Jeyhun On Tue, Apr 16, 2024 at 12:37 PM Hong Liang wrote: > +1 (binding) > > Thanks Priya for driving this! This has been a requested feature for a > while now, and will benefit the community :) > > Hong > > On Tue, Apr 16, 2024 at

[jira] [Created] (FLINK-35125) Implement ValueState for ForStStateBackend

2024-04-16 Thread Jinzhong Li (Jira)
Jinzhong Li created FLINK-35125: --- Summary: Implement ValueState for ForStStateBackend Key: FLINK-35125 URL: https://issues.apache.org/jira/browse/FLINK-35125 Project: Flink Issue Type:

Re: [DISCUSS] Connector releases for Flink 1.19

2024-04-16 Thread Muhammet Orazov
Thanks Sergey and Danny for clarifying, indeed it requires committer to go through the process. Anyway, please let me know if I can be any help. Best, Muhammet On 2024-04-16 11:19, Danny Cranmer wrote: Hello, I have opened the VOTE thread for the AWS connectors release [1]. If I'm not

Re: [Vote] FLIP-438: Amazon SQS Sink Connector

2024-04-16 Thread Robert Metzger
+1 binding On Tue, Apr 16, 2024 at 2:05 PM Jeyhun Karimov wrote: > Thanks Priya for driving the FLIP. > > +1 (non-binding) > > Regards, > Jeyhun > > On Tue, Apr 16, 2024 at 12:37 PM Hong Liang wrote: > > > +1 (binding) > > > > Thanks Priya for driving this! This has been a requested feature

Re: [DISCUSS] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-16 Thread Mate Czagany
Thank you Gyula! I think that is a great idea. I have updated the Google doc to only have 1 new configuration option of boolean type, which can be used to signal the Operator to use the old mode. Also described in the configuration description, the Operator will fallback to the old mode if the

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-16 Thread Feng Jin
Congratulations! Best, Feng On Tue, Apr 16, 2024 at 10:43 PM Ferenc Csaky wrote: > Congratulations! > > Best, > Ferenc > > > > > On Tuesday, April 16th, 2024 at 16:28, Jeyhun Karimov < > je.kari...@gmail.com> wrote: > > > > > > > Congratulations Zakelly! > > > > Regards, > > Jeyhun > > > > On

Re: [DISCUSS] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-16 Thread Őrhidi Mátyás
+1 Looking forward to it On Tue, Apr 16, 2024 at 8:56 AM Mate Czagany wrote: > Thank you Gyula! > > I think that is a great idea. I have updated the Google doc to only have 1 > new configuration option of boolean type, which can be used to signal the > Operator to use the old mode. > > Also