[jira] [Created] (FLINK-35322) PubSub Connector Weekly build fails

2024-05-09 Thread Ahmed Hamdy (Jira)
Ahmed Hamdy created FLINK-35322: --- Summary: PubSub Connector Weekly build fails Key: FLINK-35322 URL: https://issues.apache.org/jira/browse/FLINK-35322 Project: Flink Issue Type: Bug

Re: [DISCUSSION] FLIP-456: CompiledPlan support for Batch Execution Mode

2024-05-09 Thread Jim Hughes
Hi Alexey, Overall, the FLIP looks good and makes sense to me. 1. For the testing, I'd call the tests "execution" tests rather than "restore" tests. For streaming execution, restore tests have the compiled plan and intermediate state; the tests verify that those can work together and continue

Re: [DISCUSS] Flink CDC 3.2 Release Planning

2024-05-09 Thread Peter Huang
Thanks Qingsheng for driving the release! +1. I also would like to provide some help on CDC 3.2. Best Regards Peter Huang On Thu, May 9, 2024 at 3:21 AM Xiqian YU wrote: > Thanks Qingsheng for driving the release! > > +1. Would love to provide my help on CDC 3.2. > > Regards, > Xiqian > > De

Re: [VOTE] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

2024-05-09 Thread Lincoln Lee
+1 (binding) Best, Lincoln Lee Benchao Li 于2024年5月9日周四 13:02写道: > +1 (binding) > > Timo Walther 于2024年5月8日周三 17:15写道: > > > > +1 (binding) > > > > Thanks, > > Timo > > > > On 08.05.24 11:10, Stefan Richter wrote: > > > Hi Alan, > > > > > > Thanks for this proposal, the ability to exclude

Re: Re: [VOTE] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-09 Thread Lincoln Lee
+1 (binding) Best, Lincoln Lee Feng Jin 于2024年5月9日周四 19:45写道: > +1 (non-binding) > > > Best, > Feng > > > On Thu, May 9, 2024 at 7:37 PM Xuyang wrote: > > > +1 (non-binding) > > > > > > -- > > > > Best! > > Xuyang > > > > > > > > > > > > At 2024-05-09 13:57:07, "Ron Liu" wrote: > >

[jira] [Created] (FLINK-35321) CheckpointCommittableManagerImpl re-registers pendingCommittables gauge on every commit operation

2024-05-09 Thread Lennon Yu (Jira)
Lennon Yu created FLINK-35321: - Summary: CheckpointCommittableManagerImpl re-registers pendingCommittables gauge on every commit operation Key: FLINK-35321 URL: https://issues.apache.org/jira/browse/FLINK-35321

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-09 Thread Aleksandr Pilipenko
Thanks for driving this! +1 (non-binding) Thanks, Aleksandr On Thu, 9 May 2024 at 12:21, Ahmed Hamdy wrote: > Thanks for the effort and the data provided. > > +1 (non-binding) > Best Regards > Ahmed Hamdy > > > On Thu, 9 May 2024 at 08:20, Keith Lee > wrote: > > > +1 (non-binding) > > > >

[VOTE] FLIP-449: Reorganization of flink-connector-jdbc

2024-05-09 Thread Joao Boto
Hi everyone, Thanks for all the feedback, I'd like to start a vote on the FLIP-449: Reorganization of flink-connector-jdbc [1]. The discussion thread is here [2]. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1]

Re: Re: [VOTE] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-09 Thread Feng Jin
+1 (non-binding) Best, Feng On Thu, May 9, 2024 at 7:37 PM Xuyang wrote: > +1 (non-binding) > > > -- > > Best! > Xuyang > > > > > > At 2024-05-09 13:57:07, "Ron Liu" wrote: > >Sorry for the re-post, just to format this email content. > > > >Hi Dev > > > >Thank you to everyone for

Re:Re: [VOTE] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-09 Thread Xuyang
+1 (non-binding) -- Best! Xuyang At 2024-05-09 13:57:07, "Ron Liu" wrote: >Sorry for the re-post, just to format this email content. > >Hi Dev > >Thank you to everyone for the feedback on FLIP-448: Introduce Pluggable >Workflow Scheduler Interface for Materialized Table[1][2].

Re: Question around Flink's AdaptiveBatchScheduler

2024-05-09 Thread Venkatakrishnan Sowrirajan
Xia, Thanks for the reviews. Unfortunately due to work commitments I am little delayed in addressing your review comments. Mostly will be done by end of this week. Just a quick heads up. Jinrui, Thanks, that would be great. On Mon, May 6, 2024, 12:45 AM Junrui Lee wrote: > Hi, > Thanks for

Request for access

2024-05-09 Thread Vikas Garg
Hello, Our product Vertica is one of the most widely used analytical database and is supported by wide range of BI/ETL tools. We have developed a connector for connecting Flink with Vertica and are looking for its approval by Flink. Our organization will be providing support for connector.

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-09 Thread Ahmed Hamdy
Thanks for the effort and the data provided. +1 (non-binding) Best Regards Ahmed Hamdy On Thu, 9 May 2024 at 08:20, Keith Lee wrote: > +1 (non-binding) > > Great work on the benchmark data. > > Keith > > On Thu, May 9, 2024 at 4:53 AM 周仁祥 wrote: > > > +1 (non-binding) > > > > ConradJam

[jira] [Created] (FLINK-35320) RabbitMQ Source fails to consume from quorum queues when prefetch Count is set

2024-05-09 Thread Ahmed Hamdy (Jira)
Ahmed Hamdy created FLINK-35320: --- Summary: RabbitMQ Source fails to consume from quorum queues when prefetch Count is set Key: FLINK-35320 URL: https://issues.apache.org/jira/browse/FLINK-35320

Re: [DISCUSS] Flink CDC 3.2 Release Planning

2024-05-09 Thread Xiqian YU
Thanks Qingsheng for driving the release! +1. Would love to provide my help on CDC 3.2. Regards, Xiqian De : Hang Ruan Date : jeudi, 9 mai 2024 à 17:50 À : dev@flink.apache.org Objet : Re: [DISCUSS] Flink CDC 3.2 Release Planning Thanks Qinsheng for driving. I would like to provide some

Re: [DISCUSS] Flink CDC 3.2 Release Planning

2024-05-09 Thread Hang Ruan
Thanks Qinsheng for driving. I would like to provide some helps for this verison too. +1. Best, Hang Hongshun Wang 于2024年5月9日周四 14:16写道: > Thanks Qinsheng for driving, > +1 from my side. > > Besi, > Hongshun > > On Wed, May 8, 2024 at 11:41 PM Leonard Xu wrote: > > > +1 for the proposal code

[jira] [Created] (FLINK-35319) Specific Operator getting stuck in Initialization for a long time

2024-05-09 Thread Kartikey Pant (Jira)
Kartikey Pant created FLINK-35319: - Summary: Specific Operator getting stuck in Initialization for a long time Key: FLINK-35319 URL: https://issues.apache.org/jira/browse/FLINK-35319 Project: Flink

RE: [RESULT][VOTE] FLIP-454: New Apicurio Avro format

2024-05-09 Thread David Radley
Hi Leonard, OK thanks ?C I missed that. Are any committers/PMC members prepared to back this Flip? Kind regards, David. From: Leonard Xu Date: Wednesday, 8 May 2024 at 16:43 To: dev Subject: [EXTERNAL] Re: [RESULT][VOTE] FLIP-454: New Apicurio Avro format Thanks David for driving the

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-09 Thread Keith Lee
Hi Piotr, Thank you for the proposal. Looks great. Along similar line of Aleks' question on memory usage. The proposal mentions that s5cmd utilises 100% of CPU similar to Flink 1.18. However, this will be a native process outside of the JVM. Are there risk of large/long state download starving

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-09 Thread Keith Lee
+1 (non-binding) Great work on the benchmark data. Keith On Thu, May 9, 2024 at 4:53 AM 周仁祥 wrote: > +1 (non-binding) > > ConradJam 于2024年5月7日周二 11:35写道: > > > +1 (no-binding) > > > > Piotr Nowojski 于2024年5月6日周一 20:17写道: > > > > > +1 (binding) > > > > > > Piotrek > > > > > > pon., 6 maj

[jira] [Created] (FLINK-35318) incorrect timezone handling for TIMESTAMP_WITH_LOCAL_TIME_ZONE type during predicate pushdown

2024-05-09 Thread linshangquan (Jira)
linshangquan created FLINK-35318: Summary: incorrect timezone handling for TIMESTAMP_WITH_LOCAL_TIME_ZONE type during predicate pushdown Key: FLINK-35318 URL: https://issues.apache.org/jira/browse/FLINK-35318

[jira] [Created] (FLINK-35317) Flink CDC CLI Supports submitting multiple YAML job at once

2024-05-09 Thread yux (Jira)
yux created FLINK-35317: --- Summary: Flink CDC CLI Supports submitting multiple YAML job at once Key: FLINK-35317 URL: https://issues.apache.org/jira/browse/FLINK-35317 Project: Flink Issue Type:

Re: [DISCUSS] Flink CDC 3.2 Release Planning

2024-05-09 Thread Hongshun Wang
Thanks Qinsheng for driving, +1 from my side. Besi, Hongshun On Wed, May 8, 2024 at 11:41 PM Leonard Xu wrote: > +1 for the proposal code freeze date and RM candidate. > > Best, > Leonard > > > 2024年5月8日 下午10:27,gongzhongqiang 写道: > > > > Hi Qingsheng > > > > Thank you for driving the

[jira] [Created] (FLINK-35316) Add CDC e2e test case for on Flink 1.19

2024-05-09 Thread yux (Jira)
yux created FLINK-35316: --- Summary: Add CDC e2e test case for on Flink 1.19 Key: FLINK-35316 URL: https://issues.apache.org/jira/browse/FLINK-35316 Project: Flink Issue Type: Bug Components:

Re: [VOTE] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-08 Thread Ron Liu
Sorry for the re-post, just to format this email content. Hi Dev Thank you to everyone for the feedback on FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table[1][2]. I'd like to start a vote for it. The vote will be open for at least 72 hours unless there is an

[VOTE] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-08 Thread Ron Liu
Hi Dev, Thank you to everyone for the feedback on FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table[1][2]. I'd like to start a vote for it. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1]

Re: [VOTE] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

2024-05-08 Thread Benchao Li
+1 (binding) Timo Walther 于2024年5月8日周三 17:15写道: > > +1 (binding) > > Thanks, > Timo > > On 08.05.24 11:10, Stefan Richter wrote: > > Hi Alan, > > > > Thanks for this proposal, the ability to exclude functions from constant > > folding makes sense to me. > > > > +1 (binding) > > > > Best, > >

[RESULT][VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-08 Thread yue ma
Hi everyone, Thanks for your review and the votes! I am happy to announce that FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0 [1]. has been accepted. The proposal has been accepted with 15 approving votes (9 binding) and there is no disapproval: - Zakelly Lan (binding) - Yanfei Lei (binding)

[jira] [Created] (FLINK-35315) MemoryManagerConcurrentModReleaseTest executes more than 15 minutes

2024-05-08 Thread Rui Fan (Jira)
Rui Fan created FLINK-35315: --- Summary: MemoryManagerConcurrentModReleaseTest executes more than 15 minutes Key: FLINK-35315 URL: https://issues.apache.org/jira/browse/FLINK-35315 Project: Flink

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-08 Thread 周仁祥
+1 (non-binding) ConradJam 于2024年5月7日周二 11:35写道: > +1 (no-binding) > > Piotr Nowojski 于2024年5月6日周一 20:17写道: > > > +1 (binding) > > > > Piotrek > > > > pon., 6 maj 2024 o 12:35 Roman Khachatryan > napisał(a): > > > > > +1 (binding) > > > > > > Regards, > > > Roman > > > > > > > > > On Mon, May

[jira] [Created] (FLINK-35314) Add Flink CDC pipeline transform user document

2024-05-08 Thread Wenkai Qi (Jira)
Wenkai Qi created FLINK-35314: - Summary: Add Flink CDC pipeline transform user document Key: FLINK-35314 URL: https://issues.apache.org/jira/browse/FLINK-35314 Project: Flink Issue Type: New

[jira] [Created] (FLINK-35313) Add upsert changelog mode to avoid UPDATE_BEFORE records push down

2024-05-08 Thread JJJJude (Jira)
ude created FLINK-35313: --- Summary: Add upsert changelog mode to avoid UPDATE_BEFORE records push down Key: FLINK-35313 URL: https://issues.apache.org/jira/browse/FLINK-35313 Project: Flink

[jira] [Created] (FLINK-35312) Insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_all_changes_

2024-05-08 Thread yux (Jira)
yux created FLINK-35312: --- Summary: Insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_all_changes_ Key: FLINK-35312 URL: https://issues.apache.org/jira/browse/FLINK-35312

flink-connector-kafka weekly CI job failing

2024-05-08 Thread Robert Young
Hi, I noticed the flink-connector-kafka weekly CI job is failing: https://github.com/apache/flink-connector-kafka/actions/runs/8954222477 Looks like flink-connector-kafka main has a compile error against flink 1.20-SNAPSHOT, I tried locally and get a different compile failure

Best Practices? Fault Isolation for Processing Large Number of Same-Shaped Input Kafka Topics in a Big Flink Job

2024-05-08 Thread Kevin Lam
Hi everyone, I'm currently prototyping on a project where we need to process a large number of Kafka input topics (say, a couple of hundred), all of which share the same DataType/Schema. Our objective is to run the same Flink SQL on all of the input topics, but I am concerned about doing this in

Re: [RESULT][VOTE] FLIP-454: New Apicurio Avro format

2024-05-08 Thread Leonard Xu
Thanks David for driving the FLIP forward, but we need 3 +1(binding) votes according Flink Bylaws[1] before community accepted it. Best, Leonard [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws > 2024年5月8日 下午11:05,David Radley 写道: > > Hi everyone, > I am happy to say that

Re: [RESULT][VOTE] FLIP-454: New Apicurio Avro format

2024-05-08 Thread Ahmed Hamdy
Hi David, thanks for the update We should wait to reach threshold of "binding" votes as per the process[1] 1- https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals Best Regards Ahmed Hamdy On Wed, 8 May 2024 at 16:06, David Radley wrote: > Hi everyone, > I am

Re: [DISCUSS] Flink CDC 3.2 Release Planning

2024-05-08 Thread Leonard Xu
+1 for the proposal code freeze date and RM candidate. Best, Leonard > 2024年5月8日 下午10:27,gongzhongqiang 写道: > > Hi Qingsheng > > Thank you for driving the release. > Agree with the goal and I'm willing to help. > > Best, > Zhongqiang Gong > > Qingsheng Ren 于2024年5月8日周三 14:22写道: > >> Hi

[RESULT][VOTE] FLIP-454: New Apicurio Avro format

2024-05-08 Thread David Radley
Hi everyone, I am happy to say that FLIP-454: New Apicurio Avro format [1] has been accepted and voted through this thread [2]. The proposal has been accepted with 4 approving votes and there are no vetos: - Ahmed Hamdy (non-binding) - Jeyhun Karimov (non-binding) - Mark Nuttall (non-binding) -

[jira] [Created] (FLINK-35311) FLIP-454: New Apicurio Avro format

2024-05-08 Thread david radley (Jira)
david radley created FLINK-35311: Summary: FLIP-454: New Apicurio Avro format Key: FLINK-35311 URL: https://issues.apache.org/jira/browse/FLINK-35311 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-35310) Replace RBAC verb wildcards with actual verbs

2024-05-08 Thread Tim (Jira)
Tim created FLINK-35310: --- Summary: Replace RBAC verb wildcards with actual verbs Key: FLINK-35310 URL: https://issues.apache.org/jira/browse/FLINK-35310 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Flink CDC 3.2 Release Planning

2024-05-08 Thread gongzhongqiang
Hi Qingsheng Thank you for driving the release. Agree with the goal and I'm willing to help. Best, Zhongqiang Gong Qingsheng Ren 于2024年5月8日周三 14:22写道: > Hi devs, > > As we are in the midst of the release voting process for Flink CDC 3.1.0, I > think it's a good time to kick off the upcoming

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-08 Thread Ahmed Hamdy
Hi Piotr +1 for the proposal, it seems to have a lot of gains. Best Regards Ahmed Hamdy On Mon, 6 May 2024 at 12:06, Zakelly Lan wrote: > Hi Piotrek, > > Thanks for your answers! > > Good question. The intention and use case behind `DuplicatingFileSystem` is > > different. It marks if

Re: [DISCUSS] Flink CDC 3.2 Release Planning

2024-05-08 Thread Ahmed Hamdy
Thanks Qinsheng for driving, +1 for the feature freeze date. I am happy to assist with any release duties. Best Regards Ahmed Hamdy On Wed, 8 May 2024 at 10:50, Jiabao Sun wrote: > Thanks Qingsheng, > > Improving stability is crucial for Flink CDC, looking forward to this > release. > If

Re: [DISCUSS] Flink CDC 3.2 Release Planning

2024-05-08 Thread Muhammet Orazov
Hey Qingsheng, Thanks for your efforts, agreed! I would be happy to help. Best, Muhammet On 2024-05-08 06:21, Qingsheng Ren wrote: Hi devs, As we are in the midst of the release voting process for Flink CDC 3.1.0, I think it's a good time to kick off the upcoming Flink CDC 3.2 release

Re: [DISCUSS] Flink CDC 3.2 Release Planning

2024-05-08 Thread Jiabao Sun
Thanks Qingsheng, Improving stability is crucial for Flink CDC, looking forward to this release. If assistance is needed, I am happy to help with it. Best, Jiabao Qingsheng Ren 于2024年5月8日周三 14:22写道: > Hi devs, > > As we are in the midst of the release voting process for Flink CDC 3.1.0, I >

Re: [VOTE] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

2024-05-08 Thread Timo Walther
+1 (binding) Thanks, Timo On 08.05.24 11:10, Stefan Richter wrote: Hi Alan, Thanks for this proposal, the ability to exclude functions from constant folding makes sense to me. +1 (binding) Best, Stefan On 8. May 2024, at 02:01, Alan Sheinberg wrote: Hi everyone, I'd like to start a

Re: [VOTE] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

2024-05-08 Thread Stefan Richter
Hi Alan, Thanks for this proposal, the ability to exclude functions from constant folding makes sense to me. +1 (binding) Best, Stefan > On 8. May 2024, at 02:01, Alan Sheinberg > wrote: > > Hi everyone, > > I'd like to start a vote on FLIP-452 [1]. It covers adding a new method >

Re: [DISCUSS] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-08 Thread Ron Liu
Hi, Dev Thank you all for joining this thread and giving your comments and suggestions, they have helped improve this proposal and I look forward to further feedback. If there are no further comments, I'd like to close the discussion and start the voting one day later. Best, Ron Ron Liu

Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #1

2024-05-08 Thread gongzhongqiang
Hi Qingsheng, I open a block issue[1] to add ci for NOTICE check and fix NOTICE file. Best, Zhongqiang Gong [1] https://issues.apache.org/jira/browse/FLINK-35309 Qingsheng Ren 于2024年5月8日周三 14:28写道: > Thanks everyone for your verification! > > I'll investigate the issue and prepare another RC

[jira] [Created] (FLINK-35309) Enable Notice file ci check and fix Notice

2024-05-08 Thread Zhongqiang Gong (Jira)
Zhongqiang Gong created FLINK-35309: --- Summary: Enable Notice file ci check and fix Notice Key: FLINK-35309 URL: https://issues.apache.org/jira/browse/FLINK-35309 Project: Flink Issue

[jira] [Created] (FLINK-35308) StarRocks sink maps TINYINT type to BOOLEAN incorrectly

2024-05-08 Thread yux (Jira)
yux created FLINK-35308: --- Summary: StarRocks sink maps TINYINT type to BOOLEAN incorrectly Key: FLINK-35308 URL: https://issues.apache.org/jira/browse/FLINK-35308 Project: Flink Issue Type: Bug

Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #1

2024-05-08 Thread Qingsheng Ren
Thanks everyone for your verification! I'll investigate the issue and prepare another RC once it is resolved. Much appreciated to Jingsong! This release candidate is now canceled. Best, Qingsheng On Wed, May 8, 2024 at 2:20 PM Jingsong Li wrote: > -1 > > Thanks Qingsheng for preparing this

[DISCUSS] Flink CDC 3.2 Release Planning

2024-05-08 Thread Qingsheng Ren
Hi devs, As we are in the midst of the release voting process for Flink CDC 3.1.0, I think it's a good time to kick off the upcoming Flink CDC 3.2 release cycle. In this release cycle I would like to focus on the stability of Flink CDC, especially for the newly introduced YAML-based data

Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #1

2024-05-08 Thread Jingsong Li
-1 Thanks Qingsheng for preparing this RC. If you bundle third-party dependencies (non Flink dependencies) in your published jar, you need to write them in the NOTICE file. I recommend adding a test using flink-ci-tools to verify if the NOTICE file is correct. Of course, you cannot rely too

[jira] [Created] (FLINK-35307) Add Compile CI check on jdk17

2024-05-07 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35307: --- Summary: Add Compile CI check on jdk17 Key: FLINK-35307 URL: https://issues.apache.org/jira/browse/FLINK-35307 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-35306) Flink cannot compile with jdk17

2024-05-07 Thread Rui Fan (Jira)
Rui Fan created FLINK-35306: --- Summary: Flink cannot compile with jdk17 Key: FLINK-35306 URL: https://issues.apache.org/jira/browse/FLINK-35306 Project: Flink Issue Type: Bug Affects Versions:

Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction

2024-05-07 Thread Hongshun Wang
Hi Martijn, Thanks for the proposal +1 from me.Some sinks still use sinkfunction; it's time to take a step forward. Best, Hongshun On Mon, May 6, 2024 at 5:44 PM Leonard Xu wrote: > +1 from my side, thanks Martijn for the effort. > > Best, > Leonard > > > 2024年5月4日 下午7:41,Ahmed Hamdy 写道: > >

[jira] [Created] (FLINK-35305) FLIP-438: Amazon SQS Sink Connector

2024-05-07 Thread Priya Dhingra (Jira)
Priya Dhingra created FLINK-35305: - Summary: FLIP-438: Amazon SQS Sink Connector Key: FLINK-35305 URL: https://issues.apache.org/jira/browse/FLINK-35305 Project: Flink Issue Type: New

[VOTE] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

2024-05-07 Thread Alan Sheinberg
Hi everyone, I'd like to start a vote on FLIP-452 [1]. It covers adding a new method FunctionDefinition.supportsConstantFolding() as part of the Flink Table/SQL API to allow skipping invocation of functions while constant-folding. It has been discussed in this thread [2]. I would like to start a

Re: Issue in PrefetchCount

2024-05-07 Thread Talat Uyarer
Hi ajay, When you have 3 parallelisms you will have 3 independent clients. If you want to keep prefetch count 3 you need to set setRequestedChannelMax as 1 and setParallelism 3. So All 3 clients can have one connection. Talat On Tue, May 7, 2024 at 5:52 AM ajay pandey wrote: > Hi Flink Team,

Re: [Vote] FLIP-438: Amazon SQS Sink Connector

2024-05-07 Thread Dhingra, Priya
Hi All, I'm happy to announce that FLIP-438: Amazon SQS Sink Connector[1] has been accepted with 9 approving votes (4 binding). I would like to thank everyone who took part in the discussion and/or voted. - Ahmed Hamdy (non-binding) - Samrat(non-binding) - Danny (binding) - Aleksandr

Re: [DISCUSSION] FLIP-456: CompiledPlan support for Batch Execution Mode

2024-05-07 Thread Alexey Leonov-Vendrovskiy
Hey Paul, Yes, no interchangeability. Just a wire-through for more uniformity. Thanks, Alexey On Tue, May 7, 2024 at 2:10 AM Paul Lam wrote: > Hi Alexey, > > Thanks a lot for bringing up the discussion. I’m big +1 for the FLIP. > > I suppose the goal doesn’t involve the interchangeability of

[RabbitMQ Connector]: missing support for per consumer QoS (needed for quorum queues)

2024-05-07 Thread charlie vuillemez
Hello devs, Seems RabbitMQ Connector support only global QoS (Prefetch), which is not compatible with RabbitMQ quorum queues.Cf https://www.rabbitmq.com/docs/quorum-queues#feature-matrix Quorum queues is the only option for HA, cause mirrored classic queues are deprecated since RabbitMQ

Issue in PrefetchCount

2024-05-07 Thread ajay pandey
Hi Flink Team, I am currently reading streaming data from RabbitMQ and using the RMQConnectionConfig for establishing the connection. Here's how I'm setting up the connection: and we use flink version 1.16.2 and RabbitMQ version 3.10.7 RMQConnectionConfig connectionConfig = new

Re: [DISCUSS] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-07 Thread Ron Liu
Hi, dev Following the recent PoC[1], and drawing on the excellent code design within Flink, I have made the following optimizations to the Public Interfaces section of FLIP: 1. I have renamed WorkflowOperation to RefreshWorkflow. This change better conveys its purpose. RefreshWorkflow is used to

Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc

2024-05-07 Thread lorenzo . affetti
Thanks joao for your replies! I also saw the latest PR that allows properties to be specified. Thank for adding the pain points as well, that clarifies a lot. On May 7, 2024 at 09:50 +0200, Muhammet Orazov , wrote: > Thanks João for pointing it out. I didn't know about the PR, I am going > to

[jira] [Created] (FLINK-35304) Mongo ITCase fails due to duplicate records after resuming

2024-05-07 Thread yux (Jira)
yux created FLINK-35304: --- Summary: Mongo ITCase fails due to duplicate records after resuming Key: FLINK-35304 URL: https://issues.apache.org/jira/browse/FLINK-35304 Project: Flink Issue Type: Bug

Re: [DISCUSSION] FLIP-456: CompiledPlan support for Batch Execution Mode

2024-05-07 Thread Paul Lam
Hi Alexey, Thanks a lot for bringing up the discussion. I’m big +1 for the FLIP. I suppose the goal doesn’t involve the interchangeability of json plans between batch mode and streaming mode, right? In other words, a json plan compiled in a batch program can’t be run in streaming mode without

[SUMMARY] Flink 1.20 Release Sync 05/07/2024

2024-05-07 Thread Rui Fan
Dear devs, Today is the third meeting for Flink 1.20 release cycle. I'd like to share the information synced in the meeting. - Feature Freeze As we discussed before, the feature freeze time is June 15, 2024, 00:00 CEST(UTC+2). It is worth noting that there are only 6 weeks left until the

[jira] [Created] (FLINK-35303) Support logical deletion of datax

2024-05-07 Thread melin (Jira)
melin created FLINK-35303: - Summary: Support logical deletion of datax Key: FLINK-35303 URL: https://issues.apache.org/jira/browse/FLINK-35303 Project: Flink Issue Type: New Feature

Re: [Vote] FLIP-438: Amazon SQS Sink Connector

2024-05-07 Thread Hong Liang
Hi Priya, Great to see the FLIP has passed the vote. It would be good to report the final result of the vote at the end of the thread, listing the binding / non-binding votes in an email. See example here [1] [2] Regards, Hong [1] https://lists.apache.org/thread/3sj88kk0104vzj4hklfgbn3rpdnjxj8v

Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc

2024-05-07 Thread Muhammet Orazov
Thanks João for pointing it out. I didn't know about the PR, I am going to check it. Best, Muhammet On 2024-05-06 14:45, João Boto wrote: Hi Muhammet, Have you had a chance to review the recently merged pull request [1]? We've introduced a new feature allowing users to include ad hoc

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-07 Thread Rui Fan
Hi Piotr, Overall this FLIP is fine for me. I have a minor concern: IIUC, the code path will become more complex after this FLIP due to the addition of shouldIntterupt() checks, right? If so, it's better to add a benchmark to check whether the job performance regresses when one job has a lot of

[DISCUSSION] FLIP-456: CompiledPlan support for Batch Execution Mode

2024-05-07 Thread Alexey Leonov-Vendrovskiy
Hi everyone, PTAL at the proposed FLIP-456: CompiledPlan support for Batch Execution Mode. It is pretty self-describing. Any thoughts are welcome! Thanks, Alexey [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode .

Re: [DISCUSS] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-07 Thread Ron Liu
> 4. It appears that in the section on `public interfaces`, within `WorkflowOperation`, `CreatePeriodicWorkflowOperation` should be changed to `CreateWorkflowOperation`, right? After discussing with Xuyang offline, we need to support periodic workflow and one-time workflow, they need different

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-06 Thread ConradJam
+1 (no-binding) Piotr Nowojski 于2024年5月6日周一 20:17写道: > +1 (binding) > > Piotrek > > pon., 6 maj 2024 o 12:35 Roman Khachatryan napisał(a): > > > +1 (binding) > > > > Regards, > > Roman > > > > > > On Mon, May 6, 2024 at 11:56 AM gongzhongqiang < > gongzhongqi...@apache.org> > > wrote: > > > >

[jira] [Created] (FLINK-35302) Flink REST server throws exception on unknown fields in RequestBody

2024-05-06 Thread Juntao Hu (Jira)
Juntao Hu created FLINK-35302: - Summary: Flink REST server throws exception on unknown fields in RequestBody Key: FLINK-35302 URL: https://issues.apache.org/jira/browse/FLINK-35302 Project: Flink

[jira] [Created] (FLINK-35301) Fix deadlock when loading driver classes

2024-05-06 Thread Xiao Huang (Jira)
Xiao Huang created FLINK-35301: -- Summary: Fix deadlock when loading driver classes Key: FLINK-35301 URL: https://issues.apache.org/jira/browse/FLINK-35301 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-35300) Improve MySqlStreamingChangeEventSource to skip null events in event deserializer

2024-05-06 Thread Xiao Huang (Jira)
Xiao Huang created FLINK-35300: -- Summary: Improve MySqlStreamingChangeEventSource to skip null events in event deserializer Key: FLINK-35300 URL: https://issues.apache.org/jira/browse/FLINK-35300

Re: [Discuss] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

2024-05-06 Thread Lincoln Lee
Thanks Alan for starting this flip! +1 for the new `supportsConstantFolding`. We do need to reserve the ability to not perform constant folding for certain cases. Best, Lincoln Lee Alan Sheinberg 于2024年5月7日周二 08:32写道: > Hi Muhammet, Timo > > The `isDeterministic()`[1] function is mentioned

Re: [Discuss] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

2024-05-06 Thread Alan Sheinberg
Hi Muhammet, Timo The `isDeterministic()`[1] function is mentioned in the documentation, > I would suggest to add maybe a section for `supportsConstantFolding()`, > with short description and examples use cases (similar to the > motivation of the FLIP) where this could be useful in UDFs. > Thanks

Re: [Discuss] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

2024-05-06 Thread Timo Walther
Hi Alan, thanks for the update. From my side, the FLIP seems good for voting. Since it only touches a small API surface, I guess the proposal is not very controversial. Feel free to start a vote thread by tomorrow. If there are no objections? Thanks, Timo On 02.05.24 09:45, Muhammet

Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc

2024-05-06 Thread João Boto
Hi Jeyhun, > > I would also ask to include a sample usage and changes for end-users in the > > FLIP. We want to ensure a seamless transition for end-users, minimizing any disruptions in their current usage of the connector. To achieve this, we will uphold consistency by maintaining the same

Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc

2024-05-06 Thread João Boto
Hi Muhammet, Have you had a chance to review the recently merged pull request [1]? We've introduced a new feature allowing users to include ad hoc configurations in the 'JdbcConnectionOptions' class. ``` new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()

[jira] [Created] (FLINK-35299) FlinkKinesisConsumer does not respect StreamInitialPosition for new Kinesis Stream when restoring from snapshot

2024-05-06 Thread Hong Liang Teoh (Jira)
Hong Liang Teoh created FLINK-35299: --- Summary: FlinkKinesisConsumer does not respect StreamInitialPosition for new Kinesis Stream when restoring from snapshot Key: FLINK-35299 URL:

Re: [DISCUSS] FLIP-451: Refactor Async sink API

2024-05-06 Thread Ahmed Hamdy
Hi Muhammet, Thanks for the feedback. > Could you please add more here why it is harder? Would the > `completeExceptionally` > method be related to it? Maybe you can add usage example for it also. > this is mainly due to the current implementation of fatal exception failures which depends on

Re: [DISCUSS] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-06 Thread Ron Liu
Hi, Xuyang Thanks for joining this discussion > 1. In the sequence diagram, it appears that there is a missing step for obtaining the refresh handler from the catalog during the suspend operation. Good catch > 2. The term "cascade refresh" does not seem to be mentioned in FLIP-435. The

Re:[DISCUSS] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

2024-05-06 Thread Xuyang
Hi, Ron. Thanks for driving this. After reading the entire flip, I have the following questions: 1. In the sequence diagram, it appears that there is a missing step for obtaining the refresh handler from the catalog during the suspend operation. 2. The term "cascade refresh" does not

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-06 Thread Piotr Nowojski
+1 (binding) Piotrek pon., 6 maj 2024 o 12:35 Roman Khachatryan napisał(a): > +1 (binding) > > Regards, > Roman > > > On Mon, May 6, 2024 at 11:56 AM gongzhongqiang > wrote: > > > +1 (non-binding) > > > > Best, > > Zhongqiang Gong > > > > yue ma 于2024年5月6日周一 10:54写道: > > > > > Hi everyone, >

[jira] [Created] (FLINK-35298) Improve metric reporter logic

2024-05-06 Thread Xiao Huang (Jira)
Xiao Huang created FLINK-35298: -- Summary: Improve metric reporter logic Key: FLINK-35298 URL: https://issues.apache.org/jira/browse/FLINK-35298 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-06 Thread Zakelly Lan
Hi Piotrek, Thanks for your answers! Good question. The intention and use case behind `DuplicatingFileSystem` is > different. It marks if `FileSystem` can quickly copy/duplicate files > in the remote `FileSystem`. For example an equivalent of a hard link or > bumping a reference count in the

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-06 Thread Piotr Nowojski
Hi All! Thanks for your comments. Muhammet and Hong, about the config options. > Could you please also add the configuration property for this? An example showing how users would set this parameter would be helpful. > 1/ Configure the implementation of PathsCopyingFileSystem used > 2/

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-06 Thread Roman Khachatryan
+1 (binding) Regards, Roman On Mon, May 6, 2024 at 11:56 AM gongzhongqiang wrote: > +1 (non-binding) > > Best, > Zhongqiang Gong > > yue ma 于2024年5月6日周一 10:54写道: > > > Hi everyone, > > > > Thanks for all the feedback, I'd like to start a vote on the FLIP-447: > > Upgrade FRocksDB from 6.20.3

[jira] [Created] (FLINK-35297) Add validation for option connect.timeout

2024-05-06 Thread Xiao Huang (Jira)
Xiao Huang created FLINK-35297: -- Summary: Add validation for option connect.timeout Key: FLINK-35297 URL: https://issues.apache.org/jira/browse/FLINK-35297 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-06 Thread Zakelly Lan
Hi Piotr, I'm saying the scenario where things happen in the following order: 1. advance watermark and process timers. 2. the cp arrives and interrupts the timer processing, after this the continuation mail is in the mailbox queue. 3. `snapshotState` is called, where the async state access

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-06 Thread Piotr Nowojski
Hi Zakelly, Can you elaborate a bit more on what you have in mind? How marking mails as interruptible helps with something? If an incoming async state access response comes, it could just request to interrupt any currently ongoing computations, regardless the currently executed mail is or is not

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-06 Thread gongzhongqiang
+1 (non-binding) Best, Zhongqiang Gong yue ma 于2024年5月6日周一 10:54写道: > Hi everyone, > > Thanks for all the feedback, I'd like to start a vote on the FLIP-447: > Upgrade FRocksDB from 6.20.3 to 8.10.0 [1]. The discussion thread is here > [2]. > > The vote will be open for at least 72 hours

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-06 Thread Roc Marshal
+1 (non-binding) Best, Regards. Yuepeng Pan Replied Message | From | Rui Fan<1996fan...@gmail.com> | | Date | 05/06/2024 11:28 | | To | dev | | Subject | Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0 | +1 (binding) Best, Rui On Mon, May 6, 2024 at 11:01 AM Yanfei Lei

[jira] [Created] (FLINK-35296) Flink mysql-cdc connector stops reading data

2024-05-06 Thread Gang Yang (Jira)
Gang Yang created FLINK-35296: - Summary: Flink mysql-cdc connector stops reading data Key: FLINK-35296 URL: https://issues.apache.org/jira/browse/FLINK-35296 Project: Flink Issue Type: Bug

<    2   3   4   5   6   7   8   9   10   11   >