Re: [DISCUSS] Planning Flink 1.20

2024-03-22 Thread Xintong Song
+1 for the proposed timeline and Weijie & Rui as the release managers. I think it would be welcomed if another 1-2 volunteers join as the release managers, but that's not a must. We used to have only 1-2 release managers for each release, Best, Xintong On Fri, Mar 22, 2024 at 2:55 PM Jark Wu

Re: Flink Kubernetes Operator Failing Over FlinkDeployments to a New Cluster

2024-03-22 Thread Gyula Fóra
I agree, we would need some FLIPs to cover this. Actually there is already some work on this topic initiated by Matthias Pohl (ccd). Please see this: https://cwiki.apache.org/confluence/display/FLINK/FLIP-360%3A+Merging+the+ExecutionGraphInfoStore+and+the+JobResultStore+into+a+single+component+Comp

Re: [VOTE] FLIP-402: Extend ZooKeeper Curator configurations

2024-03-22 Thread Alex Nitavsky
Thank you all! Closing the vote. The result will be sent in a separate email. On Thu, Mar 21, 2024 at 11:01 AM gongzhongqiang wrote: > +1 (non-binding) > > > Best, > Zhongqiang Gong > > Alex Nitavsky 于2024年3月7日周四 23:09写道: > > > Hi everyone, > > > > I'd like to start a vote on FLIP-402 [1]. It

[jira] [Created] (FLINK-34912) change shade pattern from com.ververica.cdc.connectors.shaded to org.apache.flink.cdc.connectors.shaded

2024-03-22 Thread Xiao Huang (Jira)
Xiao Huang created FLINK-34912: -- Summary: change shade pattern from com.ververica.cdc.connectors.shaded to org.apache.flink.cdc.connectors.shaded Key: FLINK-34912 URL: https://issues.apache.org/jira/browse/FLINK-3491

[RESULT][VOTE] FLIP-402: Extend ZooKeeper Curator configurations

2024-03-22 Thread Alex Nitavsky
Hi devs, I'm happy to announce that FLIP-402: Extend ZooKeeper Curator configurations In Flink[1] has been accepted with 5 approving votes (3 binding) [2]: - Matthias Pohl (binding) - Yang Wang (binding) - Ferenc Csaky (non-binding) - Martijn Visser (binding) - Zhongqiang Gong (non-binding)

[jira] [Created] (FLINK-34913) ConcurrentModificationException SubTaskInitializationMetricsBuilder.addDurationMetric

2024-03-22 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-34913: -- Summary: ConcurrentModificationException SubTaskInitializationMetricsBuilder.addDurationMetric Key: FLINK-34913 URL: https://issues.apache.org/jira/browse/FLINK-34913

Re: [DISCUSS] FLIP-399: Flink Connector Doris

2024-03-22 Thread wudi
Hi, Feng, Thank you, that's a great suggestion ! I have already implemented FilterPushDown and removed that parameter on DorisDynamicTableSource[1], and also updated FLIP. Regarding the mention of [Doris also aborts transactions], it may not have been described accurately. It mainly refers to

Re: [DISCUSS] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State

2024-03-22 Thread yue ma
Hi jinzhong, Thanks for you reply. I still have some doubts about the first question. Is there such a case When you made a snapshot during the synchronization phase, you recorded the current and manifest 8, but before asynchronous phase, the manifest reached the size threshold and then the CURRENT

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-22 Thread Muhammet Orazov
Congratulations! Thanks for the great effort! Best, Muhammet Orazov On 2024-03-20 13:34, Leonard Xu wrote: Hi devs and users, We are thrilled to announce that the donation of Flink CDC as a sub-project of Apache Flink has completed. We invite you to explore the new resources available: - G

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

2024-03-22 Thread Timo Walther
Hi Ron, thanks for the detailed answer. Sorry, for my late reply, we had a conference that kept me busy. > In the current concept[1], it actually includes: Dynamic Tables & > & Continuous Query. Dynamic Table is just an abstract logical concept This explanation makes sense to me. But the docs

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-03-22 Thread David Radley
Hi Jeyhun, Thanks for your feedback. So for outbound messages, the message includes the global ID. We register the schema and match on the artifact id. So if the schema then evolved, adding a new version, the global ID would still be unique and the same version would be targeted. If you wanted

[RESULT][VOTE] FLIP-436: Introduce Catalog-related Syntax

2024-03-22 Thread Yubin Li
Hi everyone, I'm happy to announce that FLIP-436: Introduce Catalog-related Syntax [1] has been accepted with 10 approving votes (5 binding) [2]: - Yuepeng Pan (non-binding) - Jark Wu (binding) - Ferenc Csaky (non-binding) - Feng Jin (non-binding) - Lincoln Lee (binding) - Leonard Xu (binding) -

[jira] [Created] (FLINK-34914) FLIP-436: Introduce Catalog-related Syntax

2024-03-22 Thread Yubin Li (Jira)
Yubin Li created FLINK-34914: Summary: FLIP-436: Introduce Catalog-related Syntax Key: FLINK-34914 URL: https://issues.apache.org/jira/browse/FLINK-34914 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-34915) Introduce `DESCRIBE CATALOG` syntax

2024-03-22 Thread Yubin Li (Jira)
Yubin Li created FLINK-34915: Summary: Introduce `DESCRIBE CATALOG` syntax Key: FLINK-34915 URL: https://issues.apache.org/jira/browse/FLINK-34915 Project: Flink Issue Type: Sub-task Co

[jira] [Created] (FLINK-34916) Support `ALTER CATALOG` syntax

2024-03-22 Thread Yubin Li (Jira)
Yubin Li created FLINK-34916: Summary: Support `ALTER CATALOG` syntax Key: FLINK-34916 URL: https://issues.apache.org/jira/browse/FLINK-34916 Project: Flink Issue Type: Sub-task Compone

[jira] [Created] (FLINK-34917) Support enhanced `CREATE CATALOG` syntax

2024-03-22 Thread Yubin Li (Jira)
Yubin Li created FLINK-34917: Summary: Support enhanced `CREATE CATALOG` syntax Key: FLINK-34917 URL: https://issues.apache.org/jira/browse/FLINK-34917 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-399: Flink Connector Doris

2024-03-22 Thread Feng Jin
Hi Di, Thank you for the update, as well as quickly implementing corresponding capabilities including filter push down and project push down. Regarding the transaction timeout, I still have some doubts. I would like to confirm if we can control this timeout parameter in the connector, such as set

[jira] [Created] (FLINK-34918) Introduce the support of catalog for comments

2024-03-22 Thread Yubin Li (Jira)
Yubin Li created FLINK-34918: Summary: Introduce the support of catalog for comments Key: FLINK-34918 URL: https://issues.apache.org/jira/browse/FLINK-34918 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34919) WebMonitorEndpointTest.cleansUpExpiredExecutionGraphs fails starting REST server

2024-03-22 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-34919: --- Summary: WebMonitorEndpointTest.cleansUpExpiredExecutionGraphs fails starting REST server Key: FLINK-34919 URL: https://issues.apache.org/jira/browse/FLINK-34919 Projec

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

2024-03-22 Thread Lincoln Lee
Hi Timo, Thanks for your thoughtful inputs! Yes, expanding the MATERIALIZED VIEW(MV) could achieve the same function, but our primary concern is that by using a view, we might limit future opportunities to optimize queries through automatic materialization rewriting [1], leveraging the support fo

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-03-22 Thread Jeyhun Karimov
Hi David, Thanks a lot for clarification. Sounds good to me. Regards, Jeyhun On Fri, Mar 22, 2024 at 10:54 AM David Radley wrote: > Hi Jeyhun, > Thanks for your feedback. > > So for outbound messages, the message includes the global ID. We register > the schema and match on the artifact id. So

[jira] [Created] (FLINK-34920) ZooKeeperLeaderRetrievalConnectionHandlingTest

2024-03-22 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-34920: --- Summary: ZooKeeperLeaderRetrievalConnectionHandlingTest Key: FLINK-34920 URL: https://issues.apache.org/jira/browse/FLINK-34920 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

2024-03-22 Thread Timo Walther
Hi everyone, this is a very important change to the Flink SQL syntax but we can't wait until the SQL standard is ready for this. So I'm +1 on introducing the MODEL concept as a first class citizen in Flink. For your information: Over the past months I have already spent a significant amount

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

2024-03-22 Thread Jark Wu
Hi Timo, Introducing user-defined PTF is very useful in Flink, I'm +1 for this. But I think the ML model FLIP is not blocked by this, because we can introduce ML_PREDICT and ML_EVALUATE as built-in PTFs just like TUMBLE/HOP. And support user-defined ML functions as a future FLIP. Regarding the si

Re: [DISCUSS] FLIP-399: Flink Connector Doris

2024-03-22 Thread wudi
Hi, Feng, 1. Are you suggesting that when a commit gets stuck, we can interrupt the commit request using a timeout parameter? Currently, there is no such parameter. In my understanding, in a two-phase commit, checkpoint must be enabled, so the commit timeout is essentially the checkpoint timeou

[jira] [Created] (FLINK-34921) SystemProcessingTimeServiceTest fails due to missing output

2024-03-22 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34921: - Summary: SystemProcessingTimeServiceTest fails due to missing output Key: FLINK-34921 URL: https://issues.apache.org/jira/browse/FLINK-34921 Project: Flink

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-22 Thread lorenzo . affetti
Thank you Yanfei for addressing all the questions! > I'm not sure if I understand your question. In Java, this case(modifying the local local variable) is not allowed[1], but there are ways to get around the limitation of lambda. In this case, the modification to x may be concurrent, which needs t

Re: [DISCUSS] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State

2024-03-22 Thread Zakelly Lan
Hi Yue, Thanks for bringing this up! The CURRENT FILE is the special one, which should be snapshot during the sync phase (temporary load into memory). Thus we can solve this. Best, Zakelly On Fri, Mar 22, 2024 at 4:55 PM yue ma wrote: > Hi jinzhong, > Thanks for you reply. I still have some

Re: Flink Kubernetes Operator Failing Over FlinkDeployments to a New Cluster

2024-03-22 Thread Kevin Lam
Thanks for sharing this work Gyula! That's great to see the FLIP covers some of the limitations already. I will follow the FLIP and associated JIRA ticket. Hi Matthias Pohl. I'd be interested to learn if there has been any progress on the FLIP-360 or associated JIRA issue FLINK-31709. On Fri, Mar

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

2024-03-22 Thread Ahmed Hamdy
Hi Ron, Sorry for joining the discussion late, thanks for the effort. I think the base idea is great, however I have a couple of comments: - I want to iterate on Timo's comments regarding the confusion between "Dynamic Table" and current Flink "Table". Should the refactoring of the system happen i

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

2024-03-22 Thread Jark Wu
Sorry, I mean we can bump the Calcite version if needed in Flink 1.20. On Fri, 22 Mar 2024 at 22:19, Jark Wu wrote: > Hi Timo, > > Introducing user-defined PTF is very useful in Flink, I'm +1 for this. > But I think the ML model FLIP is not blocked by this, because we > can introduce ML_PREDICT

[jira] [Created] (FLINK-34922) Exception History should support multiple Global failures

2024-03-22 Thread Panagiotis Garefalakis (Jira)
Panagiotis Garefalakis created FLINK-34922: -- Summary: Exception History should support multiple Global failures Key: FLINK-34922 URL: https://issues.apache.org/jira/browse/FLINK-34922 Project:

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

2024-03-22 Thread Hao Li
Hi Timo and Jark, I agree Oracle's syntax seems concise and more descriptive. For the built-in `ML_PREDICT` and `ML_EVALUATE` functions I agree with Jark we can support them as built-in PTF using `SqlTableFunction` for this FLIP. We can have a different FLIP discussing user defined PTF and adopt t

[jira] [Created] (FLINK-34923) Behavioral discrepancy between `TableEnvironment.execute_sql()` and `TableEnvironment.sql_query()`

2024-03-22 Thread Chloe He (Jira)
Chloe He created FLINK-34923: Summary: Behavioral discrepancy between `TableEnvironment.execute_sql()` and `TableEnvironment.sql_query()` Key: FLINK-34923 URL: https://issues.apache.org/jira/browse/FLINK-34923

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

2024-03-22 Thread Ron liu
Hi, Feng Thanks for your feedback. > Although currently we restrict users from modifying the query, I wonder if we can provide a better way to help users rebuild it without affecting downstream OLAP queries. Considering the problem of data consistency, so in the first step we are strictly limite