[RESULT][VOTE] FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

Hi devs, I'm happy to announce that FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes [1] has been accepted with 7 approving votes (4 binding) [2]: - Yun Tang (binding) - Jeyhun Karimov (non-binding) - Hangxiang Yu (binding) - Yanfei Lei (binding) - Zakelly Lan (non-bindi

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

Hi Jing, Thanks for your attention to this flip! I'll try to answer the following questions. > 1. How to define query of dynamic table? > Use flink sql or introducing new syntax? > If use flink sql, how to handle the difference in SQL between streaming and > batch processing? > For example, a que

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

Hi, Timo Sorry for later response, thanks for your feedback. Regarding your questions: > Flink has introduced the concept of Dynamic Tables many years ago. How does the term "Dynamic Table" fit into Flink's regular tables and also how does it relate to Table API? > I fear that adding the DYN

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

Hi Kevin, Thanks, these are some great points. Just to clarify, I do agree that the subject should be an option (like in the case of RegistryAvroFormatFactory). We could fallback to subject and auto-register schemas, if schema-Id not provided explicitly. In general, I think it would be good to be

Re: [VOTE] Release 1.19.0, release candidate #2

+1 (non-binding) * Verified the signature and checksum. * Reviewed the release note PR * Reviewed the web announcement PR * Start a standalone cluster to submit the state machine example, which works well. * Checked the pre-built jars are generated via JDK8 * Verified the process pro

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

Hi Mingge, Hao, Thanks for your replies. > PTF is actually the ideal approach for model functions, and we do have the plans to use PTF for all model functions (including prediction, evaluation etc..) once the PTF is supported in FlinkSQL confluent extension. It sounds that PTF is the ideal way a

Re: [DISCUSS] FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax

Hi, Yubin. Thanks for the FLIP. +1 for it. Best, Hang Yubin Li 于2024年3月14日周四 10:15写道： > Hi Jingsong, Feng, and Jeyhun > > Thanks for your support and feedback! > > > However, could we add a new method `getCatalogDescriptor()` to > > CatalogManager instead of directly exposing CatalogStore? > >

Re: [DISCUSS] FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax

Thank you Yubin, +1 for the proposal. We have been lacking catalog related syntax to operate catalogs. It's a good chance to complete the syntax as we have introduced CatalogStore. >From what I can see, some useful commands are still missing for catalogs, such as, alter catalog, describe catalog.

[jira] [Created] (FLINK-34664) Add .asf.yaml for Flink CDC

Qingsheng Ren created FLINK-34664: - Summary: Add .asf.yaml for Flink CDC Key: FLINK-34664 URL: https://issues.apache.org/jira/browse/FLINK-34664 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax

Hi Jingsong, Feng, and Jeyhun Thanks for your support and feedback! > However, could we add a new method `getCatalogDescriptor()` to > CatalogManager instead of directly exposing CatalogStore? Good point, Besides the audit tracking issue, The proposed feature only requires `getCatalogDescriptor(

Re: Flink Kubernetes Operator Failing Over FlinkDeployments to a New Cluster

Hey Kevin! The general mismatch I see here is that operators and resources are pretty cluster dependent. The operator itself is running in the same cluster so it feels out of scope to submit resources to different clusters, this doesn't really sound like what any Kubernetes Operator should do in g

Re: Flink Kubernetes Operator Failing Over FlinkDeployments to a New Cluster

Hi Max, It feels a bit hacky to need to back-up the resources directly from the cluster, as opposed to being able to redeploy our checked-in k8s manifests such that they failover correctly, but that makes sense to me and we can look into this approach. Thanks for the suggestion! I'd still be inte

Unable to override PrometheusReporter using PrometheusGateway 'filterLabelValueCharacters'

Hello, I believe there is a bug with the AbstractPrometheusReporter where it requires using `metrics.reporter.promgateway.filterLabelValueCharacters` to ignore filtering label value characters.

[jira] [Created] (FLINK-34663) flink-opensearch connector Unable to parse response body for Response

wael shehata created FLINK-34663: Summary: flink-opensearch connector Unable to parse response body for Response Key: FLINK-34663 URL: https://issues.apache.org/jira/browse/FLINK-34663 Project: Flink

Re: [DISCUSS] FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax

Hi Yubin, Thanks for the proposal. +1 for it. I have one comment: I would like to see the SQL syntax for the proposed statement. Showing the SQL parser implementation in the FLIP for the SQL syntax might be a bit confusing. Also, the formal definition is missing for this SQL clause. Maybe someth

Re: [DISCUSS] FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax

Hi Yubin Thank you for initiating this FLIP. I have just one minor question: I noticed that we added a new function `getCatalogStore` to expose CatalogStore, and it seems fine. However, could we add a new method `getCatalogDescriptor()` to CatalogManager instead of directly exposing CatalogStore

Re: Flink Kubernetes Operator Failing Over FlinkDeployments to a New Cluster

Hi Kevin, Theoretically, as long as you move over all k8s resources, failover should work fine on the Flink and Flink Operator side. The tricky part is the handover. You will need to backup all resources from the old cluster, shutdown the old cluster, then re-create them on the new cluster. The op

Re: [DISCUSS] FLIP-434: Support optimizations for pre-partitioned data sources

Hi Jane, Thanks for your comments. 1. Concerning the `sourcePartitions()` method, the partition information > returned during the optimization phase may not be the same as the partition > information during runtime execution. For long-running jobs, partitions may > be continuously created. Is th

Re: [DISCUSS] FLIP-434: Support optimizations for pre-partitioned data sources

Hi Jim, Thanks for your comments. I wonder if it'd make sense to > generalize FLIP-434 to be about "pre-divided" data to cover "buckets" and > "partitions" (and maybe even situations where a data source is partitioned > and bucketed). Now that I go through FLIP-376 [1] again, your suggestion d

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

indeed! I missed that part. Thanks for the hint! Best regards, Jing On Wed, Mar 13, 2024 at 6:02 AM Zakelly Lan wrote: > Hi Jing, > > The deprecation and removal of original APIs is beyond the scope of > current FLIP, but I do add/highlight such information under "Compatibility, > Deprecation,

[jira] [Created] (FLINK-34662) Add new issue template for Flink CDC repo

Qingsheng Ren created FLINK-34662: - Summary: Add new issue template for Flink CDC repo Key: FLINK-34662 URL: https://issues.apache.org/jira/browse/FLINK-34662 Project: Flink Issue Type: Sub-t

Re: [VOTE] FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

Hi Jinzhong, Thanks for driving this, +1 (non-binding) Best Regards Ahmed Hamdy On Wed, 13 Mar 2024 at 01:33, Jing Ge wrote: > +1 (binding) Thanks! > > Best regards, > Jing > > On Sun, Mar 10, 2024 at 5:32 PM Jing Ge wrote: > > > Hi Jinzhong, > > > > Thanks for driving this topic and sorry for

Re: [DISCUSS] FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax

+1 for this. We are missing a series of catalog related syntaxes. Especially after the introduction of catalog store. [1] [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations Best, Jingsong On Wed, M

[jira] [Created] (FLINK-34661) TaskExecutor supports retain partitions after JM crashed.

Junrui Li created FLINK-34661: - Summary: TaskExecutor supports retain partitions after JM crashed. Key: FLINK-34661 URL: https://issues.apache.org/jira/browse/FLINK-34661 Project: Flink Issue Typ

[DISCUSS] FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax

Hi devs, I'd like to start a discussion about FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax [1]. At present, the `SHOW CREATE TABLE` statement provides strong support for users to easily reuse created tables. However, despite the increasing importance of the `Catalog` in user's business, there

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

Hi, Lincoln & Ron, Thanks for the proposal. I agree with the question raised by Timo. Besides, I have some other questions. 1. How to define query of dynamic table? Use flink sql or introducing new syntax? If use flink sql, how to handle the difference in SQL between streaming and batch processi

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

Okay, sorry, I'm not looking at the latest version of the FLIP. You've answered my question in updated FLIP. :) Best regards, Weijie weijie guo 于2024年3月13日周三 14:56写道： > Hi Zakelly, > > Thanks for the proposal! I like this idea and I can see the performance > improvements it brings. > > In the

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

Hi Jark, Thanks for your questions. These are good questions! 1. The polymorphism table function I was referring to takes a table as input and outputs a table. So the syntax would be like ``` SELECT * FROM ML_PREDICT('model', (SELECT * FROM my_table)) ``` As far as I know, this is not supported y