[jira] [Created] (FLINK-35664) Flink CDC pipeline transform supports mask function
melin created FLINK-35664: - Summary: Flink CDC pipeline transform supports mask function Key: FLINK-35664 URL: https://issues.apache.org/jira/browse/FLINK-35664 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: melin Fix For: cdc-3.2.0 Flink CDC pipeline transform supports mask function -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35538) pipeline sink support caseSensitive
melin created FLINK-35538: - Summary: pipeline sink support caseSensitive Key: FLINK-35538 URL: https://issues.apache.org/jira/browse/FLINK-35538 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.2.0, cdc-3.1.1 Reporter: melin Attachments: image-2024-06-06-15-51-02-428.png source is case sensitive, sink is case insensitive. Even paimon doesn't allow capital letters !image-2024-06-06-15-51-02-428.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35511) Enhance metrics. The incremental and full phases count the number of records separately
melin created FLINK-35511: - Summary: Enhance metrics. The incremental and full phases count the number of records separately Key: FLINK-35511 URL: https://issues.apache.org/jira/browse/FLINK-35511 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: melin Attachments: image-2024-06-03-22-06-38-591.png dataworks 实时同步的metric 信息 !image-2024-06-03-22-06-38-591.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35478) Pipeline sink Unified primary.key parameter
melin created FLINK-35478: - Summary: Pipeline sink Unified primary.key parameter Key: FLINK-35478 URL: https://issues.apache.org/jira/browse/FLINK-35478 Project: Flink Issue Type: Improvement Reporter: melin Customize the primary key of the sink table. For example, if multiple different area tables write to the same paimon table, the primary key must be combined to avoid duplicate primary keys -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35417) JobManager and TaskManager support merging and run in a single process
melin created FLINK-35417: - Summary: JobManager and TaskManager support merging and run in a single process Key: FLINK-35417 URL: https://issues.apache.org/jira/browse/FLINK-35417 Project: Flink Issue Type: Improvement Components: API / Core Reporter: melin flink is widely used in data integration scenarios, where a single concurrency is not high, and in many cases a single concurrency can run a task. Consider the high availability, application mode, and large number of JobManger nodes that cost a lot of resources. If the Session mode is used, the stability is not high. In application mode, JobManager and TaskManager can be run together to achieve reliability and save resources. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35394) paimon pipeline support dynamic bucket
melin created FLINK-35394: - Summary: paimon pipeline support dynamic bucket Key: FLINK-35394 URL: https://issues.apache.org/jira/browse/FLINK-35394 Project: Flink Issue Type: New Feature Components: Flink CDC Reporter: melin paimon pipeline support dynamic bucket -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35393) Support flink kafka catalog
melin created FLINK-35393: - Summary: Support flink kafka catalog Key: FLINK-35393 URL: https://issues.apache.org/jira/browse/FLINK-35393 Project: Flink Issue Type: New Feature Components: Connectors / Kafka Reporter: melin Support for flink kafka catalog -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35329) Support modify table/column comments
melin created FLINK-35329: - Summary: Support modify table/column comments Key: FLINK-35329 URL: https://issues.apache.org/jira/browse/FLINK-35329 Project: Flink Issue Type: New Feature Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: melin The database has ddl sql to change the comment of the table/column. It is recommended to add the change comment event. You can also set unsynchronized comments. Some scenarios cannot synchronize comments. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35303) Support logical deletion of datax
melin created FLINK-35303: - Summary: Support logical deletion of datax Key: FLINK-35303 URL: https://issues.apache.org/jira/browse/FLINK-35303 Project: Flink Issue Type: New Feature Components: Flink CDC Reporter: melin delete event is logical deletion. Add a field to the table. For example: is_delete, the default is false, if it is a delete event, is_delete is set to true。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35248) Support table/column comment to modify DDL
melin created FLINK-35248:
-
Summary: Support table/column comment to modify DDL
Key: FLINK-35248
URL: https://issues.apache.org/jira/browse/FLINK-35248
Project: Flink
Issue Type: New Feature
Components: Flink CDC
Reporter: melin
Table and column comment changes are synchronized to downstream tables：
{code:java}
alter table orders comment '订单'; alter table orders modify address
varchar(1024) null comment '地址'; {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35179) add postgres pipeline data sink connector
melin created FLINK-35179: - Summary: add postgres pipeline data sink connector Key: FLINK-35179 URL: https://issues.apache.org/jira/browse/FLINK-35179 Project: Flink Issue Type: New Feature Components: Flink CDC Reporter: melin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35172) DDL statement is added to the Schema Change Event
melin created FLINK-35172: - Summary: DDL statement is added to the Schema Change Event Key: FLINK-35172 URL: https://issues.apache.org/jira/browse/FLINK-35172 Project: Flink Issue Type: New Feature Components: Flink CDC Reporter: melin The current implementation of the kafka pipeline data sink connector does not write ddl statements to the topic because the original dddl statements are missing. ddl cannot be generated backwards using a Schema Change Event. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35104) Add kafka pipeline data source connector
melin created FLINK-35104: - Summary: Add kafka pipeline data source connector Key: FLINK-35104 URL: https://issues.apache.org/jira/browse/FLINK-35104 Project: Flink Issue Type: New Feature Components: Flink CDC Reporter: melin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35099) Support for multiple datasources
melin created FLINK-35099: - Summary: Support for multiple datasources Key: FLINK-35099 URL: https://issues.apache.org/jira/browse/FLINK-35099 Project: Flink Issue Type: New Feature Components: Flink CDC Reporter: melin In a hyperscale scenario, databases are distributed on multiple mysql database servers instead of one database server. In our case, there are up to 16 servers. Support for multiple datasources in a single flink cdc task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32766) Support kafka catalog
melin created FLINK-32766: - Summary: Support kafka catalog Key: FLINK-32766 URL: https://issues.apache.org/jira/browse/FLINK-32766 Project: Flink Issue Type: New Feature Reporter: melin Support kafka catalog -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32251) flink jdbc catalog cannot add parameters to url
melin created FLINK-32251: - Summary: flink jdbc catalog cannot add parameters to url Key: FLINK-32251 URL: https://issues.apache.org/jira/browse/FLINK-32251 Project: Flink Issue Type: New Feature Reporter: melin For example, add an encoding parameter to the mysql url： jdbc:mysql://host:port/database?useUnicode=true=UTF-8 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32163) Support the same application run multiple jobs in HA mode
melin created FLINK-32163: - Summary: Support the same application run multiple jobs in HA mode Key: FLINK-32163 URL: https://issues.apache.org/jira/browse/FLINK-32163 Project: Flink Issue Type: New Feature Reporter: melin Support the same application run multiple jobs in HA mode -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32125) Support Unified job submission API
melin created FLINK-32125: - Summary: Support Unified job submission API Key: FLINK-32125 URL: https://issues.apache.org/jira/browse/FLINK-32125 Project: Flink Issue Type: New Feature Reporter: melin Based on the flink secondary development platform, different deployment modes and yarn and kubernetes modes differ greatly, and there is no unified API. SparkLauncher is recommended。https://github.com/apache/spark/blob/master/launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30126) Delay registration of the catalog, register the catalog as needed
melin created FLINK-30126: - Summary: Delay registration of the catalog, register the catalog as needed Key: FLINK-30126 URL: https://issues.apache.org/jira/browse/FLINK-30126 Project: Flink Issue Type: New Feature Reporter: melin Data platform has registered many relational database data sources such as mysql, data source code is used as the catalog name, we are not sure which data source needs to register the catalog in flink, we hope that the required catalog can be dynamically loaded when sql is executed, flink provides the interface. Users can customize the register time catalog -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30123) Optimize ApplicationDeployer API design
melin created FLINK-30123: - Summary: Optimize ApplicationDeployer API design Key: FLINK-30123 URL: https://issues.apache.org/jira/browse/FLINK-30123 Project: Flink Issue Type: New Feature Reporter: melin Attachments: image-2022-11-22-09-28-38-631.png, image-2022-11-22-09-28-54-660.png 采用 ApplicationDeployer api 提交任务，run是同步等待提交完成，如果任务提交yarn，可能一直都是accepted 状态，这个时候没有获取yarn applicationID，如果想取消任务是比较难的，建议参考 org.apache.spark.launcher.SparkLauncher 设计，异步提交任务，能够尽快获取applicationId，如果想提前删掉任务，直接yarn application -kill xxx; !image-2022-11-22-09-28-38-631.png! !image-2022-11-22-09-28-54-660.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30121) Support Stored procedures
melin created FLINK-30121: - Summary: Support Stored procedures Key: FLINK-30121 URL: https://issues.apache.org/jira/browse/FLINK-30121 Project: Flink Issue Type: New Feature Components: Table SQL / Runtime Reporter: melin Supports operations like hudi/iceberg calls, such as savepoint [https://hudi.apache.org/docs/procedures/] CALL system.procedure_name(arg_1, arg_2, ... arg_n) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-25760) Support for extended SQL syntax
melin created FLINK-25760: - Summary: Support for extended SQL syntax Key: FLINK-25760 URL: https://issues.apache.org/jira/browse/FLINK-25760 Project: Flink Issue Type: Improvement Reporter: melin Supports extended SQL syntax, similar to Spark Extensions Feature. Custom SQL syntax can be implemented -- This message was sent by Atlassian Jira (v8.20.1#820001)
