[
https://issues.apache.org/jira/browse/FLINK-37132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17946051#comment-17946051
]
Yanquan Lv edited comment on FLINK-37132 at 4/21/25 6:16 AM:
-------------------------------------------------------------
Hi, [~MOBIN].
Consider the situation that the user wishes to add an identifier_name field to
all tables, but adds an additional field to a certain table.
The job would be written as:
transform:
- source-table: mydb.web_order
projection: *, order_id, UPPER(product_name) as product_name,
__namespace_name__ || '.' || __schema_name__ || '.' || __table_name__
identifier_name
partition-keys: product_name
- source-table: \.{*}.\.{*}
projection: *, __namespace_name__ || '.' || __schema_name__ || '.' ||
__table_name__ identifier_name
Because we do not have the exclude.table function in transform, if we do not
support this kind of configuration, it will be difficult for users to meet
their needs.
was (Author: JIRAUSER304414):
Hi, [~MOBIN].
Consider the situation that the user wishes to add an identifier_name field to
all tables, but adds an additional field to a certain table.
The job would be written as:
transform:
- source-table: mydb.web_order
projection: *, order_id, UPPER(product_name) as product_name,
_{_}namespace_name{_}_ || '.' || _{_}schema_name{_}_ || '.' ||
_{_}table_name{_}_ identifier_name
partition-keys: product_name
- source-table: \.{*}.\.{*}
projection: *, _{_}namespace_name{_}_ || '.' || _{_}schema_name{_}_ || '.'
|| _{_}table_name{_}_ identifier_name
Because we do not have the exclude.table function in transform, if we do not
support this kind of configuration, it will be difficult for users to meet
their needs.
> Add schema validation in Multi Transform
> ----------------------------------------
>
> Key: FLINK-37132
> URL: https://issues.apache.org/jira/browse/FLINK-37132
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.2.0, cdc-3.2.1
> Reporter: MOBIN
> Assignee: MOBIN
> Priority: Major
> Labels: pull-request-available
>
> The following scenarios should throw an exception of [different column count]
> {code:java}
> void testMultiTransformSchemaColumnsCompatibilityWithDiffColumnCount(
> ValuesDataSink.SinkApi sinkApi) {
> assertThatThrownBy(
> () ->
> runGenericTransformTest(
> sinkApi,
> Arrays.asList(
> new TransformDef(
>
> "default_namespace.default_schema.mytable2",
> "*",
> "age < 18",
> null,
> null,
> null,
> null,
> null),
> new TransformDef(
>
> "default_namespace.default_schema.mytable2",
> // reference part column
> "id,UPPER(name) AS name",
> "age >= 18",
> null,
> null,
> null,
> null,
> null)),
> Collections.emptyList()))
> .rootCause()
> .isExactlyInstanceOf(IllegalStateException.class)
> .hasMessage(
> "Unable to merge schema columns={`id` BIGINT NOT
> NULL,`name` VARCHAR(255),`age` TINYINT,`description` STRING}, primaryKeys=id,
> options=() "
> + "and columns={`id` BIGINT NOT NULL,`name`
> STRING}, primaryKeys=id, options=() with different column counts.");
> } {code}
> In Multi Transform, metadata fields like primaryKeys, partitionKeys, and
> options also need to be consistent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
