[
https://issues.apache.org/jira/browse/FLINK-35256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yux updated FLINK-35256:
------------------------
Description:
Flink CDC 3.1.0 brought transform feature, allowing column type / value
transformation prior to data routing process. However after the transformation,
column type marked as `NOT NULL` lost their annotation, causing some downstream
sinks to fail since they require primary key to be NOT NULL.
Here's the minimum reproducible example about this problem:
```yaml
source:
type: mysql
...
sink:
type: starrocks
name: StarRocks Sink
...
pipeline:
name: Sync MySQL Database to StarRocks
parallelism: 4
transform:
- source-table: reicigo.\.*
projection: ID, UPPER(ID) AS UPID
```
In the MySQL source table, primary key column `ID` is marked as `NOT NULL`, but
such information was lost at downstream, causing the following exception (see
attachment).
was:
Flink CDC 3.1.0 brought transform feature, allowing column type / value
transformation prior to data routing process. However after the transformation,
column type marked as `NOT NULL` lost their annotation, causing some downstream
sinks to fail since they require primary key to be NOT NULL.
Here's the minimum reproducible example about this problem:
```yaml
source:
type: mysql
...
sink:
type: starrocks
name: StarRocks Sink
...
pipeline:
name: Sync MySQL Database to StarRocks
parallelism: 4
transform:
- source-table: reicigo.\.*
projection: ID, UPPER(ID) AS UPID
```
In the MySQL source table, primary key column `ID` is marked as `NOT NULL`, but
such information was lost at downstream, causing the following exception (see
attachment).
> Pipeline transform ignores column type nullability
> --------------------------------------------------
>
> Key: FLINK-35256
> URL: https://issues.apache.org/jira/browse/FLINK-35256
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Reporter: yux
> Priority: Major
> Attachments: log.txt
>
>
> Flink CDC 3.1.0 brought transform feature, allowing column type / value
> transformation prior to data routing process. However after the
> transformation, column type marked as `NOT NULL` lost their annotation,
> causing some downstream sinks to fail since they require primary key to be
> NOT NULL.
> Here's the minimum reproducible example about this problem:
> ```yaml
> source:
> type: mysql
> ...
> sink:
> type: starrocks
> name: StarRocks Sink
> ...
> pipeline:
> name: Sync MySQL Database to StarRocks
> parallelism: 4
> transform:
> - source-table: reicigo.\.*
> projection: ID, UPPER(ID) AS UPID
> ```
> In the MySQL source table, primary key column `ID` is marked as `NOT NULL`,
> but such information was lost at downstream, causing the following exception
> (see attachment).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
