[
https://issues.apache.org/jira/browse/FLINK-39171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tianfy updated FLINK-39171:
---------------------------
Description:
In Oracle CDC {*}pipeline mode{*}, transform projection can fail when
projection uses quoted uppercase source columns (for example {{{}`ID`{}}}),
with:
{{Column 'ID' not found in any table}}
Observed behavior suggests Oracle pipeline schema columns are normalized to
lowercase internally (for example {{{}id{}}}), while transform SQL validation
treats quoted identifiers as exact/case-sensitive, causing mismatch.
h3. Reproduction
# Create Oracle table (unquoted column names, Oracle stores as uppercase):
** {{TESTUSER.ORDERS(ID, PRODUCT_ID, QUANTITY, UPDATED_AT)}}
# Run Oracle pipeline with transform projection using quoted uppercase source
columns, e.g.:
pipeline:
schema.change.behavior: LENIENT
transform:
- source-table: ORCL19.TESTUSER.ORDERS
primary-keys: id
projection: "`ID` as `id`,`PRODUCT_ID` as `product_id`,`QUANTITY` as
`quantity`,`UPDATED_AT` as `updated_at`"
source:
type: oracle
database: ORCL19
tables: ORCL19.TESTUSER.ORDERS
scan.startup.mode: initial
3. Start the job.
h3. Actual behavior
Job fails during post-transform schema handling:
*
org.apache.flink.cdc.runtime.operators.transform.exceptions.TransformException
* Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column
'ID' not found in any table
Stack path includes: PostTransformOperator ->
TransformParser.generateProjectionColumns -> Calcite validator
h3. Expected behavior
Oracle pipeline should use a consistent column-name strategy between schema
and transform projection:
# Either preserve Oracle column case consistently, or
# Normalize both schema and projection consistently, or
# Resolve quoted identifiers safely for Oracle semantics.
At minimum, projection generated from source metadata should not fail with
column-not-found because of internal case normalization mismatch.
h3. Impact
* Oracle pipeline job fails at startup for common projection patterns using
original DB column names.
* Blocks end-to-end sync in transform-enabled pipelines.
h3. Workaround
* Convert Oracle source field names in projection to lowercase before
submission (for example {{id}} instead of {{ID}}).
* Avoid quoted uppercase source identifiers in transform projection.
h3. Additional notes
If needed, I can provide a minimal runnable reproducer (DDL + full pipeline
YAML + complete stack trace).
was:
In Oracle CDC *pipeline mode*, transform projection can fail when projection
uses quoted uppercase source columns (for example {{`ID`}}), with:
{{Column 'ID' not found in any table}}
Observed behavior suggests Oracle pipeline schema columns are normalized to
lowercase internally (for example {{id}}), while transform SQL validation
treats quoted identifiers as exact/case-sensitive, causing mismatch.
This does not reproduce in our non-pipeline Oracle CDC source path.
h3. Reproduction
# Create Oracle table (unquoted column names, Oracle stores as uppercase):
** {{TESTUSER.ORDERS(ID, PRODUCT_ID, QUANTITY, UPDATED_AT)}}
# Run Oracle pipeline with transform projection using quoted uppercase source
columns, e.g.:
pipeline:
schema.change.behavior: LENIENT
transform:
- source-table: ORCL19.TESTUSER.ORDERS
primary-keys: id
projection: "`ID` as `id`,`PRODUCT_ID` as `product_id`,`QUANTITY` as
`quantity`,`UPDATED_AT` as `updated_at`"
source:
type: oracle
database: ORCL19
tables: ORCL19.TESTUSER.ORDERS
scan.startup.mode: initial
3. Start the job.
> Oracle pipeline transform fails with quoted uppercase projection columns
> ------------------------------------------------------------------------
>
> Key: FLINK-39171
> URL: https://issues.apache.org/jira/browse/FLINK-39171
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.6.0
> Environment: * Flink: 1.20.1
> * Oracle: 19c (Non-CDB,)
> * Startup mode: {{initial}}
> * Flink CDC：3.6.0
> Reporter: tianfy
> Priority: Major
>
> In Oracle CDC {*}pipeline mode{*}, transform projection can fail when
> projection uses quoted uppercase source columns (for example {{{}`ID`{}}}),
> with:
> {{Column 'ID' not found in any table}}
> Observed behavior suggests Oracle pipeline schema columns are normalized to
> lowercase internally (for example {{{}id{}}}), while transform SQL validation
> treats quoted identifiers as exact/case-sensitive, causing mismatch.
> h3. Reproduction
> # Create Oracle table (unquoted column names, Oracle stores as uppercase):
> ** {{TESTUSER.ORDERS(ID, PRODUCT_ID, QUANTITY, UPDATED_AT)}}
> # Run Oracle pipeline with transform projection using quoted uppercase
> source columns, e.g.:
> pipeline:
> schema.change.behavior: LENIENT
>
> transform:
> - source-table: ORCL19.TESTUSER.ORDERS
> primary-keys: id
> projection: "`ID` as `id`,`PRODUCT_ID` as `product_id`,`QUANTITY` as
> `quantity`,`UPDATED_AT` as `updated_at`"
>
> source:
> type: oracle
> database: ORCL19
> tables: ORCL19.TESTUSER.ORDERS
> scan.startup.mode: initial
>
> 3. Start the job.
> h3. Actual behavior
> Job fails during post-transform schema handling:
> *
> org.apache.flink.cdc.runtime.operators.transform.exceptions.TransformException
> * Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column
> 'ID' not found in any table
> Stack path includes: PostTransformOperator ->
> TransformParser.generateProjectionColumns -> Calcite validator
> h3. Expected behavior
> Oracle pipeline should use a consistent column-name strategy between schema
> and transform projection:
> # Either preserve Oracle column case consistently, or
> # Normalize both schema and projection consistently, or
> # Resolve quoted identifiers safely for Oracle semantics.
> At minimum, projection generated from source metadata should not fail with
> column-not-found because of internal case normalization mismatch.
> h3. Impact
> * Oracle pipeline job fails at startup for common projection patterns using
> original DB column names.
> * Blocks end-to-end sync in transform-enabled pipelines.
> h3. Workaround
> * Convert Oracle source field names in projection to lowercase before
> submission (for example {{id}} instead of {{ID}}).
> * Avoid quoted uppercase source identifiers in transform projection.
> h3. Additional notes
> If needed, I can provide a minimal runnable reproducer (DDL + full pipeline
> YAML + complete stack trace).
>
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
