[
https://issues.apache.org/jira/browse/FLINK-39171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tianfy updated FLINK-39171:
---------------------------
Description:
In Oracle CDC {*}pipeline mode{*}, transform projection can fail when
projection uses quoted uppercase source columns (for example {{{}`ID`{}}}),
with:
{{Column 'ID' not found in any table}}
Observed behavior suggests Oracle pipeline schema columns are normalized to
lowercase internally (for example {{{}id{}}}), while transform SQL validation
treats quoted identifiers as exact/case-sensitive, causing mismatch.
h3. Reproduction
# Create Oracle table (unquoted column names, Oracle stores as uppercase):
{{ TESTUSER.ORDERS(ID, PRODUCT_ID, QUANTITY, UPDATED_AT)}}
2. Run Oracle pipeline with transform projection using quoted uppercase
source columns, e.g.:
pipeline:
schema.change.behavior: LENIENT
transform:
- source-table: ORCL19.TESTUSER.ORDERS
primary-keys: id
projection: "`ID` as `id`,`PRODUCT_ID` as `product_id`,`QUANTITY` as
`quantity`,`UPDATED_AT` as `updated_at`"
source:
type: oracle
database: ORCL19
tables: ORCL19.TESTUSER.ORDERS
scan.startup.mode: initial
3. Start the job.
h3. Actual behavior
Job fails during post-transform schema handling:
*
org.apache.flink.cdc.runtime.operators.transform.exceptions.TransformException
* Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column
'ID' not found in any table
Stack path includes: PostTransformOperator ->
TransformParser.generateProjectionColumns -> Calcite validator
h3. Expected behavior
Oracle pipeline should use a consistent column-name strategy between schema and
transform projection:
# Either preserve Oracle column case consistently, or
2. Normalize both schema and projection consistently, or
3. Resolve quoted identifiers safely for Oracle semantics.
At minimum, projection generated from source metadata should not fail with
column-not-found because of internal case normalization mismatch.
h3. Impact
* Oracle pipeline job fails at startup for common projection patterns using
original DB column names.
* Blocks end-to-end sync in transform-enabled pipelines.
h3. Workaround
* Convert Oracle source field names in projection to lowercase before
submission (for example {{id}} instead of {{{}ID{}}}).
* Avoid quoted uppercase source identifiers in transform projection.
h3. Additional notes
If needed, I can provide a minimal runnable reproducer (DDL + full pipeline
YAML + complete stack trace).
was:
In Oracle CDC {*}pipeline mode{*}, transform projection can fail when
projection uses quoted uppercase source columns (for example {{{}`ID`{}}}),
with:
{{Column 'ID' not found in any table}}
Observed behavior suggests Oracle pipeline schema columns are normalized to
lowercase internally (for example {{{}id{}}}), while transform SQL validation
treats quoted identifiers as exact/case-sensitive, causing mismatch.
h3. Reproduction
# Create Oracle table (unquoted column names, Oracle stores as uppercase):
* {{TESTUSER.ORDERS(ID, PRODUCT_ID, QUANTITY, UPDATED_AT)}}
2. Run Oracle pipeline with transform projection using quoted uppercase
source columns, e.g.:
pipeline:
schema.change.behavior: LENIENT
transform:
- source-table: ORCL19.TESTUSER.ORDERS
primary-keys: id
projection: "`ID` as `id`,`PRODUCT_ID` as `product_id`,`QUANTITY` as
`quantity`,`UPDATED_AT` as `updated_at`"
source:
type: oracle
database: ORCL19
tables: ORCL19.TESTUSER.ORDERS
scan.startup.mode: initial
3. Start the job.
h3. Actual behavior
Job fails during post-transform schema handling:
*
org.apache.flink.cdc.runtime.operators.transform.exceptions.TransformException
* Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column
'ID' not found in any table
Stack path includes: PostTransformOperator ->
TransformParser.generateProjectionColumns -> Calcite validator
h3. Expected behavior
Oracle pipeline should use a consistent column-name strategy between schema and
transform projection:
# Either preserve Oracle column case consistently, or
# Normalize both schema and projection consistently, or
# Resolve quoted identifiers safely for Oracle semantics.
At minimum, projection generated from source metadata should not fail with
column-not-found because of internal case normalization mismatch.
h3. Impact
* Oracle pipeline job fails at startup for common projection patterns using
original DB column names.
* Blocks end-to-end sync in transform-enabled pipelines.
h3. Workaround
* Convert Oracle source field names in projection to lowercase before
submission (for example {{id}} instead of {{{}ID{}}}).
* Avoid quoted uppercase source identifiers in transform projection.
h3. Additional notes
If needed, I can provide a minimal runnable reproducer (DDL + full pipeline
YAML + complete stack trace).
> Oracle pipeline transform fails with quoted uppercase projection columns
> ------------------------------------------------------------------------
>
> Key: FLINK-39171
> URL: https://issues.apache.org/jira/browse/FLINK-39171
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.6.0
> Environment: * Flink: 1.20.1
> * Oracle: 19c (Non-CDB,)
> * Startup mode: {{initial}}
> * Flink CDC：3.6.0
> Reporter: tianfy
> Priority: Major
>
> In Oracle CDC {*}pipeline mode{*}, transform projection can fail when
> projection uses quoted uppercase source columns (for example {{{}`ID`{}}}),
> with:
> {{Column 'ID' not found in any table}}
> Observed behavior suggests Oracle pipeline schema columns are normalized to
> lowercase internally (for example {{{}id{}}}), while transform SQL validation
> treats quoted identifiers as exact/case-sensitive, causing mismatch.
> h3. Reproduction
> # Create Oracle table (unquoted column names, Oracle stores as uppercase):
> {{ TESTUSER.ORDERS(ID, PRODUCT_ID, QUANTITY, UPDATED_AT)}}
> 2. Run Oracle pipeline with transform projection using quoted
> uppercase source columns, e.g.:
> pipeline:
> schema.change.behavior: LENIENT
>
> transform:
> - source-table: ORCL19.TESTUSER.ORDERS
> primary-keys: id
> projection: "`ID` as `id`,`PRODUCT_ID` as `product_id`,`QUANTITY` as
> `quantity`,`UPDATED_AT` as `updated_at`"
>
> source:
> type: oracle
> database: ORCL19
> tables: ORCL19.TESTUSER.ORDERS
> scan.startup.mode: initial
>
> 3. Start the job.
> h3. Actual behavior
> Job fails during post-transform schema handling:
> *
> org.apache.flink.cdc.runtime.operators.transform.exceptions.TransformException
> * Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column
> 'ID' not found in any table
> Stack path includes: PostTransformOperator ->
> TransformParser.generateProjectionColumns -> Calcite validator
> h3. Expected behavior
> Oracle pipeline should use a consistent column-name strategy between schema
> and transform projection:
> # Either preserve Oracle column case consistently, or
> 2. Normalize both schema and projection consistently, or
> 3. Resolve quoted identifiers safely for Oracle semantics.
> At minimum, projection generated from source metadata should not fail with
> column-not-found because of internal case normalization mismatch.
> h3. Impact
> * Oracle pipeline job fails at startup for common projection patterns using
> original DB column names.
> * Blocks end-to-end sync in transform-enabled pipelines.
> h3. Workaround
> * Convert Oracle source field names in projection to lowercase before
> submission (for example {{id}} instead of {{{}ID{}}}).
> * Avoid quoted uppercase source identifiers in transform projection.
> h3. Additional notes
> If needed, I can provide a minimal runnable reproducer (DDL + full pipeline
> YAML + complete stack trace).
>
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
