[
https://issues.apache.org/jira/browse/FLINK-38196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tchivs updated FLINK-38196:
---------------------------
Description:
### Problem Statement
Flink CDC fails with `IndexOutOfBoundsException` when processing PostgreSQL
tables containing `numeric(0)` fields, particularly when these fields have NULL
values. This causes complete pipeline failure during binary decimal data
deserialization.
### Error Details
```
Caused by: java.lang.IndexOutOfBoundsException: pos: -2130706416, length: 48,
index: -2130706432, offset: 0
at org.apache.flink.core.memory.MemorySegment.get(MemorySegment.java:467)
at
org.apache.flink.cdc.common.data.binary.BinarySegmentUtils.copyToBytes(BinarySegmentUtils.java:131)
at
org.apache.flink.cdc.common.data.binary.BinarySegmentUtils.readDecimalData(BinarySegmentUtils.java:1003)
at
org.apache.flink.cdc.common.data.binary.BinaryRecordData.getDecimal(BinaryRecordData.java:163)
at
org.apache.flink.cdc.common.data.RecordData.lambda$createFieldGetter$7b8ca8ef$1(RecordData.java:195)
```
### Steps to Reproduce
1. Create a PostgreSQL table with `numeric(0)` columns:
```sql
CREATE TABLE test_table (
id SERIAL PRIMARY KEY,
amount numeric(0), -- This causes the issue
name VARCHAR(100)
);
```
2. Insert data including NULL values:
```sql
INSERT INTO test_table (amount, name) VALUES (NULL, 'test');
INSERT INTO test_table (amount, name) VALUES (123, 'test2');
```
3. Configure Flink CDC pipeline:
```yaml
source:
type: postgres
hostname: localhost
port: 5432
username: postgres
password: password
database-name: testdb
schema-name: public
table-name: test_table
debezium.decimal.handling.mode: string
```
4. Run the pipeline - it will crash with IndexOutOfBoundsException
### Expected Result
- Pipeline should successfully process all rows including those with NULL
`numeric(0)` values
- No exceptions should be thrown during data processing
### Actual Result
- Pipeline crashes with `IndexOutOfBoundsException`
- Processing stops completely, preventing any data from being synchronized
### Root Cause Analysis
1. **Type Mapping Issue**: PostgreSQL `numeric(0)` fields are incorrectly
mapped to `DECIMAL` with maximum precision in `PostgresTypeUtils.java`
2. **Binary Serialization Problem**: High-precision DECIMAL types create
complex binary representations that are prone to corruption
3. **Missing Validation**: `BinarySegmentUtils.readDecimalData()` lacks bounds
checking for invalid binary data, allowing negative memory offsets
### Impact Assessment
- **Severity**: Major - Complete pipeline failure
- **Scope**: Affects any PostgreSQL database using `numeric(0)` columns
- **Workaround**: None available (users must modify table schema)
### Proposed Solution
1. Map PostgreSQL `numeric(0)` fields to `BIGINT` instead of `DECIMAL` to avoid
complex binary serialization
2. Add defensive bounds checking in `BinarySegmentUtils.readDecimalData()`
3. Handle edge cases gracefully by returning zero values instead of crashing
### Additional Context
- This issue commonly occurs in PostgreSQL databases where `numeric(0)` is used
to store whole numbers without decimal places
- The problem is exacerbated when fields are nullable, as NULL values create
invalid binary offset calculations
- Different `debezium.decimal.handling.mode` settings (string, double, precise)
all exhibit the same issue
### Test Case Requirements
- Unit tests for `PostgresTypeUtils` covering `numeric(0)` mapping
- Unit tests for `BinarySegmentUtils` defensive bounds checking
- Integration test with actual PostgreSQL table containing `numeric(0)` fields
### Documentation Impact
- Update connector documentation to clarify `numeric(0)` field handling
- Add troubleshooting section for IndexOutOfBoundsException issues
was:
### Problem Statement
Flink CDC fails with `IndexOutOfBoundsException` when processing PostgreSQL
tables containing `numeric(0)` fields, particularly when these fields have NULL
values. This causes complete pipeline failure during binary decimal data
deserialization.
### Error Details
```
Caused by: java.lang.IndexOutOfBoundsException: pos: -2130706416, length: 48,
index: -2130706432, offset: 0
at org.apache.flink.core.memory.MemorySegment.get(MemorySegment.java:467)
at
org.apache.flink.cdc.common.data.binary.BinarySegmentUtils.copyToBytes(BinarySegmentUtils.java:131)
at
org.apache.flink.cdc.common.data.binary.BinarySegmentUtils.readDecimalData(BinarySegmentUtils.java:1003)
at
org.apache.flink.cdc.common.data.binary.BinaryRecordData.getDecimal(BinaryRecordData.java:163)
at
org.apache.flink.cdc.common.data.RecordData.lambda$createFieldGetter$7b8ca8ef$1(RecordData.java:195)
```
### Steps to Reproduce
1. Create a PostgreSQL table with `numeric(0)` columns:
```sql
CREATE TABLE test_table (
id SERIAL PRIMARY KEY,
amount numeric(0), -- This causes the issue
name VARCHAR(100)
);
```
2. Insert data including NULL values:
```sql
INSERT INTO test_table (amount, name) VALUES (NULL, 'test');
INSERT INTO test_table (amount, name) VALUES (123, 'test2');
```
3. Configure Flink CDC pipeline:
```yaml
source:
type: postgres
hostname: localhost
port: 5432
username: postgres
password: password
database-name: testdb
schema-name: public
table-name: test_table
debezium.decimal.handling.mode: string
```
4. Run the pipeline - it will crash with IndexOutOfBoundsException
### Expected Result
- Pipeline should successfully process all rows including those with NULL
`numeric(0)` values
- No exceptions should be thrown during data processing
### Actual Result
- Pipeline crashes with `IndexOutOfBoundsException`
- Processing stops completely, preventing any data from being synchronized
### Root Cause Analysis
1. **Type Mapping Issue**: PostgreSQL `numeric(0)` fields are incorrectly
mapped to `DECIMAL` with maximum precision in `PostgresTypeUtils.java`
2. **Binary Serialization Problem**: High-precision DECIMAL types create
complex binary representations that are prone to corruption
3. **Missing Validation**: `BinarySegmentUtils.readDecimalData()` lacks bounds
checking for invalid binary data, allowing negative memory offsets
### Impact Assessment
- **Severity**: Major - Complete pipeline failure
- **Scope**: Affects any PostgreSQL database using `numeric(0)` columns
- **Workaround**: None available (users must modify table schema)
### Proposed Solution
1. Map PostgreSQL `numeric(0)` fields to `BIGINT` instead of `DECIMAL` to avoid
complex binary serialization
2. Add defensive bounds checking in `BinarySegmentUtils.readDecimalData()`
3. Handle edge cases gracefully by returning zero values instead of crashing
### Additional Context
- This issue commonly occurs in PostgreSQL databases where `numeric(0)` is used
to store whole numbers without decimal places
- The problem is exacerbated when fields are nullable, as NULL values create
invalid binary offset calculations
- Different `debezium.decimal.handling.mode` settings (string, double, precise)
all exhibit the same issue
### Test Case Requirements
- Unit tests for `PostgresTypeUtils` covering `numeric(0)` mapping
- Unit tests for `BinarySegmentUtils` defensive bounds checking
- Integration test with actual PostgreSQL table containing `numeric(0)` fields
### Documentation Impact
- Update connector documentation to clarify `numeric(0)` field handling
- Add troubleshooting section for IndexOutOfBoundsException issues
> IndexOutOfBoundsException when processing PostgreSQL tables with numeric(0)
> fields in Flink CDC
> -----------------------------------------------------------------------------------------------
>
> Key: FLINK-38196
> URL: https://issues.apache.org/jira/browse/FLINK-38196
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.4.0
> Environment: - **Database**: PostgreSQL with tables containing
> `numeric(0)` columns
> - **Connector**: PostgreSQL CDC Connector
> - **Configuration**: Any `debezium.decimal.handling.mode` setting
> Reporter: tchivs
> Priority: Major
>
> ### Problem Statement
> Flink CDC fails with `IndexOutOfBoundsException` when processing PostgreSQL
> tables containing `numeric(0)` fields, particularly when these fields have
> NULL values. This causes complete pipeline failure during binary decimal data
> deserialization.
> ### Error Details
> ```
> Caused by: java.lang.IndexOutOfBoundsException: pos: -2130706416, length: 48,
> index: -2130706432, offset: 0
> at org.apache.flink.core.memory.MemorySegment.get(MemorySegment.java:467)
> at
> org.apache.flink.cdc.common.data.binary.BinarySegmentUtils.copyToBytes(BinarySegmentUtils.java:131)
> at
> org.apache.flink.cdc.common.data.binary.BinarySegmentUtils.readDecimalData(BinarySegmentUtils.java:1003)
> at
> org.apache.flink.cdc.common.data.binary.BinaryRecordData.getDecimal(BinaryRecordData.java:163)
> at
> org.apache.flink.cdc.common.data.RecordData.lambda$createFieldGetter$7b8ca8ef$1(RecordData.java:195)
> ```
> ### Steps to Reproduce
> 1. Create a PostgreSQL table with `numeric(0)` columns:
> ```sql
> CREATE TABLE test_table (
> id SERIAL PRIMARY KEY,
> amount numeric(0), -- This causes the issue
> name VARCHAR(100)
> );
> ```
> 2. Insert data including NULL values:
> ```sql
> INSERT INTO test_table (amount, name) VALUES (NULL, 'test');
> INSERT INTO test_table (amount, name) VALUES (123, 'test2');
> ```
> 3. Configure Flink CDC pipeline:
> ```yaml
> source:
> type: postgres
> hostname: localhost
> port: 5432
> username: postgres
> password: password
> database-name: testdb
> schema-name: public
> table-name: test_table
> debezium.decimal.handling.mode: string
> ```
> 4. Run the pipeline - it will crash with IndexOutOfBoundsException
> ### Expected Result
> - Pipeline should successfully process all rows including those with NULL
> `numeric(0)` values
> - No exceptions should be thrown during data processing
> ### Actual Result
> - Pipeline crashes with `IndexOutOfBoundsException`
> - Processing stops completely, preventing any data from being synchronized
> ### Root Cause Analysis
> 1. **Type Mapping Issue**: PostgreSQL `numeric(0)` fields are incorrectly
> mapped to `DECIMAL` with maximum precision in `PostgresTypeUtils.java`
> 2. **Binary Serialization Problem**: High-precision DECIMAL types create
> complex binary representations that are prone to corruption
> 3. **Missing Validation**: `BinarySegmentUtils.readDecimalData()` lacks
> bounds checking for invalid binary data, allowing negative memory offsets
> ### Impact Assessment
> - **Severity**: Major - Complete pipeline failure
> - **Scope**: Affects any PostgreSQL database using `numeric(0)` columns
> - **Workaround**: None available (users must modify table schema)
> ### Proposed Solution
> 1. Map PostgreSQL `numeric(0)` fields to `BIGINT` instead of `DECIMAL` to
> avoid complex binary serialization
> 2. Add defensive bounds checking in `BinarySegmentUtils.readDecimalData()`
> 3. Handle edge cases gracefully by returning zero values instead of crashing
> ### Additional Context
> - This issue commonly occurs in PostgreSQL databases where `numeric(0)` is
> used to store whole numbers without decimal places
> - The problem is exacerbated when fields are nullable, as NULL values create
> invalid binary offset calculations
> - Different `debezium.decimal.handling.mode` settings (string, double,
> precise) all exhibit the same issue
> ### Test Case Requirements
> - Unit tests for `PostgresTypeUtils` covering `numeric(0)` mapping
> - Unit tests for `BinarySegmentUtils` defensive bounds checking
> - Integration test with actual PostgreSQL table containing `numeric(0)` fields
> ### Documentation Impact
> - Update connector documentation to clarify `numeric(0)` field handling
> - Add troubleshooting section for IndexOutOfBoundsException issues
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
