yraivo commented on issue #69303:
URL: https://github.com/apache/doris/issues/69303#issuecomment-7043322851
Hi, I reproduced this issue at the `fs_brokers/cdc_client` component level
against MySQL `8.0.46` (`ROW` binlog + `GTID` + `general_log` enabled) across
`4.1.5-release` (`917b20be8387`) and `master` (`5bc6ed62f9b0`), using
`CdcClientReadHarness` and `CdcClientWriteHarness` (`MockDorisServer`).
### Root Cause in `MySqlSnapshotSplitReadTask`
In `flink-connector-mysql-cdc` (`3.6.0-1.20`),
`MySqlSnapshotSplitReadTask#createDataEventsForTable` has two coupled
assumptions that break when MySQL omits `INVISIBLE` columns from `SELECT *`:
1. `StatementUtils.buildSplitScanQuery(...)` emits `SELECT * FROM ...` even
though Debezium's `Table` model (parsed from `SHOW CREATE TABLE`) includes
`INVISIBLE` columns.
2. In the `ResultSet` read loop, `new
Object[columnArray.getGreatestColumnPosition()]` under-allocates when an
`INVISIBLE` PK is at the end of the table (`Data row is smaller than a column
index`), while `Column actualColumn = table.columns().get(i)` indexes into the
full schema by `ResultSet` index `i` instead of `columnArray.getColumns()[i]`,
shifting subsequent column decoders when an `INVISIBLE` column precedes visible
columns (e.g., leading GIPK `my_row_id` or middle `INVISIBLE` column ->
`Unexpected format for DATE column`).
<details>
<summary>Captured source metadata, SHOW CREATE TABLE & MySQL general_log
queries</summary>
In our test runs (`@@global.show_gipk_in_create_table_and_information_schema
= 1`, `@@session.show_gipk_in_create_table_and_information_schema = 1`; for
Case C, `@@session.sql_generate_invisible_primary_key = 1` on the DDL
connection and `@@global.sql_generate_invisible_primary_key = 0`), `SHOW CREATE
TABLE` and `INFORMATION_SCHEMA.COLUMNS` expose the `INVISIBLE` columns and
`EXTRA` flags, JDBC `DatabaseMetaData#getPrimaryKeys` returns `["id"]` and
`["my_row_id"]`, while `SELECT *` omits them:
- **Case B (Trailing explicit `INVISIBLE` PK — matches the reported
symptom)**
```sql
CREATE TABLE `t_trail_inv` (
`c1` varchar(32) DEFAULT NULL,
`c2` int DEFAULT NULL,
`c3` datetime DEFAULT NULL,
`id` bigint NOT NULL /*!80023 INVISIBLE */,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
```
- `INFORMATION_SCHEMA.COLUMNS`: `1: c1` | `2: c2` | `3: c3` | `4: id (PRI,
INVISIBLE)` (JDBC `getPrimaryKeys`: `["id"]`)
- MySQL `general_log`:
```sql
SELECT * FROM `inv_db_910004`.`t_trail_inv`
```
- First exception: `org.apache.kafka.connect.errors.ConnectException: Data
row is smaller than a column index, internal schema representation is probably
out of sync with real database schema`
- **Case C (Leading MySQL GIPK `my_row_id`)**
```sql
CREATE TABLE `t_gipk` (
`my_row_id` bigint unsigned NOT NULL AUTO_INCREMENT /*!80023 INVISIBLE
*/,
`ts` datetime NOT NULL,
`note` varchar(32) DEFAULT NULL,
PRIMARY KEY (`my_row_id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4
COLLATE=utf8mb4_0900_ai_ci
```
- `INFORMATION_SCHEMA.COLUMNS`: `1: my_row_id (PRI, auto_increment
INVISIBLE)` | `2: ts` | `3: note` (JDBC `getPrimaryKeys`: `["my_row_id"]`)
- MySQL `general_log`:
```sql
SELECT * FROM `inv_db_910003`.`t_gipk`
```
- First exception: `java.lang.RuntimeException: Unexpected format for DATE
column: g1`
</details>
### Component-Level Verification & Next Step
Replacing the `SELECT *` prefix with an explicit backtick-quoted column list
from `table.columns()` and reading via `columnArray.getColumns()[i]` (sizing
`row` to `Math.max(table.columns().size(),
columnArray.getGreatestColumnPosition())`) was tested at the `cdc_client`
component level:
On MySQL 8.0.46, the visible-key control passed and the four
invisible-column cases failed on `4.1.5-release` (`917b20be8387`), `master`
(`5bc6ed62f9b0`), and #69948 (`fb16c6400d88`). With the candidate fix on master
and on top of #69948, all five cases passed. For the trailing invisible key, we
verified five snapshot splits and received all 14 expected IDs exactly once at
the mock sink during that run. In a separate trailing-key case we checked
insert, update, and delete events at the CDC reader. The leading GIPK case
covered a snapshot and a subsequent insert. (Composite invisible keys,
concurrent writes during snapshot, and full FE/BE job restart recovery have not
been tested yet.)
Since #69948 also carries this Flink CDC class, where would you prefer the
fix to land?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]