yraivo commented on issue #69303:
URL: https://github.com/apache/doris/issues/69303#issuecomment-7043322851

   Hi, I reproduced this issue at the `fs_brokers/cdc_client` component level 
against MySQL `8.0.46` (`ROW` binlog + `GTID` + `general_log` enabled) across 
`4.1.5-release` (`917b20be8387`) and `master` (`5bc6ed62f9b0`), using 
`CdcClientReadHarness` and `CdcClientWriteHarness` (`MockDorisServer`).
   
   ### Root Cause in `MySqlSnapshotSplitReadTask`
   
   In `flink-connector-mysql-cdc` (`3.6.0-1.20`), 
`MySqlSnapshotSplitReadTask#createDataEventsForTable` has two coupled 
assumptions that break when MySQL omits `INVISIBLE` columns from `SELECT *`:
   1. `StatementUtils.buildSplitScanQuery(...)` emits `SELECT * FROM ...` even 
though Debezium's `Table` model (parsed from `SHOW CREATE TABLE`) includes 
`INVISIBLE` columns.
   2. In the `ResultSet` read loop, `new 
Object[columnArray.getGreatestColumnPosition()]` under-allocates when an 
`INVISIBLE` PK is at the end of the table (`Data row is smaller than a column 
index`), while `Column actualColumn = table.columns().get(i)` indexes into the 
full schema by `ResultSet` index `i` instead of `columnArray.getColumns()[i]`, 
shifting subsequent column decoders when an `INVISIBLE` column precedes visible 
columns (e.g., leading GIPK `my_row_id` or middle `INVISIBLE` column -> 
`Unexpected format for DATE column`).
   
   <details>
   <summary>Captured source metadata, SHOW CREATE TABLE & MySQL general_log 
queries</summary>
   
   In our test runs (`@@global.show_gipk_in_create_table_and_information_schema 
= 1`, `@@session.show_gipk_in_create_table_and_information_schema = 1`; for 
Case C, `@@session.sql_generate_invisible_primary_key = 1` on the DDL 
connection and `@@global.sql_generate_invisible_primary_key = 0`), `SHOW CREATE 
TABLE` and `INFORMATION_SCHEMA.COLUMNS` expose the `INVISIBLE` columns and 
`EXTRA` flags, JDBC `DatabaseMetaData#getPrimaryKeys` returns `["id"]` and 
`["my_row_id"]`, while `SELECT *` omits them:
   
   - **Case B (Trailing explicit `INVISIBLE` PK — matches the reported 
symptom)**
     ```sql
     CREATE TABLE `t_trail_inv` (
       `c1` varchar(32) DEFAULT NULL,
       `c2` int DEFAULT NULL,
       `c3` datetime DEFAULT NULL,
       `id` bigint NOT NULL /*!80023 INVISIBLE */,
       PRIMARY KEY (`id`)
     ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
     ```
     - `INFORMATION_SCHEMA.COLUMNS`: `1: c1` | `2: c2` | `3: c3` | `4: id (PRI, 
INVISIBLE)` (JDBC `getPrimaryKeys`: `["id"]`)
     - MySQL `general_log`:
       ```sql
       SELECT * FROM `inv_db_910004`.`t_trail_inv`
       ```
     - First exception: `org.apache.kafka.connect.errors.ConnectException: Data 
row is smaller than a column index, internal schema representation is probably 
out of sync with real database schema`
   
   - **Case C (Leading MySQL GIPK `my_row_id`)**
     ```sql
     CREATE TABLE `t_gipk` (
       `my_row_id` bigint unsigned NOT NULL AUTO_INCREMENT /*!80023 INVISIBLE 
*/,
       `ts` datetime NOT NULL,
       `note` varchar(32) DEFAULT NULL,
       PRIMARY KEY (`my_row_id`)
     ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4 
COLLATE=utf8mb4_0900_ai_ci
     ```
     - `INFORMATION_SCHEMA.COLUMNS`: `1: my_row_id (PRI, auto_increment 
INVISIBLE)` | `2: ts` | `3: note` (JDBC `getPrimaryKeys`: `["my_row_id"]`)
     - MySQL `general_log`:
       ```sql
       SELECT * FROM `inv_db_910003`.`t_gipk`
       ```
     - First exception: `java.lang.RuntimeException: Unexpected format for DATE 
column: g1`
   
   </details>
   
   ### Component-Level Verification & Next Step
   
   Replacing the `SELECT *` prefix with an explicit backtick-quoted column list 
from `table.columns()` and reading via `columnArray.getColumns()[i]` (sizing 
`row` to `Math.max(table.columns().size(), 
columnArray.getGreatestColumnPosition())`) was tested at the `cdc_client` 
component level:
   
   On MySQL 8.0.46, the visible-key control passed and the four 
invisible-column cases failed on `4.1.5-release` (`917b20be8387`), `master` 
(`5bc6ed62f9b0`), and #69948 (`fb16c6400d88`). With the candidate fix on master 
and on top of #69948, all five cases passed. For the trailing invisible key, we 
verified five snapshot splits and received all 14 expected IDs exactly once at 
the mock sink during that run. In a separate trailing-key case we checked 
insert, update, and delete events at the CDC reader. The leading GIPK case 
covered a snapshot and a subsequent insert. (Composite invisible keys, 
concurrent writes during snapshot, and full FE/BE job restart recovery have not 
been tested yet.)
   
   Since #69948 also carries this Flink CDC class, where would you prefer the 
fix to land?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to