Doris-Breakwater commented on issue #69303: URL: https://github.com/apache/doris/issues/69303#issuecomment-6275403350
This is actionable as a MySQL CDC snapshot correctness issue. The supplied trace and reproduction are consistent with a projection/schema mismatch in the snapshot reader. The issue is currently open with no labels; suggested routing is to the Doris streaming-job/CDC maintainers, with an upstream Flink CDC follow-up for the query construction. I inspected Doris commit `917b20be83878e98f94e3ebf7efccc80346e3e50` and the corresponding dependency sources. This is source-level verification, not an independent end-to-end reproduction. The verified code path is: 1. Doris builds the target definition from JDBC column and primary-key metadata. `StreamingJobUtils` checks for a primary key and creates a UNIQUE KEY table, but this path has no invisible-column compatibility check. Successful target creation therefore does not establish snapshot support. See [StreamingJobUtils at the reported commit](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java). 2. Doris `MySqlSourceReader.getSnapshotSplitReader()` constructs Flink CDC's `SnapshotSplitReader`. The [CDC client POM](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fs_brokers/cdc_client/pom.xml) specifies Flink CDC `3.6.0-1.20`; that connector depends on Debezium `1.9.8.Final`. 3. In that [Flink CDC source artifact](https://repo.maven.apache.org/maven2/org/apache/flink/flink-connector-mysql-cdc/3.6.0-1.20/flink-connector-mysql-cdc-3.6.0-1.20-sources.jar), `MySqlSnapshotSplitReadTask.createDataEventsForTable()` calls `StatementUtils.buildSplitScanQuery()`. Its data-scan branch explicitly uses `"*"` as the projection. Schema discovery separately parses `SHOW CREATE TABLE`, with a `DESC` fallback. MySQL documents that invisible columns are available through metadata and explicit selection but omitted from wildcard results. See [MySQL invisible-column behavior](https://dev.mysql.com/doc/refman/8.4/en/invisible-columns.html). 4. Debezium `ColumnUtils.toArray()` maps returned column names to schema positions; the snapshot task allocates the row using the greatest position among those returned columns. With eight visible columns followed by an invisible key at position 9, it allocates eight slots. `TableSchemaBuilder.createKeyGenerator()` then requests zero-based index 8 and its bounds check throws exactly the reported exception. See `ColumnUtils` and `TableSchemaBuilder` in the [Debezium source artifact](https://repo.maven.apache.org/maven2/io/debezium/debezium-core/1.9.8.Final/debezium-core-1.9.8.Final-sources.jar). 5. Doris propagates reader failures; `StreamingInsertJob.onStreamTaskFail()` records the failure and sets the job to `PAUSED`. This explains the job-wide impact. See [StreamingInsertJob](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java). The broader exposure needs separate validation. The wildcard also omits invisible non-key columns. For an invisible column earlier in the schema, the row array can instead contain an unfilled slot; the reader additionally chooses conversion metadata using `table.columns().get(i)`, which can differ from the returned column after an omission. Thus the exact bounds exception is position-dependent. Generated `my_row_id` keys share the projection problem, but their precise failure signature and binlog correctness are not established by the supplied trace. [MySQL's GIPK documentation](https://dev.mysql.com/doc/refman/8.4/en/create-table-gipks.html) also describes a separate metadata-visibility setting, `show_gipk_in_create_table_and_information_schema`, relevant to discovery. Recommended next steps: - Build the snapshot projection from the complete source Debezium table schema, with identifiers quoted and columns in schema order. Preserve split predicates and snapshot/binlog watermark handling. Merely enlarging the row array would still leave the key/value unread. Review conversion metadata alignment alongside the projection. - Add coverage for invisible columns at the beginning, middle, and end; invisible keys and non-key columns; composite keys; and generated invisible primary keys. Verify actual values, subsequent INSERT/UPDATE/DELETE, concurrent snapshot changes, and restart/resume, not only that the job remains running. - If support is deferred, reject affected tables before target creation with an explicit table/column diagnostic. Skipping a table should be an explicit user choice. The existing report is sufficient to begin that work. To confirm the deployed path and GIPK variant, please attach the logged `using select statement` line, source `SHOW CREATE TABLE` and column ordinal/`EXTRA` metadata, plus the generated-key case's own first exception and metadata-visibility setting under the CDC account. If the bundled client differs from the release build, include its embedded dependency versions or checksum. Redact credentials and sensitive identifiers. The reported `include_tables` workaround is consistent with this path. Making the source column visible also addresses the wildcard omission, subject to the application compatibility concern already described. No fixed Doris release or verified binlog-only support is established by this investigation. Breakwater-GitHub-Analysis-Slot: slot_7420af732753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
