fatmanverse commented on PR #11487: URL: https://github.com/apache/seatunnel/pull/11487#issuecomment-6593703995
Thanks @DanielLeens. I pushed cece2bf8a3 and 6690968c78 for both issues. **Issue 1 (case-sensitive identities).** You were right: `db.public.Orders` in the checkpoint passed when only `db.public.orders` was configured. I did not switch the comparison on `SnapshotPhaseState.isTableIdCaseSensitive`, because that flag does not mean the same thing in every dialect. `PostgresDialect` returns `true` for case-sensitive, while `MySqlConnectionUtils.isTableIdCaseSensitive` returns `!"0".equals(lower_case_table_names)`, which is passed to Debezium's `MySqlDatabaseSchema(..., boolean tableIdCaseInsensitive)` and so means the opposite. The check now uses the identity the readers already use. A table counts as configured only when its `TableId` is a key of `CatalogTableUtils.convertTables(catalogTables)`, which is the table map the MySQL and PostgreSQL dialects build and look tables up in by exact `TableId` (`MySqlDialect` / `MySqlSchema` `tableMap.get`, `PostgresSchema` `tableMap.containsKey(idWithCatalog)`; `TableId.equals` is an exact string compare). The discovered ids have the same shape as those keys: `(db, null, table)` for MySQL and OceanBase, `(db, schema, table)` for PostgreSQL and OpenGauss. Because of this, the exact match only rejects checkpoints whose tables the readers could not resolve anyway: MySQL fails when the source is built (NPE in `MySqlSchema`), and PostgreSQL fails with `Can't obtain schema` from `PostgresSchema`. It does not reject a restore that would have worked, including MySQL with `lower_case_table_names` 1 or 2, where a mismatched name never produces a checkpoint. Tests: `testSnapshotOnlyRestoreRejectsTableThatDiffersInCase` checks that `db.public.Orders` vs `db.public.orders`, `db.b` vs `DB.B`, and `db.public.c` vs `db.c` are all rejected and named in the error. `testSnapshotOnlyRestoreAcceptsConfiguredTables` now expects exact MySQL and PostgreSQL ids, including mixed-case `db.public.Orders`. The MySQL-CDC and PostgreSQL-CDC docs (en/zh) now say the names must match exactly, including letter case. **Issue 2 (E2E window and cleanup).** `MysqlCDCSnapshotOnlyIT` now stops the job on every exit path. On Zeta it submits with a `JobIdGenerator` id and, in a `finally`, cancels the job if it is still running. A failed cancel is only logged, so it does not hide the assertion that failed. Only the Zeta container implements `executeJob(conf, jobId)` and `cancelJob`, so on Flink the job is submitted as before; that container is not shared and is torn down after the test (`TestCaseInvocationContextProvider` `AfterTestExecutionCallback`), which stops the job. I kept the test enabled on Flink because it is the only E2E that checks that a MySQL snapshot-only job ends by itself there. I did not add a split-level barrier: it would need a test-only hook in the source. The window is still probabilistic but wide (103 splits of one row, `batch_size = 1`), and a missed window now fails cleanly instead of leaving a job running. Verified on JDK 17 (Zeta and Flink): on the previous head, the new test fails because only `db.public.c` is reported (the case-folded names passed). On this head, `IncrementalSourceTest` passes, the cdc-base/mysql/oceanbase/postgres/opengauss UTs pass (142/59/10/93/7, 0 failures), and `MysqlCDCSnapshotOnlyIT` passes 5/5 with both commits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
