SEPURI-SAI-KRISHNA commented on issue #13223: URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6815857849
@nzw921rx Phase 1 is up as #13243. I am claiming these Step 2 rows and will build them on its API: - **Iceberg** (`IcebergSinkWriter`), both modes: delta writers and append writers - **JDBC / exactly-once (XA)** (`JdbcExactlyOnceSinkWriter`) - **JDBC / normal**, the parts #13242 does not cover: the UNIQUE-key routing fix and the destination-level E2E **Iceberg.** With identifier fields or upsert mode it uses the delta writers, which apply deletes by key, so it needs `KEYED` on a key column. Without them it uses the append writers, where I will verify whether `RANDOM` is safe. On dev `535d4237dd` (same Iceberg and routing code as now) I measured the keyed case with real Zeta jobs (Hadoop catalog, `iceberg.table.primary-keys = "id"`, 1000 keys and 3500 CDC rows, 500 rows expected): replica 1 was correct, replica 2 left 296 deleted ids in the table and 183 rows with an old value. Evidence: `IcebergSinkCDCIT` back at replica 2 plus a case for repeated changes of one key. #12881 changes the delta writers; this row only adds the declaration and the routing key, so I will rebase on it. **JDBC / exactly-once (XA).** Replicas already get separate XIDs (job id, the writer index from `SinkContextProxy`, checkpoint id), and `recoverAndRollback` only rolls back the writer's own index. The writer has no `primaryKey()`, so I will build it after #13242 and #13109 and verify it with the existing exactly-once E2E configs at replica 2, including a restore with the same layout. **JDBC / normal.** I offered this row on 11-11; @goutamadwant leads it with #13242 (`byte[]` hash and key refresh). The gap #13242 does not cover: a table with only a UNIQUE key and no `primary_keys` set. `JdbcSinkFactory` (lines 121-144 on dev `9cd7defe84`) then sets `primary_keys` from that UNIQUE key, so the sink upserts by it, but `JdbcSink.getPrimaryKeyIndex` (line 161) reads only the schema primary key, so the writer has no routing key. On dev `535d4237dd` such a table was routed at random: with replica 2 on DuckDB the job failed with `Duplicate key "id: 1" violates unique constraint`, while replica 1 was correct. With #13243 it is rejected instead. I will fix it (route by the key the sink writes by) and add this row's E2E (successive updates, delete then reinsert, repeated upserts, replicas 1, 2 and 4) in separate PRs on top of #13242. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
