SEPURI-SAI-KRISHNA opened a new pull request, #13268:
URL: https://github.com/apache/seatunnel/pull/13268
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Part of #13223, the `Iceberg` row. Stacked on #13243 (Phase 1), which adds
`replicaMode()`: until #13243 merges, this PR also shows its commits, and only
the two `[Iceberg]` commits belong here.
With #13243, a writer that declares no replica mode is rejected when
`multi_table_sink_replica` is above 1. `IcebergSinkWriter` declared none, so
#13243 set the Iceberg schema change E2E and the Iceberg docs to replica 1. On
dev the Iceberg writers got random routing instead: with
`iceberg.table.primary-keys = "id"` and replica 2, a table that should hold 500
rows ended with deleted rows back and old values.
Changes:
- `IcebergSinkWriter.replicaMode()`: `KEYED` when the table is written by a
delta writer, which applies deletes and upserts by key (the table has
identifier fields, or `iceberg.table.primary-keys` or
`iceberg.table.upsert-mode-enabled` is set); `RANDOM` when it is written by an
append writer, which only adds rows.
- `IcebergSinkWriter.primaryKey()`: the position in the current row type of
the first column that is one of those key fields.
`RowConverter.indexOfAnyField` finds it with the same name lookup that
`RowConverter.convert` uses (name mapping and `case_sensitive`), so routing and
writing agree on the column. Empty when no column of the row is a key field, so
such a `KEYED` table is rejected with several replicas.
- `IcebergWriterFactory.identifierFieldIds` and `writesDeltas`: the
identifier field logic of `createTaskWriter` moved into one place that both the
task writer selection and the routing use.
- `RecordWriter.writesDeltas()` and `routingKeyIndex()`, with defaults that
keep an unknown writer `KEYED` without a key, so it is rejected with several
replicas.
- E2E: `mysql_cdc_to_iceberg_for_schema_change.conf` back to
`multi_table_sink_replica = 2`, as on dev, and
`IcebergSinkParallelCommitIT#testReplicasKeepTheChangesOfOneKeyInOrder`.
- Docs (EN and ZH): the `multi_table_sink_replica` row of the Iceberg sink
page, and the impact line of the #13243 entry in `incompatible-changes.md`. As
asked in the review of #13243, the common pages describe the replica modes
without naming connectors, so Iceberg is not added there.
Notes for reviewers:
- The mode and key depend on the loaded table, so `replicaMode()` and
`primaryKey()` create the record writer the same way the first `write()` does.
`MultiTableSinkWriter` asks for them when it starts, so the table is now loaded
when the writer starts instead of at the first row, also with one replica.
- Routing by one key column keeps every change of one record on one writer,
also for a key with several columns.
- After a schema change, the key position is read again only with #13242,
which refreshes `primaryKey()` after the schema change barrier. `primaryKey()`
here is computed from the current row type and table each time, so it follows a
moved key column once #13242 is in. The DDL in `IcebergSinkCDCIT` adds,
modifies and drops columns after the key, so the key position does not move
there.
- `RANDOM` for the append writers: `UnpartitionedWriter` and
`PartitionedAppendWriter` append every record as a data row whatever its row
kind (only the delta writers read `RowKind`), so there is no delete path and no
order between changes to keep. The table gets the same rows whichever writer a
row reaches, as the CDC input row in the table below shows.
- #12881 changes the delta writers and `IcebergWriterFactory`; this PR only
moves the identifier field logic into a helper and adds the declaration. If
#12881 merges first, I will rebase this PR on it before marking it ready.
### Does this PR introduce _any_ user-facing change?
Yes. Compared with #13243, an Iceberg sink can run with
`multi_table_sink_replica` above 1 again: keyed tables route all changes of one
key to one writer, and append-only tables spread rows over the writers.
Compared with dev, keyed Iceberg tables with several replicas no longer apply
the changes of one key out of order. A keyed table whose rows have no key
column is rejected with several replicas. No option, default or state changes.
### How was this patch tested?
- `IcebergSinkWriterTest` (real `InMemoryCatalog` tables): identifier fields
route by that column even when the row orders its columns differently from the
table; `iceberg.table.primary-keys` replaces the identifier fields; an
append-only table is `RANDOM` without a key; a key column missing from the row
leaves `KEYED` without a key.
- I broke the change on purpose and a test failed each time:
`writesDeltas()` always false (3 tests fail), and the key position taken from
the table column order instead of the row (3 tests fail).
- `connector-iceberg` unit tests: 136 run, 0 failures (1 skipped,
unrelated). `IcebergFormatVersionTest` (24) covers the moved task writer
selection.
- Real Zeta jobs, FakeSource to Iceberg (Hadoop catalog), read back with a
second job. Keyed input: 1000 keys, 3500 CDC rows (insert, update, delete of
every second key), 500 rows expected. Append input: 1000 inserts.
| case | replicas | result |
| --- | --- | --- |
| `iceberg.table.primary-keys = "id"` | 1, 2, 4 | 500 rows, correct (1, 2, 4
queues) |
| no key option, table created by save mode with identifier field `id` | 2,
4 | 500 rows, correct |
| append only, no key | 1, 2, 4 | 1000 rows, none lost or duplicated |
| append only, no key, the keyed CDC input (insert, update, delete) | 1, 4 |
3500 rows each, the same rows sorted (every change is appended, as with one
replica) |
| `primary-keys = "id"`, writer forced to `RANDOM` (as on dev) | 2 | 942
rows, 314 deleted keys back, 194 old values, 128 keys twice |
- E2E on Zeta: `IcebergSinkCDCIT` (3 tests, the schema change ones at
replica 2 with 2 writer queues) and `IcebergSinkParallelCommitIT` (the new case
at replica 4 with 4 queues) pass. With the writer forced to `RANDOM` (as on
dev), the new case fails: the table has 139 rows instead of 50, with 43 deleted
keys back, 46 keys with more than one row and 58 rows with the old value.
`IcebergSinkCDCIT` and the other four parallel commit tests still pass in that
run, so only the new case catches the order of changes.
- `spotless:check`; `MarkdownTest`.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. (the #13243 entry)
* [ ] If you are contributing the connector code, please check that the
following files are updated: (not a new connector)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]