SEPURI-SAI-KRISHNA commented on issue #13221:
URL: https://github.com/apache/seatunnel/issues/13221#issuecomment-6788223497

   @nzw921rx Thanks, that makes sense: the sink owns its key and nothing is 
guessed. Before I change the code, here is what the check would hit on dev, so 
you can tell me how you want each case handled.
   
   1. Among the connectors, only `JdbcSinkWriter` implements `primaryKey()`, 
and only for a table whose schema has a primary key. So the check rejects 
`multi_table_sink_replica > 1` for every other sink. That includes sinks that 
only append rows (Console, the file sinks, Http and others), where replicas 
work today. The option is documented on 49 of the 99 sink pages. Should those 
sinks be rejected too, or should a writer be able to declare that it does not 
need key routing?
   
   2. Sinks that apply changes by key (Iceberg, Paimon, Doris, StarRocks, 
Elasticsearch and others) would need their own `primaryKey()` to keep using 
replicas. `mysql_cdc_to_iceberg_for_schema_change.conf` runs Iceberg with 
`multi_table_sink_replica = 2`, so that E2E would fail with only the check. 
Should I add `primaryKey()` to the Iceberg writer in the same PR, and to the 
other keyed sinks in follow-up PRs?
   
   3. In Jdbc, a table without a primary key, and every table with 
`is_exactly_once = true`, would be rejected. In a whole-database job, one such 
table fails the whole job.
   
   4. With declared keys as the only route, the two Jdbc gaps from the report 
apply to every sink that declares one: a `BYTES` key is hashed by identity, and 
the declared index is fixed when the writer is built, so it is stale after a 
schema change moves the key column. I would hash a `byte[]` by content in 
`MultiTableSinkWriter`, and read `primaryKey()` again after each schema change, 
with `JdbcSinkWriter` updating its index.
   
   A softer variant for cases 1 to 3: a table whose writer declares no key gets 
all its rows on one replica, with a WARN, instead of failing the job. That 
table loses its extra replicas, but its order is kept, and the other tables 
still use every replica. Would you accept that, or do you prefer the plain 
rejection?
   
   Either way I will name the table and the option in the message, and add the 
change to `incompatible-changes.md`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to