SEPURI-SAI-KRISHNA opened a new pull request, #13161:
URL: https://github.com/apache/seatunnel/pull/13161

   Yes, glad to contribute to Apache SeaTunnel.
   
   ### Purpose of this pull request
   
   Closes #13160.
   
   In CDC mode the Databend sink could silently leave a row with old values, or 
remove it, while the job succeeded. Two causes in the same merge, both needed 
for "the latest event of a key is applied":
   
   - **A matched INSERT was ignored.** The merge now has `WHEN MATCHED AND 
b.action IN ('insert', 'update') THEN UPDATE *`, in both copies 
(`DatabendSinkWriter` and `DatabendSinkAggregatedCommitter`).
   - **Events of one key could tie on `add_time`.** The merge keeps the latest 
`add_time` per key. The writer passed it through `setTimestamp`, and 
databend-jdbc 0.3.7 formats that with `yyyy-MM-dd HH:mm:ss.SSS`, so a burst of 
events tied and an older one could win. `DatabendSinkWriter#nextAddTime` now 
gives each CDC row a strictly increasing value in microseconds, and passes it 
as a string. It follows the clock unless a writer exceeds a million events per 
second.
   
   I kept `add_time` rather than adding a sequence column. As I read the code, 
the sink keeps the raw table's name, so a job restored from a savepoint could 
insert into a raw table without the new column. I did not run that case.
   
   One limit, which is not new: ordering holds per writer. With parallelism 
above 1, events of one key can reach different writers. I did not measure that 
case.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, and it is in `incompatible-changes.md` (EN and ZH), with a line in the 
sink docs. An INSERT on an existing key now overwrites the row, and the latest 
event of a key from one writer always wins. No option or API changes.
   
   ### How was this patch tested?
   
   **Unit tests** (JDK 11 and 17): `DatabendSinkWriterTest` asserts the new 
clause in the writer and committer SQL, and that `nextAddTime` increases 
strictly while staying close to the clock. Connector tests: 20 run, 0 failures. 
On `dev` the three SQL assertions fail.
   
   **Real jobs on Zeta** against `datafuselabs/databend:nightly` (v1.2.960), 
FakeSource to Databend. Local disk storage has no presigned upload, so these 
runs add `presigned_url_disabled=true` to the URL, and the IT uses MinIO as 
before.
   
   | Events | `dev` | Clause only | This PR |
   |---|---|---|---|
   | `(5, old)` then INSERT `(5, new)` | `old` | `new` | `new` |
   | `(6, old)` then dropped DELETE and INSERT `(6, new)` | `old` | `new` | 
`new` |
   | INSERT, UPDATE, UPDATE of 200 keys in one merge | `v1` 147, `v2` 22, `v3` 
31 | not run | `v3` 200 |
   | 300 keys, DELETE then INSERT in one merge | 117 and 168 rows left in two 
runs | 75 rows left | 300 rows, all `new` |
   | INSERT then UPDATE, INSERT then DELETE, new INSERT, UPDATE of a new key, 
`batch_size = 1` | correct | correct | correct |
   
   The clause alone does not fix the DELETE then INSERT case, which is why both 
changes are here. With this PR all 600 raw rows of those two bursts have a 
distinct `add_time`, against 221 of 600 on `dev` in one run. These runs skip 
presigned upload, but the driver formats values before that step, so the path 
does not change the result; the IT below covers the S3 path.
   
   The clause was also run directly on a Databend v1.2.881 server for matched 
and not matched keys with insert, update and delete: only matched insert 
changes.
   
   **`DatabendCDCSinkIT`** gains `testInsertOverwritesAnExistingKey` and 
`testLatestEventOfAKeyWinsInOneMerge`. In [fork run 
41811368734](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/41811368734)
 on this head, the class ran 18 tests with 0 failures on JDK 11 and on JDK 17: 
3 tests on each of Zeta, Flink 1.13, 1.18 and 1.20, and Spark 2.4.6 and 3.3.0, 
with the upload going through MinIO. They skip Flink 1.15 like the existing 
test. I did not run these two ITs against `dev` code; the `dev` results above 
come from the real jobs.
   
   Also run: `spotless:check`.
   
   ### Check list
   
   * [x] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
 (no new jar)
   * [x] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [x] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused by this PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to