rmahindra123 opened a new pull request, #19660:
URL: https://github.com/apache/hudi/pull/19660
### Describe the issue this Pull Request addresses
The Debezium payloads order records with two different mechanisms.
`preCombine` (intra-batch dedup) compares `orderingVal` — the value of the
configured precombine field captured at payload construction — while
`combineAndGetUpdateValue` (incoming vs stored record) drops the payload
`Properties` and always compares the connector-hardcoded column (`_event_lsn`
for Postgres, `_event_seq` for MySQL) via `shouldPickCurrentRecord`: the 3-arg
`combineAndGetUpdateValue(current, schema, properties)` funnels into the
props-less template, so `Properties` never reaches the ordering decision.
The two agree only by convention (precombine set to the connector column).
Any table where precombine differs silently orders against storage by a column
the user never configured.
### Summary and Changelog
Debezium payloads now honor the configured ordering field(s) when merging
against storage, matching what `preCombine` already does within a batch.
- `AbstractDebeziumAvroPayload` keeps `OverwriteWithLatestAvroPayload` as
its parent and adds a `Properties`-aware `combineAndGetUpdateValue` template
(the 2-arg overload delegates with `CollectionUtils.emptyProps()`) which
dispatches the ordering decision:
- ordering fields absent → the connector-specific
`shouldPickCurrentRecord` (unchanged, original 3-arg signature);
- any configured ordering is used as-is, single or composite (multi-field,
element-wise: the first non-equal field decides, ties go to the incoming
record) via `needUpdatingPersistedRecord`, implemented in the abstract class
mirroring `DefaultHoodieRecordPayload`'s comparison (kept local rather than
reusing `OrderingValues` because a null persisted element after a tied element
must take the incoming record instead of throwing; keeping the parent unchanged
also avoids inheriting `Default`'s per-record-deserializing `isDeleted` and its
`DELETE_KEY` handling — `isDeleted`/`getInsertValue` behavior is byte-identical
to before this PR).
- One compatibility carve-out, owned by `MySqlDebeziumAvroPayload`: the
single legacy ordering field `_event_seq` (what v8 tables carry until the v9
upgrade rewrites it to `_event_bin_file,_event_pos`) routes to the connector's
segment-wise numeric seq parser — a plain `Comparable` compare of the encoded
`"file.pos"` string is lexicographic (`"9.1" > "10.1"`) and flips merge winners
for unpadded positions (caught by `TestPayloadDeprecationFlow`).
- `preCombine(oldValue, properties)` applies the same dispatch, so
intra-batch dedup (`hoodie.combine.before.upsert`) and the against-storage
merge order by the same column(s) — a configured non-connector field previously
crashed MySQL's seq parser on the first repeated key in a batch (NFE for
non-numerics, AIOOBE for equal plain values). The props-less `preCombine`
overrides (MySQL seq compare, delete natural-order) are untouched; Postgres
toast-column merging is unchanged and covered by a test through the
configured-ordering path.
- A null incoming ordering value on the configured path throws
`HoodieDebeziumAvroPayloadException` naming the field and record; a null
persisted value (e.g. bootstrapped rows) takes the incoming record; ties go to
the incoming record in every path, matching the legacy strict comparisons.
Configured values are trimmed. No code was copied from external sources.
### Impact
| Table config | Before | After |
|---|---|---|
| precombine = connector column | connector ordering | identical (numeric
seq compare preserved for MySQL via the carve-out; plain `Comparable` on the
LSN is the same ordering for Postgres) |
| precombine = other field | `preCombine` by config field,
`combineAndGetUpdateValue` by connector column | both by config field |
| composite ordering (e.g. `_event_bin_file,_event_pos`) | connector column
only | element-wise composite |
| no ordering props passed | connector ordering | identical |
No public API change; behavior-preserving for the conventional
configurations.
### Risk Level
low
Behavior-preserving for the common configuration (precombine = connector
column) and for callers that pass no ordering properties. Verification:
`TestPostgresDebeziumAvroPayload` 16/16 and `TestMySqlDebeziumAvroPayload`
31/31, including tests where the configured field contradicts the connector
ordering, tie/null-persisted/null-incoming cases, whitespace-trim
discriminators, mixed-type composite tests bound to
`MySqlDebeziumAvroPayload.ORDERING_FIELDS` with real binlog shapes,
unpadded-seq discriminators where lexicographic ordering inverts the winner,
and toast-column merging through the configured-ordering path.
`TestPayloadDeprecationFlow` covers the v8→v9 upgrade flow for every built-in
payload.
### Documentation Update
none — the payloads honor `hoodie.payload.ordering.field` / precombine as
already documented for `DefaultHoodieRecordPayload`.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
