rangareddy commented on issue #14669: URL: https://github.com/apache/hudi/issues/14669#issuecomment-4902718922
This issue was reviewed as part of the JIRA-migrated backlog triage, and verified with a test. Findings: this is implemented by `DefaultHoodieRecordPayload` (hudi-common `org/apache/hudi/common/model/DefaultHoodieRecordPayload.java`). It honors the ordering field in both `preCombine` and `combineAndGetUpdateValue` and skips late-arriving/older records via `needUpdatingPersistedRecord`. The "1 or more configurable fields" requirement is supported: `ConfigUtils.getOrderingFields()` splits the precombine field on `,`, and `needUpdatingPersistedRecord` builds a composite ordering value with `OrderingValues.create(orderingFields, ...)`. Verified on Hudi 1.2.0 (Spark 3.4) with a COW table, `payload.class=DefaultHoodieRecordPayload` and a two-field precombine `ts1,ts2` (lexicographic ordering): - seed `v1 (100,1)` - upsert `v2 (100,5)` -> APPLIED (composite `(100,5) > (100,1)`) - upsert `v3 (100,3)` -> SKIPPED (`(100,3) < (100,5)`) - upsert `v4 (90,999)` -> SKIPPED (lower `ts1` dominates the larger `ts2` - genuine composite ordering) - upsert `v5 (200,0)` -> APPLIED (`(200,0) > (100,5)`) - final row = `v5 (200,0)` as expected. So both single-field and multi-field late-arriving skip work as requested. If you need behavior beyond this (e.g. a different tie-breaking or per-field ordering direction), please reopen with specifics. Closing as fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
