lokeshj1703 opened a new pull request, #20049:
URL: https://github.com/apache/hudi/pull/20049

   ### Describe the issue this Pull Request addresses
   
   closes #19838
   
   ### Summary and Changelog
   
   A record whose configured ordering field is null was accepted into the write 
with an unvalidated ordering value, and only failed later in the merger, 
outside the error table's catch, so one bad record killed the job instead of 
being quarantined. Record creation now rejects it, and a configured error table 
parks it as a `RECORD_CREATION` failure.
   
   The check lives in `OrderingValues.isMissing`, next to the `create` it 
mirrors, and the Spark row write path in `HoodieCreateRecordUtils` uses it for 
its per-field check too, so both write paths share one definition. It unwraps a 
multi-field ordering value rather than testing the ordering value for null: a 
single field resolves to the field value, so a null field gives a null ordering 
value, but several fields resolve to a non-null `ArrayComparable` that holds 
the nulls. Deletes are covered too, since a delete with a null ordering value 
is not a commit-time-ordering delete (that requires the default value) and 
would reach the same comparison.
   
   This includes the regression test from #19839, rebased onto current master 
and enabled, so #19839 can be closed once this merges. Its merge-handle 
parameter is dropped because it no longer selects between two record-creation 
paths now that `requiresPayload` is gone from this method. Added coverage: a 
null in one of several ordering fields, a delete carrying a null ordering 
value, and the no-error-table case.
   
   ### Impact
   
   Behavior change for Hudi Streamer upserts on a table with an ordering field 
configured, deletes included. Measured on each line rather than reasoned about:
   
   | | 0.x | 1.x before | 1.x after |
   | --- | --- | --- | --- |
   | record creation, no error table | throws `HoodieException` | accepted | 
throws `IllegalArgumentException` |
   | record creation, with error table | quarantined `RECORD_CREATION` | 
accepted | quarantined `RECORD_CREATION` |
   | outcome at merge | never reached | **live row silently deleted** | never 
reached |
   | delete exempt? | no | n/a | no |
   
   The 1.x "before" row is the reason for the change. A delete carrying a null 
ordering value removes a live row with a higher ordering value, unconditionally 
and with nothing logged, on both the payload and file-group-reader paths. 0.x 
has always rejected a null ordering value at record creation, deletes included, 
because `HoodieRecordUtils.loadPayload` there has no null short-circuit and 
`BaseAvroPayload` throws. 1.x added that short-circuit, which made the guard 
unreachable. This restores the 0.x contract.
   
   Verified by running: 0.x on 0.14.1 (four combinations of delete/non-delete 
and with/without error table), and the "before" row on both `master` 
(1.3.0-SNAPSHOT) and the 1.1 line, which agree.
   
   MOR note: a delete whose ordering field is null is now quarantined rather 
than applied. That is the intended effect, since applying it is the data loss 
above.
   
   The Spark record path is untouched and still coerces a null ordering value 
to the default.
   
   ### Risk Level
   
   low
   
   ### Documentation Update
   
   None.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to