yihua opened a new pull request, #19839:
URL: https://github.com/apache/hudi/pull/19839

   ### Describe the issue this Pull Request addresses
   
   issue: #19838
   
   An upsert whose ordering field value is null is not routed to the error 
table: the record is written through, or, when it updates a stored record, the 
write fails in the record merger with a NullPointerException in 
`BufferedRecordMergerFactory.shouldKeepNewerRecord`. No streamer-level test 
reaches this path. Merge after #20049, which adds the record-creation guard and 
its unit tests.
   
   ### Summary and Changelog
   
   Covers the case in the existing error table write flow functional test 
rather than in a new test with its own streamer syncs, since each sync is 
expensive. The bad records the test injects become a case table 
(`BadRecordCase`): each case is one column mutation of valid generated records 
plus, per write operation, whether the record must go to the error table or the 
base table. A null ordering value must be quarantined on an upsert, which 
combines records by it, and written through on insert and bulk insert, which do 
not read it. Covering another failure mode is one more enum entry.
   
   - The test now applies its write-operation parameter; it was never set on 
the streamer config, so every parameter ran as an upsert.
   - It checks records rather than counts: the base table's record key to rider 
mapping, and the records both handed to and committed by the error table 
writer, each with the error reason of its case (the committed count for the 
unified write).
   - An upsert parameter runs a second sync that updates stored records: a 
valid update must be applied, every bad record must reach the error table, and 
a null ordering value on a stored record must leave it unchanged. Without the 
guard this is where the merger fails. All checks run after the last sync.
   - The unit test this PR first added is removed, since #20049 adds an 
equivalent, enabled one.
   
   The same test change was verified on a downstream 1.x branch: without the 
guard the two upsert parameters fail in the second sync with the 
NullPointerException above, and with the guard all eight parameters pass. It 
has not been run on this branch locally, so CI here is its first run, and its 
upsert parameters are expected to fail until #20049 merges.
   
   ### Impact
   
   None. Test-only.
   
   ### Risk Level
   
   none
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to