yihua opened a new issue, #19838:
URL: https://github.com/apache/hudi/issues/19838
**Bug Description**
When Hudi Streamer runs with an error table enabled and an ordering
(precombine) field configured, a record whose ordering field value is null is
not routed to the error table as a `RECORD_CREATION` failure. It is accepted
into the write with an unvalidated ordering value.
`HoodieStreamerUtils.createHoodieRecords` wraps record construction in a
try/catch that turns a failure into an `ErrorEvent` with
`ErrorEvent.ErrorReason.RECORD_CREATION`, so anything that throws there is
quarantined and ingestion continues. A null ordering value does not throw there
on either of the two record-creation paths:
- **File-group-reader path** (`requiresPayload == false`, the default since
`hoodie.write.merge.handle.class` defaults to
`FileGroupReaderBasedMergeHandle`): `HoodieRecordUtils.createHoodieRecord`
builds a `HoodieAvroIndexedRecord` that stores the ordering value without
validating it.
- **Payload path** (`requiresPayload == true`): `BaseAvroPayload` does have
a null-ordering guard, but it is unreachable, because
`HoodieRecordUtils.loadPayload(String, GenericRecord, Comparable)` opens with
```java
if (orderingValue == null) {
return loadPayload(recordPayloadClass, record);
}
```
which routes to the record-only overload, so the `(GenericRecord,
Comparable)` constructor carrying the throw is never invoked.
The null is therefore only forced later, during the index/write stage, which
sits outside the error table's catch. A bare `compareTo` on the ordering value
there (for example `BufferedRecordMergerFactory.shouldKeepNewerRecord`) fails
the write instead of parking the offending record.
**Expected behavior:** a record that cannot supply a required ordering value
is a record-creation failure and belongs in the error table, so ingestion
survives one bad record. That is the contract the error table's
`RECORD_CREATION` reason already implies, and it is what the 0.x line does,
where the same class of failure throws inside the record-creation catch and is
quarantined.
**Environment**
- Hudi version: master (1.3.0-SNAPSHOT)
- Spark version: 3.5
- Running on Docker? no
**Logs and Stack Trace**
A test added in the companion PR pins the intended behavior. Against master
today it fails on both record-creation paths with the record flowing through
unquarantined rather than landing in the error table:
```
expected: <[]> but was: <[HoodieRecord{key=HoodieKey(recordKey=key1,
partitionPath=path1), currentLocation='null', newLocation='null'}]>
```
With a reject-on-null added at record creation, both paths quarantine and
the test passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
