rangareddy opened a new pull request, #19463:
URL: https://github.com/apache/hudi/pull/19463
### Describe the issue this Pull Request addresses
Relates to #14673 (HUDI-1286), which reports a MOR `_rt` query failing with:
```
org.apache.hudi.exception.HoodieException: Error ordering fields for storage
read.
#fieldNames: 4, #fieldPositions: 5
at HoodieRealtimeRecordReaderUtils.orderFields
```
This PR does **not** fix that failure — the projection lists still diverge,
and why they diverge under
`CombineHiveInputFormat` is still open. It makes the exception usable,
because as written it can hide the
very thing you need to see.
`orderFields` maps Hive's `hive.io.file.readcolumn.names` and
`hive.io.file.readcolumn.ids` onto an ordered
projection list. It de-duplicates both, because Hive tolerates duplicate
names but not duplicate ids, then
compares the two sizes. The comparison uses the **de-duplicated** name count
while the message reports the
**raw** one, so a mismatch caused by duplicate names reports two equal
numbers:
```
names "rider,driver,fare,fare", ids "0,1,2,3"
-> Error ordering fields for storage read. #fieldNames: 4,
#fieldPositions: 4
```
The message also omits the two lists themselves, which come from the query
engine and are the only way to
work out why they disagree.
### Summary and Changelog
- `HoodieRealtimeRecordReaderUtils#orderFields` reports the de-duplicated
counts that were actually compared,
and includes the read column names and ids in the exception.
- Adds the first unit coverage for this method
(`TestHoodieRealtimeRecordReaderUtils`): ordering by Hive
position, the empty case, the duplicate-name de-duplication it
deliberately performs, and the mismatch
message.
- Reproduces the reported failure on the real reader path
(`TestHoodieRealtimeRecordReader#testReaderFailsClearlyWhenHiveProjectionListsDiverge`):
a MOR table with a
parquet base file and a log block, then a `HoodieRealtimeRecordReader`
constructed with one more read-column
id than there are names. That test walks the cause chain, because the
failure is wrapped twice on the way
out — which is why the original report only shows the useful line under
`Caused by`.
### Verification
The reader test reproduces the reported exception, and both tests are red
with the production change reverted
and green with it. What the reader actually reports, same table and same
conf:
```
before Exception when constructing record reader
| Could not create HoodieRealtimeRecordReader on path
.../fileid0_1-0-1_100.parquet
| Error ordering fields for storage read. #fieldNames: 11,
#fieldPositions: 12
after Exception when constructing record reader
| Could not create HoodieRealtimeRecordReader on path
.../fileid0_1-0-1_100.parquet
| Error ordering fields for storage read. #fieldNames: 11,
#fieldPositions: 12,
read column names:
[_hoodie_commit_time,_hoodie_commit_seqno,_hoodie_record_key,
_hoodie_partition_path,_hoodie_file_name,field1,field2,name,favorite_number,
favorite_color,favorite_movie], read column ids:
[0,1,2,3,4,5,6,7,8,9,10,11]
```
`mvn test -pl hudi-hadoop-mr
-Dtest='TestHoodieRealtimeRecordReader,TestHoodieRealtimeRecordReaderUtils'`
runs 40 tests green, with checkstyle and `apache-rat:check` clean.
Note that this reader case shows equal-looking value for the counts either
way; the duplicate-name case where
the old message prints two *equal* numbers is covered by the unit test,
since Hive de-duplication is easier to
drive directly than through the reader.
### Impact
Exception text and new tests only. The method's behaviour for valid input is
unchanged, and the same inputs
still throw — the message just describes them correctly.
One thing noticed while adding coverage, not changed here: the
`partitioningFields` parameter is unused, it
appears only in the signature. Given that a name/id mismatch is exactly the
kind of thing partition columns
would cause, that looks like the remains of an intent that was never
implemented, and may be a useful thread
for whoever picks up HUDI-1286. I have left it alone rather than guess at
the intended semantics.
Also worth flagging for whoever triages HUDI-1286: it carries `Fix version
1.1.0`, but the failure path is
unchanged on master and #14673 is still open, so that field should not be
read as "already fixed".
### Risk Level
none — exception message and tests only.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
- [x] CI passes on my PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
