rangareddy opened a new issue, #19506:
URL: https://github.com/apache/hudi/issues/19506
## Describe the problem
`HoodieRealtimeRecordReaderUtils.orderFields` de-duplicates Hive's
read-column **names** and **ids** independently, then pairs them
**positionally**:
```java
Set<String> fieldOrdersSet = new
LinkedHashSet<>(Arrays.asList(fieldOrdersWithDups));
String[] fieldOrders = fieldOrdersSet.toArray(new String[0]);
...
Set<String> fieldNamesSet = new LinkedHashSet<>(fieldNames);
String[] fieldNamesArray = fieldNamesSet.toArray(new String[0]);
for (int ox = 0; ox < fieldOrders.length; ox++) {
orderedFieldMap.put(Integer.parseInt(fieldOrders[ox]),
fieldNamesArray[ox]);
}
```
That is only sound if duplicates fall at the same offsets on both sides, and
Hive guarantees they do not: `ColumnProjectionUtils.appendReadColumns`
**prepends** ids (`newConfStr = id + "," + old`) while `appendReadColumnNames`
**appends** names. So two accumulation rounds on one `JobConf` can produce
lists whose duplicate positions differ, and the counts still match — no
exception, silently wrong mapping:
```
READ_COLUMN_NAMES = _hoodie_commit_time,rider,driver,fare
READ_COLUMN_IDS = 2,3,0,1
orderFields -> [driver, fare, _hoodie_commit_time, rider]
```
Hive's own `getReadColumnIDs` documents the hazard: *"some code uses this
list to correlate with column names, and yet these lists may contain
duplicates, which this call will remove and the other won't."*
## Why this may look harmless today
`projectionFields` feeds `generateProjectionSchema`, and the downstream
consumers resolve by name, so the wrong order appears masked. That makes it a
latent trap rather than a live data bug as far as I can tell — but it is
unverified either way, and it sits in the method that produces the `Error
ordering fields for storage read` diagnostics.
## Suggested direction
Pair names and ids before de-duplicating, so a duplicate removes the pair
rather than shifting one side relative to the other. Any fix needs a test that
produces the prepend/append asymmetry above.
## Context
Raised out of review on #19463, which improves the diagnostics in this
method but deliberately does not change the pairing. Filed so the message
improvement is not later mistaken for a fix for this.
Credit to @voonhous for identifying the asymmetry and the Hive citation.
## Related
- #19463
- #14673 (HUDI-1286)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
