ravjot28 opened a new pull request, #20596:
URL: https://github.com/apache/hudi/pull/20596
### Describe the issue this Pull Request addresses
Closes #20586
The record reader of the Hadoop Avro input format reuses the record and its
nested record, array, map and fixed
instances for all rows of a file. `AvroDFSSource` and `HiveIncrPullSource`
emitted that instance, and the streamer
keeps source records by reference until an upsert deduplicates them, so rows
took the fixed, array, map and nested
values of the last row read, or the whole last row when the source and
target schemas are equal. The write succeeds
with wrong data.
### Summary and Changelog
Every record the two sources emit is an independent copy, so each row keeps
its own values.
- Add `AvroSource#toIndependentRecords`, which deep-copies each record read
through `AvroKeyInputFormat`
(`GenericData.deepCopy` with the record's schema).
- `AvroDFSSource` and `HiveIncrPullSource` use it.
- New tests: `TestAvroDFSSource#testFetchedRecordsKeepTheirOwnValues` (the
source alone) and
`TestAvroDFSSource#testStreamerUpsertKeepsFixedAndNestedValues` (a
streamer upsert of 20 rows with a fixed decimal, a
nested record and an array column).
### Impact
No API or configuration change. The two sources make one deep copy per
record read; other sources are unchanged.
Spark's `newAPIHadoopFile` documentation asks callers to copy the objects of
a Hadoop record reader before caching or
aggregating them; the copy is made where the reader's object leaves the
source.
Tables written by 1.1.0 or later through these sources with an upsert may
hold wrong values in fixed, array, map and
nested record columns (or in every column when the source and target schemas
are equal) and need to be rewritten from
the source data.
**Backport:** please include this fix in the next 1.2.x patch release. The
commit cherry-picks cleanly onto
`release-1.2.1`.
### Risk Level
low. A copy per record read, in two sources only.
Tests, run on master `d5230b69e22` with JDK 11 and passing checkstyle and
scalastyle checks, `mvn test -pl hudi-utilities -am
-Dtest=TestAvroDFSSource,TestHiveIncrPullSource
-DwildcardSuites=skipScalaTests`:
- Without the main code change:
`TestAvroDFSSource#testFetchedRecordsKeepTheirOwnValues`,
`TestAvroDFSSource#testStreamerUpsertKeepsFixedAndNestedValues` fail.
- With the change: `TestAvroDFSSource` 4/4, `TestHiveIncrPullSource` 2/2
pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]