ravjot28 opened a new pull request, #20596:
URL: https://github.com/apache/hudi/pull/20596

   ### Describe the issue this Pull Request addresses
   
   Closes #20586
   
   The record reader of the Hadoop Avro input format reuses the record and its 
nested record, array, map and fixed
   instances for all rows of a file. `AvroDFSSource` and `HiveIncrPullSource` 
emitted that instance, and the streamer
   keeps source records by reference until an upsert deduplicates them, so rows 
took the fixed, array, map and nested
   values of the last row read, or the whole last row when the source and 
target schemas are equal. The write succeeds
   with wrong data.
   
   ### Summary and Changelog
   
   Every record the two sources emit is an independent copy, so each row keeps 
its own values.
   
   - Add `AvroSource#toIndependentRecords`, which deep-copies each record read 
through `AvroKeyInputFormat`
     (`GenericData.deepCopy` with the record's schema).
   - `AvroDFSSource` and `HiveIncrPullSource` use it.
   - New tests: `TestAvroDFSSource#testFetchedRecordsKeepTheirOwnValues` (the 
source alone) and
     `TestAvroDFSSource#testStreamerUpsertKeepsFixedAndNestedValues` (a 
streamer upsert of 20 rows with a fixed decimal, a
     nested record and an array column).
   
   ### Impact
   
   No API or configuration change. The two sources make one deep copy per 
record read; other sources are unchanged.
   Spark's `newAPIHadoopFile` documentation asks callers to copy the objects of 
a Hadoop record reader before caching or
   aggregating them; the copy is made where the reader's object leaves the 
source.
   Tables written by 1.1.0 or later through these sources with an upsert may 
hold wrong values in fixed, array, map and
   nested record columns (or in every column when the source and target schemas 
are equal) and need to be rewritten from
   the source data.
   
   **Backport:** please include this fix in the next 1.2.x patch release. The 
commit cherry-picks cleanly onto
   `release-1.2.1`.
   
   ### Risk Level
   
   low. A copy per record read, in two sources only.
   
   Tests, run on master `d5230b69e22` with JDK 11 and passing checkstyle and 
scalastyle checks, `mvn test -pl hudi-utilities -am 
-Dtest=TestAvroDFSSource,TestHiveIncrPullSource 
-DwildcardSuites=skipScalaTests`:
   - Without the main code change: 
`TestAvroDFSSource#testFetchedRecordsKeepTheirOwnValues`, 
`TestAvroDFSSource#testStreamerUpsertKeepsFixedAndNestedValues` fail.
   - With the change: `TestAvroDFSSource` 4/4, `TestHiveIncrPullSource` 2/2 
pass.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to