ravjot28 opened a new issue, #20586:
URL: https://github.com/apache/hudi/issues/20586
### Bug Description
**What happened:**
`HoodieStreamer` with `AvroDFSSource` (or `HiveIncrPullSource`) and an
upsert writes wrong values without any error:
- When the target schema differs from the source schema, every fixed
(decimals included), array, map and nested record
column holds the value of the last row read with it from the same input
split. In a nullable column a null ends the
sharing; string, bytes and primitive columns are right.
- When the target schema equals the source schema, every data column of
every row of the split holds the last row's
values (record keys and partition paths are still computed per row).
- At the source alone, `source.fetchNext(...).getBatch().get().collect()`
returns many references to one record.
**What you expected:**
Every row holds the values of its own input record.
**Steps to reproduce:**
1. Write an Avro data file with more than one row and different values,
whose schema has a `fixed` field (for example a
decimal), an array, a map or a nested record:
```json
{"name": "price", "type": ["null", {"type": "fixed", "name":
"price_fixed", "size": 6,
"logicalType": "decimal", "precision": 12, "scale": 2}], "default":
null}
```
2. Run `HoodieStreamer` with `--source-class
org.apache.hudi.utilities.sources.AvroDFSSource`,
`--schemaprovider-class
org.apache.hudi.utilities.schema.FilebasedSchemaProvider` and `--op UPSERT`
(record key and
ordering field set, default payload and merge mode).
3. Read the table: with 20 input rows, all 20 rows store the same decimal.
**Root cause (master d5230b69e22):**
1.
`hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroDFSSource.java:65-67`
reads the files with
`AvroKeyInputFormat` and returns `avroRDD.keys().map(r ->
((GenericRecord) r.datum()))`; `HiveIncrPullSource.java:133-136`
does the same. The record reader of that format reuses its previous datum
for every row of a file: records, nested
records, arrays, maps and fixed values are the same instances, only
strings and bytes are new objects.
2.
`hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:805-806`
rewrites every record to the
target schema; `HoodieAvroUtils.rewriteRecordWithNewSchema` returns a
value unchanged when its schema is equal
(`hudi-common/src/main/java/org/apache/hudi/common/avro/HoodieAvroUtils.java:1106-1109`),
so shared nested values pass
through, and with an equal target schema the whole record does.
3. The streamer keeps the source record by reference in its Hudi record
(since #13726, 1.1.0), and the deduplication
before an upsert
(`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/HoodieWriteHelper.java:70`,
`reduceByKey`) holds every record of the partition in memory until it
serializes them for the shuffle. By then the
shared instances hold the last row's values.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1, and every
release since 1.1.0. Earlier releases share
the instances as well, but their payload classes serialize each record when
it is created, so the stored values are
right there.
**Query engine:** Spark 3.5 (HoodieStreamer). The table type and table
version do not matter: the values are already
shared before the write.
**Relevant configs:** `--source-class
org.apache.hudi.utilities.sources.AvroDFSSource` (or `HiveIncrPullSource`),
`--op UPSERT`
### Logs and Stack Trace
none: the commit succeeds and nothing is logged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]