koochiswathiTR commented on issue #8178:
URL: https://github.com/apache/hudi/issues/8178#issuecomment-4853466296
**Update**: Still seeing duplicate GUIDs after upgrading to Hudi 1.0.2 with
Spark 3.5.6 on EMR 7.13.0 in our spark streaming app
We recently upgraded from Hudi 0.11.1 to Hudi 1.0.2, but we are still
observing duplicate GUIDs in our MOR table. Our current configuration is as
follows:
- Table type: MOR
- recordKey: guid
- partitionPath: collectionName
- preCombineField: operationTime
- Index type: not explicitly configured, so it defaults to SIMPLE in Hudi
1.0.x
- HoodieMetadataConfig.ENABLE: false
- Inline compaction: enabled every 6 hrs
- Rollback using markers: false
What could be causing these duplicate GUIDs?
In MOR tables, especially for partitions with fewer incremental writes, can
the Bloom index contribute to duplicate records? If so, after upgrading to Hudi
1.0.x where SIMPLE is the default index, should existing duplicates be
automatically resolved, or would they still require a separate cleanup or
reprocessing step?
Also We are looking for a permanent fix on this issue
Please help.
We tried to reproduce the issue but we are unable to produce it.
Can I setup a meeting on this to make it faster?
@rangareddy @nsivabalan @DeepanitaTR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
