rangareddy commented on issue #16617: URL: https://github.com/apache/hudi/issues/16617#issuecomment-5434691172
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8209). **Findings: needs a reproduction - the implementation this blames has since been replaced.** The report is that unmerged (skip-merge) reads on the Hive path return log-file records twice. On `master` the Hive read path has moved onto the file group reader: `hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/` now contains `HoodieFileGroupReaderBasedRecordReader`, `HiveRecordContext` and `SchemaEvolutionContext`, and the legacy `RealtimeCompactedRecordReader` is gone. Skip-merge is now decided centrally in `HoodieFileGroupReader` via `REALTIME_SKIP_MERGE` rather than per engine. So the specific implementation this ticket blames no longer exists. But a rewrite is not evidence of correctness, and records being read twice is a silent wrong-results bug rather than a failure - exactly the kind that survives a refactor unnoticed. That is worth verifying rather than assuming. The check is small: read a MOR table through Hive with `hoodie.datasource.merge.type=skip_merge`, on a file group with log files, and assert no duplicates. Sibling worth doing in the same pass: #17270 (HUDI-8207) reports the same class of bug for the Spark relation implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
