rangareddy commented on issue #4887: URL: https://github.com/apache/hudi/issues/4887#issuecomment-5102179076
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: This is not actionable on the Hudi side, so we are closing it here with a pointer to where the change has to happen. Impala reads Hudi Copy-on-Write tables as `HUDI_PARQUET` using code that lives in the **Impala** codebase, not in Hudi -- there is no Impala integration module in apache/hudi. As @garyli1019 explained on 2022-03-14 and 2022-04-12, making partitioned tables work correctly requires bumping the Hudi version bundled in Impala. That work is tracked as [HUDI-3537](https://issues.apache.org/jira/browse/HUDI-3537) ("Upgrade Hudi version on Impala"), which is still **Open** with no fix version, so nothing has landed. Two things from the thread worth keeping, since they are the practical takeaways: - **A workaround exists.** @chrischnweiss reported on 2022-04-25 that changing the write operation from `insert_overwrite` to `upsert` made the partitioned table readable from Impala. That is consistent with the cause: `insert_overwrite` produces a *replacecommit* on the timeline, which marks earlier file slices as replaced. A reader that treats the directory as plain parquet, without understanding the Hudi timeline, cannot resolve which slices are current -- so it can come back with no rows. `upsert` does not create replacecommits, so the problem does not arise. - **Do not mix Hive sync and Impala on the same table.** Tables registered by Hudi's Hive sync use `HoodieHiveInputFormat`, while Impala reads `HUDI_PARQUET`. Those are different read paths and pointing both at one table is unsupported. Closing on the Hudi side as out of scope. If you need this to work, the place to push is Impala -- please file or follow an issue against [Apache Impala](https://issues.apache.org/jira/projects/IMPALA) referencing HUDI-3537, and in the meantime use `upsert` rather than `insert_overwrite`, or query these tables through Spark/Hive/Trino, which do read the Hudi timeline correctly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
