deepakpanda93 commented on issue #15720: URL: https://github.com/apache/hudi/issues/15720#issuecomment-4852032403
This is addressed in Hudi 1.x, which added dedicated Databricks Spark runtime support. On 0.12.2 there was no Databricks-runtime support, so Hudi's file index didn't engage on the Databricks Spark fork and SQL/count() returned 0 — the only reads that worked were the ones that bypass the file index (glob path */*/* or recursiveFileLookup=true), which is exactly what you observed. Since then Hudi added explicit Databricks runtime support: - HUDI-5092 / [#13129](https://github.com/apache/hudi/pull/13129) "Support Databricks Spark runtime" (1.0.2), - [#18291](https://github.com/apache/hudi/pull/18291)/[#18292](https://github.com/apache/hudi/pull/18292) "Databricks Spark 3.4 Runtime compatibility for reading Hudi tables" (backported to 0.15.1, and in 1.2.0 with a dedicated DatabricksRuntimeHelper), - plus partition-predicate ([#18257](https://github.com/apache/hudi/pull/18257)) and incremental-query ([#18258](https://github.com/apache/hudi/pull/18258)) fixes for the Databricks runtime. Please retest on 1.2.0 with a Databricks runtime on Spark 3.4+ (the read-compatibility work targets Spark 3.4). Reading via spark.read.format("hudi").load(basePath) — without glob/recursive workarounds — should now return the correct count. Closing as fixed in 1.x; please reopen if you still see count=0 on 1.2.0 + Databricks 3.4, with the runtime version. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
