rangareddy commented on code in PR #19617: URL: https://github.com/apache/hudi/pull/19617#discussion_r3774152994
########## website/docs/quick-start-guide.md: ########## @@ -28,6 +28,38 @@ Hudi works with Spark 3.3 and above versions. You can follow instructions [here] The *default build* Spark version indicates how we build `hudi-spark3-bundle`. ::: +### Reading Hudi tables on the Databricks runtime + +The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a modified Spark, and a few of the +internals Hudi's Spark datasource builds on differ there. Reading a Hudi table on DBR needs **no +Hudi-specific configuration** — Hudi detects those differences at runtime and adapts, rather than requiring +you to set a flag: + +* **`FileStatusCache`** — DBR changed this API. Hudi checks reflectively for the + `putLeafFiles(Path, FileStatus[])` signature before using it and falls back when it is absent, instead of + failing with `NoSuchMethodError`. +* **`PartitionDirectory`** — DBR's Spark 3.4 runtime backports `FileStatusWithMetadata` from Spark 3.5, so + `PartitionDirectory` takes a `Seq[FileStatusWithMetadata]` rather than a `Seq[FileStatus]`. That type wraps + `FileStatus` by composition instead of extending it, so the elements cannot simply be cast; Hudi constructs + the wrapper reflectively. This adaptation ships in **1.2.0**, so a DBR Spark 3.4 runtime needs 1.2.0 or + later. + +Snapshot, read-optimized and incremental queries all go through these paths, including incremental queries in Review Comment: You are right, and this was an overclaim on my part rather than a wording nit — thanks for pushing on it. The sentence *"Snapshot, read-optimized and incremental queries all go through these paths, including incremental queries in full-scan mode on Merge-on-Read tables"* was assembled from PR titles (#18003/#18258 mention MOR full-scan incremental), **not** from tracing the read paths. I had no basis for an exhaustive per-query-type claim, and your observation about module scope is exactly why it cannot be made casually: the `PartitionDirectory` construction lives in `hudi-spark3.4.x` (`HoodieSpark34PartitionedFileUtils` → `DatabricksRuntimeHelper`), so it is specific to a DBR release built on Spark 3.4, while the `FileStatusCache` guard is in the shared `hudi-spark-common` module and applies regardless. Rather than ask a committer to certify a claim I should not have made, I have removed it. The section now scopes each adaptation to the module and release it actually lives in, and makes no per-query-type coverage statement at all: > **`FileStatusCache`** — … This guard lives in the shared Spark module and has been present since 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well. > > **`PartitionDirectory`** — … This adaptation lives in the Spark 3.4 module and **ships in 1.2.0**, so a DBR release built on Spark 3.4 — DBR 13.3 LTS, for instance — needs Hudi 1.2.0 or later. If someone with a Databricks environment later confirms the MOR full-scan incremental path end to end, that is worth adding as a positive statement — but it should come from a real run, not from me inferring it. ########## website/docs/quick-start-guide.md: ########## @@ -28,6 +28,38 @@ Hudi works with Spark 3.3 and above versions. You can follow instructions [here] The *default build* Spark version indicates how we build `hudi-spark3-bundle`. ::: +### Reading Hudi tables on the Databricks runtime + +The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a modified Spark, and a few of the +internals Hudi's Spark datasource builds on differ there. Reading a Hudi table on DBR needs **no +Hudi-specific configuration** — Hudi detects those differences at runtime and adapts, rather than requiring +you to set a flag: + +* **`FileStatusCache`** — DBR changed this API. Hudi checks reflectively for the + `putLeafFiles(Path, FileStatus[])` signature before using it and falls back when it is absent, instead of + failing with `NoSuchMethodError`. Review Comment: Applied, and thank you — this is a genuinely useful completeness point, because as written the page left 1.0.x/1.1.x readers unsure which half of the story applied to them. I checked the tags rather than take the 1.1.0 attribution on trust. The `FileStatusCache` guard in `SparkHoodieTableFileIndex` (the `Databricks Spark has changed the FileStatusCache APIs` comment and the `ReflectionUtils.getMethod(...)` gate) is present at **`release-0.15.1`, `release-1.0.2`, `release-1.1.1` and `release-1.2.0`** — so it goes back further still. `DatabricksRuntimeHelper` is present only at `release-1.2.0`, absent at `1.1.1` and `1.0.2`, which is what keeps this PR scoped to `docs/` + `version-1.2.0`. The bullets now state each release separately: > This guard lives in the shared Spark module and has been present since 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well. > This adaptation lives in the Spark 3.4 module and **ships in 1.2.0** … While in there I also added the piece the issue actually asked for and I had left out — the cluster setup steps: install the `hudi-spark<spark>-bundle_<scala>` jar as a cluster library, and set Hudi's four Spark configs in the cluster Spark config box in the `key value` form that UI takes. Those four values are byte-identical to the ones this page already passes to `spark-shell` at lines 57-60, and I verified each class exists (`HoodieSparkSessionExtension.scala:29`, `HoodieCatalog.scala:58`, `HoodieSparkKryoRegistrar.scala:53`). One related change worth flagging: community guides for older releases present `hoodie.file.index.enable=false` as **mandatory** on Databricks. My first revision simply warned against it, which would have read as contradicting guides users are actually following. It now explains the history instead — that it was the workaround before these adaptations existed, that it disables the file index session-wide, that it is deprecated since 0.11.0, and that on 1.2.0+ it should not be needed but remains a fallback worth reporting if it is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
