[
https://issues.apache.org/jira/browse/HUDI-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sagar Sumit closed HUDI-5092.
-----------------------------
Fix Version/s: 0.12.2
(was: 0.13.0)
Resolution: Fixed
> Querying Hudi table throws NoSuchMethodError in Databricks runtime
> -------------------------------------------------------------------
>
> Key: HUDI-5092
> URL: https://issues.apache.org/jira/browse/HUDI-5092
> Project: Apache Hudi
> Issue Type: Bug
> Components: spark
> Affects Versions: 0.12.0
> Reporter: Ethan Guo
> Assignee: Ethan Guo
> Priority: Blocker
> Fix For: 0.12.2
>
>
> Originally reported by the user:
> [https://github.com/apache/hudi/issues/6137]
>
> Crux of the issue is that Databricks's DBR runtime diverges from OSS Spark,
> and in that case `FileStatusCache` API is very clearly divergent b/w the two.
> There are a few approaches we can take:
> # Avoid reliance on Spark's FIleStatusCache implementation altogether and
> rely on our own one
> # Apply more staggered approach where we first try to use Spark's
> FileStatusCache and if it doesn't match expected API, we fallback to our own
> impl
>
> Approach # 1 would actually mean that we're not sharing cache implementation
> w/ Spark, which in turn would entail that in some cases we might be keeping 2
> instances of the same cache. Approach # 2 remediates that and allows us to
> only fallback in case API is not compatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
