[
https://issues.apache.org/jira/browse/HUDI-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-1512:
--------------------------------------
Status: In Progress (was: Open)
> Fix hudi-spark2 unit tests failure with Spark 3.0.0
> ----------------------------------------------------
>
> Key: HUDI-1512
> URL: https://issues.apache.org/jira/browse/HUDI-1512
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Wenning Ding
> Assignee: Wenning Ding
> Priority: Major
> Labels: pull-request-available
>
> This bug is introduced by https://github.com/apache/hudi/pull/2328.
>
> hudi-spark2 unit tests failed when running with Spark 3.0.0:
> {code:java}
> mvn clean install -Dspark3 -pl hudi-spark-datasource/hudi-spark2{code}
> Threw a class not found error.
> {code:java}
> java.lang.NoClassDefFoundError: org/spark_project/guava/cache/CacheBuilder
> at
> org.apache.hudi.internal.TestHoodieBulkInsertDataInternalWriter.testGlobalFailure(TestHoodieBulkInsertDataInternalWriter.java:97)
> {code}
> When enabling spark3 profile, in hudi-spark2 module, some of the test code is
> compiled with Spark 3 and then run with Spark 2. The above error is also
> because of this incompatibility.
> To solve this compatibility issue is kind of hard, but we can skip
> hudi-spark2 unit testing when enable spark3 profile since hudi-spark2 code is
> actually not used in the Spark 3.0.0 environment.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
