[
https://issues.apache.org/jira/browse/HUDI-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenning Ding updated HUDI-1512:
-------------------------------
Description:
hudi-spark2 unit tests failed when running with Spark 3.0.0:
{code:java}
mvn clean install -Dspark3 -pl hudi-spark-datasource/hudi-spark2{code}
Threw a class not found error.
{code:java}
java.lang.NoClassDefFoundError: org/spark_project/guava/cache/CacheBuilder
at
org.apache.hudi.internal.TestHoodieBulkInsertDataInternalWriter.testGlobalFailure(TestHoodieBulkInsertDataInternalWriter.java:97)
{code}
When enabling spark3 profile, in hudi-spark2 module, some of the test code is
compiled with Spark 3 and run with Spark 2. The above error is also because of
this incompatibility.
To solve this compatibility issue is kind of hard, but we can skip hudi-spark2
unit testing when enable spark3 profile since hudi-spark2 code is actually not
used in the Spark 3.0.0 environment.
was:
hudi-spark2 unit tests failed with Spark 3.0.0:
{code:java}
mvn clean install -Dspark3 -pl hudi-spark-datasource/hudi-spark2
java.lang.NoClassDefFoundError: org/spark_project/guava/cache/CacheBuilder
at
org.apache.hudi.internal.TestHoodieBulkInsertDataInternalWriter.testGlobalFailure(TestHoodieBulkInsertDataInternalWriter.java:97)
{code}
> Fix hudi-spark2 unit tests failure with Spark 3.0.0
> ----------------------------------------------------
>
> Key: HUDI-1512
> URL: https://issues.apache.org/jira/browse/HUDI-1512
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Wenning Ding
> Priority: Major
>
> hudi-spark2 unit tests failed when running with Spark 3.0.0:
> {code:java}
> mvn clean install -Dspark3 -pl hudi-spark-datasource/hudi-spark2{code}
> Threw a class not found error.
> {code:java}
> java.lang.NoClassDefFoundError: org/spark_project/guava/cache/CacheBuilder
> at
> org.apache.hudi.internal.TestHoodieBulkInsertDataInternalWriter.testGlobalFailure(TestHoodieBulkInsertDataInternalWriter.java:97)
> {code}
> When enabling spark3 profile, in hudi-spark2 module, some of the test code is
> compiled with Spark 3 and run with Spark 2. The above error is also because
> of this incompatibility.
> To solve this compatibility issue is kind of hard, but we can skip
> hudi-spark2 unit testing when enable spark3 profile since hudi-spark2 code is
> actually not used in the Spark 3.0.0 environment.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
