[
https://issues.apache.org/jira/browse/HUDI-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412198#comment-17412198
]
Robert Dunton edited comment on HUDI-1870 at 9/9/21, 2:31 AM:
--------------------------------------------------------------
This issue seems to affect building Huid with any Spark 3.0.X version aside
from 3.0.0, not just any 3.1 version. I was working on compiling Huid with the
latest dot releases for Spark, Hive, and Hadoop (based on Hudi parent pom in
0.9) on arm64 w/ Scala 2.12 and getting
{code:java}
error: class IncompatibleSchemaException in package avro cannot be accessed in
package org.apache.spark.sql.avro{code}
for anything *NOT* exactly Spark 3.0.0, eg -
* Spark 3.0.3, Hive 2.3.9, Hadoop 2.7.7 in Scala 2.12 - Fails
->[https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367542690]
* Spark 3.0.2, Hive 2.3.9, Hadoop 2.7.7 in Scala 2.12 - Fails
->[https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367547796]
* Spark 3.0.1, Hive 2.3.9, Hadoop 2.7.7 in Scala 2.12 - Fails
->[https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367548060]
* Spark 3.0.0, Hive 2.3.9, Hadoop 2.7.7 in Scala 2.12 - WORKS
->[https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367548118]
was (Author: duntonr):
This issue seems to affect building Huid with any Spark 3.0.X version aside
from 3.0.0, not just any 3.1 version. I was working on compiling Huid with the
latest dot releases for Spark, Hive, and Hadoop (based on Hudi parent pom in
0.9) on arm64 w/ Scala 2.12 and getting
{code:java}
error: class IncompatibleSchemaException in package avro cannot be accessed in
package org.apache.spark.sql.avro{code}
for anything *NOT* exactly Spark 3.0.0, eg -
* Spark 3.0.3, Hive 2.3.7, Hadoop 2.7.7 in Scala 2.12 - Fails
->[https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367542690]
* Spark 3.0.2, Hive 2.3.7, Hadoop 2.7.7 in Scala 2.12 - Fails
->[https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367547796]
* Spark 3.0.1, Hive 2.3.7, Hadoop 2.7.7 in Scala 2.12 - Fails
->[https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367548060]
* Spark 3.0.0, Hive 2.3.7, Hadoop 2.7.7 in Scala 2.12 - WORKS
->[https://gitlab.com/therackio/big-data/binaries/apache-hudi-bin-arm64/-/pipelines/367548118]
> Move spark avro serialization class into hudi repo
> --------------------------------------------------
>
> Key: HUDI-1870
> URL: https://issues.apache.org/jira/browse/HUDI-1870
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Gary Li
> Priority: Major
>
> in Spark 3.1.1, avro serialization-related class become private. We need to
> mvoe those classes into Hudi's repo.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
