[
https://issues.apache.org/jira/browse/HUDI-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17445536#comment-17445536
]
Raymond Xu edited comment on HUDI-1870 at 11/17/21, 11:02 PM:
--------------------------------------------------------------
when build with and change spark3.version to 3.0.[0-3]
{code:java}
➜ mvn clean install -DskipTests -Drat.skip -Djacoco.skip -Dcheckstyle.skip -P
"scala-2.12,spark3,spark3.0.x"{code}
i ran into
{code:java}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile
(default-testCompile) on project hudi-spark3_2.12: Compilation failure
[ERROR]
/Users/rxu/Code/xushiyan/hudi/hudi-spark-datasource/hudi-spark3/src/test/java/org/apache/hudi/spark3/internal/TestReflectUtil.java:[55,40]
cannot find symbol
[ERROR] symbol: method userSpecifiedCols()
[ERROR] location: variable newStatment of type
org.apache.spark.sql.catalyst.plans.logical.InsertIntoStatement{code}
the original issue resolved by [[email protected]] in the 3.1 support PR.
still some different issue remaining with 3.0.[0-3]
was (Author: xushiyan):
when build with and change spark3.version to 3.0.\{1,2,3}
{code:java}
➜ mvn clean install -DskipTests -Drat.skip -Djacoco.skip -Dcheckstyle.skip
-Dscala-2.12 -Dspark3 -Dspark3.0.x{code}
i ran into
{code:java}
Symbol 'type org.slf4j.Logger' is missing from the classpath.
[ERROR] This symbol is required by 'method
org.apache.spark.internal.Logging.log'.
[ERROR] Make sure that type Logger is in your classpath and check for
conflicting dependencies with `-Ylog-classpath`.
[ERROR] A full rebuild may help if 'Logging.class' was compiled against an
incompatible version of org.slf4j.
[ERROR] if (log.isDebugEnabled) {
[ERROR] ^{code}
the original issue resolved by [[email protected]] in the 3.1 support PR.
still some different issue remaining with 3.0.\{1,2,3}
> Move spark avro serialization class into hudi repo
> --------------------------------------------------
>
> Key: HUDI-1870
> URL: https://issues.apache.org/jira/browse/HUDI-1870
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Spark Integration
> Reporter: Gary Li
> Assignee: XiaoyuGeng
> Priority: Blocker
> Labels: sev:critical
> Fix For: 0.10.0
>
>
> in Spark 3.1.1, avro serialization-related class become private. We need to
> mvoe those classes into Hudi's repo.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
