pan3793 commented on issue #4793: URL: https://github.com/apache/hudi/issues/4793#issuecomment-1054461384
@xushiyan thanks for helping, and sorry I didn't notice your first reply. > can you post which vanilla jars you used? Basically, we use the following jars related to Hudi, you can check more details in our project source code https://github.com/apache/incubator-kyuubi <details> <summary>pom.xml</summary> ``` <!-- Hudi dependency --> <!-- We don't use hadoop-common directly, it's only for suppressing exception: Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project kyuubi-spark-sql-engine_2.12: Error creating shaded jar: Could not resolve following dependencies: [jdk.tools:jdk.tools:jar:1.6 (system)] The issue only occurs on GitHub Action environment with Hudi 0.10.0 and JDK 11. After few days digging, only found one place introduces jdk.tools, - org.apache.hudi:hudi-common:jar:0.10.0:test - org.apache.hbase:hbase-server:jar:1.2.3:test - org.apache.hadoop:hadoop-common:jar:2.5.1:test - org.apache.hadoop:hadoop-annotations:jar:2.5.1:test - jdk.tools:jdk.tools:jar:1.6:system --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>${hadoop.version}</version> </dependency> <dependency> <groupId>org.apache.parquet</groupId> <artifactId>parquet-avro</artifactId> <version>${parquet.version}</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-avro_${scala.binary.version}</artifactId> <version>${spark.version}</version> </dependency> <dependency> <groupId>org.apache.hudi</groupId> <artifactId>hudi-spark-common_${scala.binary.version}</artifactId> <version>${hudi.version}</version> <exclusions> <exclusion> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> </exclusion> <exclusion> <groupId>org.apache.hudi</groupId> <artifactId>hudi-timeline-service</artifactId> </exclusion> <exclusion> <groupId>io.dropwizard.metrics</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>io.prometheus</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <exclusion> <groupId>org.apache.curator</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>org.apache.hadoop</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>org.apache.hbase</groupId> <artifactId>hbase-server</artifactId> </exclusion> <exclusion> <groupId>org.apache.orc</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>org.apache.hudi</groupId> <artifactId>hudi-aws</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hudi</groupId> <artifactId>hudi-spark_${scala.binary.version}</artifactId> <version>${hudi.version}</version> <exclusions> <exclusion> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> </exclusion> <exclusion> <groupId>org.apache.hudi</groupId> <artifactId>hudi-spark-common_2.11</artifactId> </exclusion> <exclusion> <groupId>org.apache.hudi</groupId> <artifactId>hudi-spark2_2.11</artifactId> </exclusion> <exclusion> <groupId>org.apache.curator</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>com.fasterxml.jackson.module</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hudi</groupId> <artifactId>hudi-spark3_${scala.binary.version}</artifactId> <version>${hudi.version}</version> <exclusions> <exclusion> <groupId>org.apache.hudi</groupId> <artifactId>hudi-spark-common_2.11</artifactId> </exclusion> </exclusions> </dependency> ``` </details> > did you build the vanilla jars with spark 3 profile? No, we use the jar published by Hudi officially, it works fine on Hudi 0.10.0, but not Hudi 0.10.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
