Raymond Xu created HUDI-4549:
--------------------------------
Summary: hive sync bundle causes class loader issue
Key: HUDI-4549
URL: https://issues.apache.org/jira/browse/HUDI-4549
Project: Apache Hudi
Issue Type: Bug
Components: dependencies
Reporter: Raymond Xu
Fix For: 0.12.0
A weird classpath issue i found: when testing deltastreamer using
hudi-utilities-slim-bundle, if i put --jars
hudi-hive-sync-bundle.jar,hudi-spark-bundle.jar then i’ll get this error when
writing
{code:java}
Caused by: java.lang.NoSuchMethodError:
org.apache.hudi.avro.MercifulJsonConverter.convert(Ljava/lang/String;Lorg/apache/avro/Schema;)Lorg/apache/avro/generic/GenericRecord;
at
org.apache.hudi.utilities.sources.helpers.AvroConvertor.fromJson(AvroConvertor.java:86)
at
org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
{code}
if i put the spark bundle before the hive sync bundle, then no issue. Without
hive-sync-bundle, also no issue. So hive-sync-bundle somehow messes up with
classpath? not sure why it reports a hudi-common API not found… caused by
shading avro?
the same behavior i observed with aws-bundle, which makes sense, as it’s a
superset of hive-sync-bundle
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
