Ranga Reddy created HUDI-9133:
---------------------------------
Summary: HoodieStreamer: Encountering
"java.lang.IllegalAccessError: tried to access field
org.apache.avro.Schema.FACTORY from class org.apache.avro.Schemas" exception
after upgrading to Hudi 1.0.x version
Key: HUDI-9133
URL: https://issues.apache.org/jira/browse/HUDI-9133
Project: Apache Hudi
Issue Type: Bug
Components: hudi-utilities
Affects Versions: 1.0.2
Reporter: Ranga Reddy
*Problem Statment:*
After resolving the {{ClassNotFoundException}} for the
{{ProtobufSchemaProvider}} as documented in
[HUDI-9057](https://issues.apache.org/jira/browse/HUDI-9057), we encountered a
new issue: a *{{java.lang.IllegalAccessError}} indicating an attempt to access
the field {{org.apache.avro.Schema.FACTORY}} from the class*
{{{}*org.apache.avro.Schemas*{}}}.
Upon further investigation, it was found that the
*{{kafka-schema-registry-client}}* JAR in use is version {*}5.5.0{*}, which
internally relies on {*}Avro version 1.9.2{*}. This version of Avro contains a
known bug related to the {{Schemas}} class, as highlighted in [this GitHub
issue](https://github.com/confluentinc/schema-registry/issues/1432). The bug
has been addressed in {*}Avro version 1.10.0{*}.
*Spark Submit Command:*
{code:java}
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer
hudi-utilities-bundle_2.12-1.1.0-SNAPSHOT.jar \
--props file:///tmp/my_hudi.properties \
--table-type MERGE_ON_READ \
--op UPSERT \
--target-base-path file:\/\/\/tmp/employees_cdc \
--target-table employees_cdc \
--source-class
org.apache.hudi.utilities.sources.debezium.PostgresDebeziumSource \
--source-ordering-field _event_lsn \
--payload-class
org.apache.hudi.common.model.debezium.PostgresDebeziumAvroPayload{code}
*Exception:*
{code:java}
25/03/07 12:12:14 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.hudi.exception.HoodieException: Failed to
run HoodieStreamer
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:652)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalAccessError: tried to access field
org.apache.avro.Schema.FACTORY from class org.apache.avro.Schemas
at org.apache.avro.Schemas.toString(Schemas.java:36)
at org.apache.avro.Schemas.toString(Schemas.java:30)
at
io.confluent.kafka.schemaregistry.avro.AvroSchema.canonicalString(AvroSchema.java:136)
at
org.apache.hudi.utilities.schema.SchemaRegistryProvider.fetchSchemaFromRegistry(SchemaRegistryProvider.java:180)
at
org.apache.hudi.utilities.sources.debezium.DebeziumSource.fetchNextBatch(DebeziumSource.java:127)
at
org.apache.hudi.utilities.sources.RowSource.readFromCheckpoint(RowSource.java:61)
at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:182)
at
org.apache.hudi.utilities.streamer.SourceFormatAdapter.fetchNewDataInAvroFormat(SourceFormatAdapter.java:190)
at
org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:721)
at
org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:593)
at
org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:565)
at
org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:470)
at
org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:917)
at
org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:101)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:228)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:649)
... 12 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
