[
https://issues.apache.org/jira/browse/HUDI-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ranga Reddy updated HUDI-9133:
------------------------------
Description:
*Problem Statment:*
After resolving the {{ClassNotFoundException}} for the
{{ProtobufSchemaProvider}} as documented in
HUDI-9057(https://issues.apache.org/jira/browse/HUDI-9057), we encountered a
new issue: a *{{java.lang.IllegalAccessError}} indicating an attempt to access
the field {{org.apache.avro.Schema.FACTORY}} from the class*
{{{}*org.apache.avro.Schemas*{}}}.
Upon further investigation, it was found that the
*{{kafka-schema-registry-client}}* JAR in use is version {*}5.5.0{*}, which
internally relies on {*}Avro version 1.9.2{*}. This version of Avro contains a
known bug related to the {{Schemas}} class, as highlighted in [this GitHub
issue]([https://github.com/confluentinc/schema-registry/issues/1432]). The bug
has been addressed in {*}Avro version 1.10.0{*}.
*Spark Submit Command:*
{code:java}
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer
hudi-utilities-bundle_2.12-1.1.0-SNAPSHOT.jar \
--props file:///tmp/my_hudi.properties \
--table-type MERGE_ON_READ \
--op UPSERT \
--target-base-path file:\/\/\/tmp/employees_cdc \
--target-table employees_cdc \
--source-class
org.apache.hudi.utilities.sources.debezium.PostgresDebeziumSource \
--source-ordering-field _event_lsn \
--payload-class
org.apache.hudi.common.model.debezium.PostgresDebeziumAvroPayload{code}
*Exception:*
{code:java}
25/03/07 12:12:14 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.hudi.exception.HoodieException: Failed to
run HoodieStreamer
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:652)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalAccessError: tried to access field
org.apache.avro.Schema.FACTORY from class org.apache.avro.Schemas
at org.apache.avro.Schemas.toString(Schemas.java:36)
at org.apache.avro.Schemas.toString(Schemas.java:30)
at
io.confluent.kafka.schemaregistry.avro.AvroSchema.canonicalString(AvroSchema.java:136)
at
org.apache.hudi.utilities.schema.SchemaRegistryProvider.fetchSchemaFromRegistry(SchemaRegistryProvider.java:180)
at
org.apache.hudi.utilities.sources.debezium.DebeziumSource.fetchNextBatch(DebeziumSource.java:127)
at
org.apache.hudi.utilities.sources.RowSource.readFromCheckpoint(RowSource.java:61)
at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:182)
at
org.apache.hudi.utilities.streamer.SourceFormatAdapter.fetchNewDataInAvroFormat(SourceFormatAdapter.java:190)
at
org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:721)
at
org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:593)
at
org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:565)
at
org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:470)
at
org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:917)
at
org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:101)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:228)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:649)
... 12 more {code}
*Hudi code:*
https://github.com/apache/hudi/blob/release-1.0.0/hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaRegistryProvider.java#L175
was:
*Problem Statment:*
After resolving the {{ClassNotFoundException}} for the
{{ProtobufSchemaProvider}} as documented in
[HUDI-9057](https://issues.apache.org/jira/browse/HUDI-9057), we encountered a
new issue: a *{{java.lang.IllegalAccessError}} indicating an attempt to access
the field {{org.apache.avro.Schema.FACTORY}} from the class*
{{{}*org.apache.avro.Schemas*{}}}.
Upon further investigation, it was found that the
*{{kafka-schema-registry-client}}* JAR in use is version {*}5.5.0{*}, which
internally relies on {*}Avro version 1.9.2{*}. This version of Avro contains a
known bug related to the {{Schemas}} class, as highlighted in [this GitHub
issue](https://github.com/confluentinc/schema-registry/issues/1432). The bug
has been addressed in {*}Avro version 1.10.0{*}.
*Spark Submit Command:*
{code:java}
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer
hudi-utilities-bundle_2.12-1.1.0-SNAPSHOT.jar \
--props file:///tmp/my_hudi.properties \
--table-type MERGE_ON_READ \
--op UPSERT \
--target-base-path file:\/\/\/tmp/employees_cdc \
--target-table employees_cdc \
--source-class
org.apache.hudi.utilities.sources.debezium.PostgresDebeziumSource \
--source-ordering-field _event_lsn \
--payload-class
org.apache.hudi.common.model.debezium.PostgresDebeziumAvroPayload{code}
*Exception:*
{code:java}
25/03/07 12:12:14 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.hudi.exception.HoodieException: Failed to
run HoodieStreamer
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:652)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalAccessError: tried to access field
org.apache.avro.Schema.FACTORY from class org.apache.avro.Schemas
at org.apache.avro.Schemas.toString(Schemas.java:36)
at org.apache.avro.Schemas.toString(Schemas.java:30)
at
io.confluent.kafka.schemaregistry.avro.AvroSchema.canonicalString(AvroSchema.java:136)
at
org.apache.hudi.utilities.schema.SchemaRegistryProvider.fetchSchemaFromRegistry(SchemaRegistryProvider.java:180)
at
org.apache.hudi.utilities.sources.debezium.DebeziumSource.fetchNextBatch(DebeziumSource.java:127)
at
org.apache.hudi.utilities.sources.RowSource.readFromCheckpoint(RowSource.java:61)
at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:182)
at
org.apache.hudi.utilities.streamer.SourceFormatAdapter.fetchNewDataInAvroFormat(SourceFormatAdapter.java:190)
at
org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:721)
at
org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:593)
at
org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:565)
at
org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:470)
at
org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:917)
at
org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:101)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:228)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:649)
... 12 more {code}
> HoodieStreamer: Encountering "java.lang.IllegalAccessError: tried to access
> field org.apache.avro.Schema.FACTORY from class org.apache.avro.Schemas"
> exception after upgrading to Hudi 1.0.x version
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HUDI-9133
> URL: https://issues.apache.org/jira/browse/HUDI-9133
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: hudi-utilities
> Affects Versions: 1.0.2
> Reporter: Ranga Reddy
> Priority: Blocker
> Labels: HoodieStreamer
>
> *Problem Statment:*
> After resolving the {{ClassNotFoundException}} for the
> {{ProtobufSchemaProvider}} as documented in
> HUDI-9057(https://issues.apache.org/jira/browse/HUDI-9057), we encountered a
> new issue: a *{{java.lang.IllegalAccessError}} indicating an attempt to
> access the field {{org.apache.avro.Schema.FACTORY}} from the class*
> {{{}*org.apache.avro.Schemas*{}}}.
> Upon further investigation, it was found that the
> *{{kafka-schema-registry-client}}* JAR in use is version {*}5.5.0{*}, which
> internally relies on {*}Avro version 1.9.2{*}. This version of Avro contains
> a known bug related to the {{Schemas}} class, as highlighted in [this GitHub
> issue]([https://github.com/confluentinc/schema-registry/issues/1432]). The
> bug has been addressed in {*}Avro version 1.10.0{*}.
> *Spark Submit Command:*
> {code:java}
> spark-submit \
> --class org.apache.hudi.utilities.streamer.HoodieStreamer
> hudi-utilities-bundle_2.12-1.1.0-SNAPSHOT.jar \
> --props file:///tmp/my_hudi.properties \
> --table-type MERGE_ON_READ \
> --op UPSERT \
> --target-base-path file:\/\/\/tmp/employees_cdc \
> --target-table employees_cdc \
> --source-class
> org.apache.hudi.utilities.sources.debezium.PostgresDebeziumSource \
> --source-ordering-field _event_lsn \
> --payload-class
> org.apache.hudi.common.model.debezium.PostgresDebeziumAvroPayload{code}
> *Exception:*
> {code:java}
> 25/03/07 12:12:14 INFO SparkContext: Successfully stopped SparkContext
> Exception in thread "main" org.apache.hudi.exception.HoodieException: Failed
> to run HoodieStreamer
> at
> org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:652)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.IllegalAccessError: tried to access field
> org.apache.avro.Schema.FACTORY from class org.apache.avro.Schemas
> at org.apache.avro.Schemas.toString(Schemas.java:36)
> at org.apache.avro.Schemas.toString(Schemas.java:30)
> at
> io.confluent.kafka.schemaregistry.avro.AvroSchema.canonicalString(AvroSchema.java:136)
> at
> org.apache.hudi.utilities.schema.SchemaRegistryProvider.fetchSchemaFromRegistry(SchemaRegistryProvider.java:180)
> at
> org.apache.hudi.utilities.sources.debezium.DebeziumSource.fetchNextBatch(DebeziumSource.java:127)
> at
> org.apache.hudi.utilities.sources.RowSource.readFromCheckpoint(RowSource.java:61)
> at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:182)
> at
> org.apache.hudi.utilities.streamer.SourceFormatAdapter.fetchNewDataInAvroFormat(SourceFormatAdapter.java:190)
> at
> org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:721)
> at
> org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:593)
> at
> org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:565)
> at
> org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:470)
> at
> org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:917)
> at
> org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
> at org.apache.hudi.common.util.Option.ifPresent(Option.java:101)
> at
> org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:228)
> at
> org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:649)
> ... 12 more {code}
> *Hudi code:*
> https://github.com/apache/hudi/blob/release-1.0.0/hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaRegistryProvider.java#L175
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
