Ranga Reddy created HUDI-9400:
---------------------------------
Summary: Hudi Hive Sync failed with
java.lang.NoClassDefFoundError: com/github/benmanes/caffeine/cache/Caffeine
Key: HUDI-9400
URL: https://issues.apache.org/jira/browse/HUDI-9400
Project: Apache Hudi
Issue Type: Improvement
Components: hive, meta-sync
Reporter: Ranga Reddy
While running the Hudi Hive sync inside Docker, I encountered the following
exception:
*Exception:*
{code:java}
Exception in thread "main" java.lang.NoClassDefFoundError:
com/github/benmanes/caffeine/cache/Caffeine
at org.apache.hudi.avro.AvroSchemaCache.<clinit>(AvroSchemaCache.java:36)
at
org.apache.hudi.common.table.log.block.HoodieDataBlock.<init>(HoodieDataBlock.java:115)
at
org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.<init>(HoodieHFileDataBlock.java:79)
at
org.apache.hudi.common.table.log.HoodieLogFileReader.readBlock(HoodieLogFileReader.java:202)
at
org.apache.hudi.common.table.log.HoodieLogFileReader.next(HoodieLogFileReader.java:386)
at
org.apache.hudi.common.table.log.HoodieLogFormatReader.next(HoodieLogFormatReader.java:102)
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.scanInternalV1(AbstractHoodieLogRecordScanner.java:282)
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.scanInternal(AbstractHoodieLogRecordScanner.java:252)
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:207)
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:123)
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:490)
at
org.apache.hudi.metadata.HoodieMetadataLogRecordReader$Builder.build(HoodieMetadataLogRecordReader.java:230)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:517)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.openReaders(HoodieBackedTableMetadata.java:436)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getOrCreateReaders(HoodieBackedTableMetadata.java:421)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.lookupKeysFromFileSlice(HoodieBackedTableMetadata.java:304)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:264)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:155)
at
org.apache.hudi.metadata.BaseTableMetadata.fetchAllPartitionPaths(BaseTableMetadata.java:316)
at
org.apache.hudi.metadata.BaseTableMetadata.getAllPartitionPaths(BaseTableMetadata.java:124)
at org.apache.hudi.common.fs.FSUtils.getAllPartitionPaths(FSUtils.java:249)
at
org.apache.hudi.sync.common.HoodieSyncClient.getAllPartitionPathsOnStorage(HoodieSyncClient.java:123)
at
org.apache.hudi.hive.HiveSyncTool.syncAllPartitions(HiveSyncTool.java:469)
at
org.apache.hudi.hive.HiveSyncTool.validateAndSyncPartitions(HiveSyncTool.java:321)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:261)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:189)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:177)
at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:547)
Caused by: java.lang.ClassNotFoundException:
com.github.benmanes.caffeine.cache.Caffeine
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 28 more {code}
*Spark Submit Command:*
{code:java}
/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
--jdbc-url jdbc:hive2://hiveserver:10000 \
--user hive \
--pass hive \
--partitioned-by dt \
--base-path /user/hive/warehouse/stock_ticks_cow \
--database default \
--table stock_ticks_cow \
--partition-value-extractor
org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor{code}
*Jar Grep Results:*
{code:java}
% jar tvf
packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-1.1.0-SNAPSHOT.jar
| grep 'AvroSchemaCache'
1813 Wed May 07 11:44:20 IST 2025
org/apache/hudi/common/util/LocalAvroSchemaCache.class
1895 Wed May 07 11:44:22 IST 2025 org/apache/hudi/avro/AvroSchemaCache.class
% jar tvf
packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-1.1.0-SNAPSHOT.jar
| grep 'Caffeine' -- Empty Results{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
