[
https://issues.apache.org/jira/browse/HUDI-9400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ranga Reddy updated HUDI-9400:
------------------------------
Fix Version/s: 1.1.0
> Hudi Hive Sync failed with java.lang.NoClassDefFoundError:
> com/github/benmanes/caffeine/cache/Caffeine
> ------------------------------------------------------------------------------------------------------
>
> Key: HUDI-9400
> URL: https://issues.apache.org/jira/browse/HUDI-9400
> Project: Apache Hudi
> Issue Type: Bug
> Components: hive, meta-sync
> Reporter: Ranga Reddy
> Priority: Major
> Fix For: 1.1.0
>
>
> While running the Hudi Hive sync inside Docker, I encountered the following
> exception:
> *Exception:*
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/github/benmanes/caffeine/cache/Caffeine
> at org.apache.hudi.avro.AvroSchemaCache.<clinit>(AvroSchemaCache.java:36)
> at
> org.apache.hudi.common.table.log.block.HoodieDataBlock.<init>(HoodieDataBlock.java:115)
> at
> org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.<init>(HoodieHFileDataBlock.java:79)
> at
> org.apache.hudi.common.table.log.HoodieLogFileReader.readBlock(HoodieLogFileReader.java:202)
> at
> org.apache.hudi.common.table.log.HoodieLogFileReader.next(HoodieLogFileReader.java:386)
> at
> org.apache.hudi.common.table.log.HoodieLogFormatReader.next(HoodieLogFormatReader.java:102)
> at
> org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.scanInternalV1(AbstractHoodieLogRecordScanner.java:282)
> at
> org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.scanInternal(AbstractHoodieLogRecordScanner.java:252)
> at
> org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:207)
> at
> org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:123)
> at
> org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:490)
> at
> org.apache.hudi.metadata.HoodieMetadataLogRecordReader$Builder.build(HoodieMetadataLogRecordReader.java:230)
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:517)
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadata.openReaders(HoodieBackedTableMetadata.java:436)
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadata.getOrCreateReaders(HoodieBackedTableMetadata.java:421)
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadata.lookupKeysFromFileSlice(HoodieBackedTableMetadata.java:304)
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:264)
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:155)
> at
> org.apache.hudi.metadata.BaseTableMetadata.fetchAllPartitionPaths(BaseTableMetadata.java:316)
> at
> org.apache.hudi.metadata.BaseTableMetadata.getAllPartitionPaths(BaseTableMetadata.java:124)
> at
> org.apache.hudi.common.fs.FSUtils.getAllPartitionPaths(FSUtils.java:249)
> at
> org.apache.hudi.sync.common.HoodieSyncClient.getAllPartitionPathsOnStorage(HoodieSyncClient.java:123)
> at
> org.apache.hudi.hive.HiveSyncTool.syncAllPartitions(HiveSyncTool.java:469)
> at
> org.apache.hudi.hive.HiveSyncTool.validateAndSyncPartitions(HiveSyncTool.java:321)
> at
> org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:261)
> at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:189)
> at
> org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:177)
> at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:547)
> Caused by: java.lang.ClassNotFoundException:
> com.github.benmanes.caffeine.cache.Caffeine
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 28 more {code}
> *Spark Submit Command:*
> {code:java}
> /var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
> --jdbc-url jdbc:hive2://hiveserver:10000 \
> --user hive \
> --pass hive \
> --partitioned-by dt \
> --base-path /user/hive/warehouse/stock_ticks_cow \
> --database default \
> --table stock_ticks_cow \
> --partition-value-extractor
> org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor{code}
> *Jar Grep Results:*
> {code:java}
> % jar tvf
> packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-1.1.0-SNAPSHOT.jar
> | grep 'AvroSchemaCache'
> 1813 Wed May 07 11:44:20 IST 2025
> org/apache/hudi/common/util/LocalAvroSchemaCache.class
> 1895 Wed May 07 11:44:22 IST 2025 org/apache/hudi/avro/AvroSchemaCache.class
> % jar tvf
> packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-1.1.0-SNAPSHOT.jar
> | grep 'Caffeine' -- Empty Results{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
