eigakow commented on issue #1398: [SUPPORT] DeltaStreamer - NoClassDefFoundError for HiveDriver URL: https://github.com/apache/incubator-hudi/issues/1398#issuecomment-598062351 Yes, I have tried adding both hive-jdbc-2.1.1-cdh6.1.0.jar and hive-service-2.1.1-cdh6.1.0.jar and ended up with another missing class error, therefore I was questioning if that is the right path. Thank you for clarifying. With both jars added I am now getting ```Cannot find class 'org.apache.hudi.hadoop.HoodieParquetInputFormat'``` while hudi tries to create a new table This class is present ``` $ jar tf packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.5.1-incubating.jar | grep HoodieParquetInputFormat org/apache/hudi/hadoop/HoodieParquetInputFormat.class ``` Stacktrace: ``` 20/03/12 07:29:31 INFO hive.HiveSyncTool: Hive table test_hudi is not found. Creating it 20/03/12 07:29:31 INFO hive.HoodieHiveClient: Creating table with CREATE EXTERNAL TABLE IF NOT EXISTS `fr24raw`.`test_hudi`( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `flights` ARRAY< STRUCT< `fcode` : string, `finfo` : ARRAY< string>>>, `full_count` int, `ts` string, `version` int) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'adl://XXX.azuredatalakestore.net/test-hudi' 20/03/12 07:29:31 INFO hive.HoodieHiveClient: Executing SQL CREATE EXTERNAL TABLE IF NOT EXISTS `fr24raw`.`test_hudi`( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `flights` ARRAY< STRUCT< `fcode` : string, `finfo` : ARRAY< string>>>, `full_count` int, `ts` string, `version` int) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'adl://XXX.azuredatalakestore.net/test-hudi' 20/03/12 07:29:31 ERROR hive.HiveSyncTool: Got runtime exception when hive syncing org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL CREATE EXTERNAL TABLE IF NOT EXISTS `fr24raw`.`test_hudi`( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `flights` ARRAY< STRUCT< `fcode` : string, `finfo` : ARRAY< string>>>, `full_count` int, `ts` string, `version` int) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'adl://XXX.azuredatalakestore.net/test-hudi' at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:488) at org.apache.hudi.hive.HoodieHiveClient.createTable(HoodieHiveClient.java:276) at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:150) at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:118) at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:91) at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncHive(DeltaSync.java:481) at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:423) at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:238) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:393) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Cannot find class 'org.apache.hudi.hadoop.HoodieParquetInputFormat' at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:266) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:252) at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:318) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:259) at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:486) ... 12 more Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Cannot find class 'org.apache.hudi.hadoop.HoodieParquetInputFormat' at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207) at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:266) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:504) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:490) at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy38.executeStatementAsync(Unknown Source) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:295) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:507) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:605) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ... 3 more Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Cannot find class 'org.apache.hudi.hadoop.HoodieParquetInputFormat' at org.apache.hadoop.hive.ql.parse.ParseUtils.ensureClassExists(ParseUtils.java:263) at org.apache.hadoop.hive.ql.parse.StorageFormat.fillStorageFormat(StorageFormat.java:57) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:11666) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10839) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10949) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10639) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:600) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1391) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:205) ... 26 more Caused by: java.lang.ClassNotFoundException: org.apache.hudi.hadoop.HoodieParquetInputFormat at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.hive.ql.parse.ParseUtils.ensureClassExists(ParseUtils.java:261) ... 36 more 20/03/12 07:29:31 INFO hive.metastore: Closed a connection to metastore, current connections: 0 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
