[
https://issues.apache.org/jira/browse/HUDI-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180979#comment-17180979
]
Nishith Agarwal edited comment on HUDI-1204 at 8/20/20, 6:12 AM:
-----------------------------------------------------------------
After some minor changes due to code refactoring and config name changes, the
ingest job runs fine but the hive sync node running into the following
exception
```
Caused by: java.lang.NoClassDefFoundError: org/json/JSONExceptionCaused by:
java.lang.NoClassDefFoundError: org/json/JSONException at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:10847)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10047)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10128)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:209)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424) at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122) at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLs(HoodieHiveClient.java:384)
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLUsingHiveDriver(HoodieHiveClient.java:367)
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:357)
at org.apache.hudi.hive.HoodieHiveClient.createTable(HoodieHiveClient.java:262)
at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:176) at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:130) at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:94) at
org.apache.hudi.utilities.deltastreamer.DeltaSync.syncHive(DeltaSync.java:538)
at
org.apache.hudi.integ.testsuite.helpers.HiveServiceProvider.syncToLocalHiveIfNeeded(HiveServiceProvider.java:53)
at
org.apache.hudi.integ.testsuite.dag.nodes.HiveSyncNode.execute(HiveSyncNode.java:38)
at
org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:92)
at
org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.lambda$execute$0(DagScheduler.java:73)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Caused by:
java.lang.ClassNotFoundException: org.json.JSONException at
java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357)
```
Looking into this.
was (Author: nishith29):
After some minor changes due to code refactoring and config name changes, the
ingest job runs fine but the query node running into the following exception
```
Caused by: java.lang.NoClassDefFoundError: org/json/JSONExceptionCaused by:
java.lang.NoClassDefFoundError: org/json/JSONException at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:10847)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10047)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10128)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:209)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424) at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122) at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLs(HoodieHiveClient.java:384)
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLUsingHiveDriver(HoodieHiveClient.java:367)
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:357)
at org.apache.hudi.hive.HoodieHiveClient.createTable(HoodieHiveClient.java:262)
at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:176) at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:130) at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:94) at
org.apache.hudi.utilities.deltastreamer.DeltaSync.syncHive(DeltaSync.java:538)
at
org.apache.hudi.integ.testsuite.helpers.HiveServiceProvider.syncToLocalHiveIfNeeded(HiveServiceProvider.java:53)
at
org.apache.hudi.integ.testsuite.dag.nodes.HiveSyncNode.execute(HiveSyncNode.java:38)
at
org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:92)
at
org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.lambda$execute$0(DagScheduler.java:73)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Caused by:
java.lang.ClassNotFoundException: org.json.JSONException at
java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357)
```
Looking into this.
> NoClassDefFoundError with AbstractSyncTool while running HoodieTestSuiteJob
> ---------------------------------------------------------------------------
>
> Key: HUDI-1204
> URL: https://issues.apache.org/jira/browse/HUDI-1204
> Project: Apache Hudi
> Issue Type: Bug
> Components: Testing
> Affects Versions: 0.6.1
> Reporter: sivabalan narayanan
> Assignee: Nishith Agarwal
> Priority: Major
>
> I was trying to run HoodieTestSuiteJob in my local docker set up and ran into
> dep issue.
>
> spark-submit --master local --class
> org.apache.hudi.integ.testsuite.HoodieTestSuiteJob --packages
> com.databricks:spark-avro_2.11:4.0.0
> /opt/hudi-integ-test-bundle-0.6.0-rc1.jar --source-ordering-field timestamp
> --target-base-path /user/hive/warehouse/hudi-test-suite/output
> --input-base-path /user/hive/warehouse/hudi-test-suite/input
> --target-table test_table --props [file:///opt/test-source.properties]
> --schemaprovider-class
> org.apache.hudi.utilities.schema.FilebasedSchemaProvider --source-class
> org.apache.hudi.utilities.sources.AvroDFSSource --input-file-size 12582912
> --workload-yaml-path
> /var/hoodie/ws/docker/demo/config/test-suite/complex-dag-cow.yaml
> --table-type COPY_ON_WRITE --workload-generator-classname yaml
>
> {code:java}
> 20/08/19 21:42:26 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hudi/sync/common/AbstractSyncTool
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$Config.<init>(HoodieDeltaStreamer.java:279)
> at
> org.apache.hudi.integ.testsuite.HoodieTestSuiteJob$HoodieTestSuiteConfig.<init>(HoodieTestSuiteJob.java:153)
> at
> org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.main(HoodieTestSuiteJob.java:114)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hudi.sync.common.AbstractSyncTool
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 26 more
> {code}
> I tried adding hudi-sync-common as dep to hudi-utilities, but didn't fix the
> issue.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
