sivabalan narayanan created HUDI-1204:
-----------------------------------------
Summary: NoClassDefFoundError with AbstractSyncTool while running
HoodieTestSuiteJob
Key: HUDI-1204
URL: https://issues.apache.org/jira/browse/HUDI-1204
Project: Apache Hudi
Issue Type: Bug
Components: Testing
Affects Versions: 0.6.1
Reporter: sivabalan narayanan
I was trying to run HoodieTestSuiteJob in my local docker set up and ran into
dep issue.
spark-submit --master local --class
org.apache.hudi.integ.testsuite.HoodieTestSuiteJob --packages
com.databricks:spark-avro_2.11:4.0.0 /opt/hudi-integ-test-bundle-0.6.0-rc1.jar
--source-ordering-field timestamp --target-base-path
/user/hive/warehouse/hudi-test-suite/output --input-base-path
/user/hive/warehouse/hudi-test-suite/input --target-table test_table
--props file:///opt/test-source.properties --schemaprovider-class
org.apache.hudi.utilities.schema.FilebasedSchemaProvider --source-class
org.apache.hudi.utilities.sources.AvroDFSSource --input-file-size 12582912
--workload-yaml-path
/var/hoodie/ws/docker/demo/config/test-suite/complex-dag-cow.yaml --table-type
COPY_ON_WRITE --workload-generator-classname yaml
```
20/08/19 21:42:26 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hudi/sync/common/AbstractSyncTool
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$Config.<init>(HoodieDeltaStreamer.java:279)
at
org.apache.hudi.integ.testsuite.HoodieTestSuiteJob$HoodieTestSuiteConfig.<init>(HoodieTestSuiteJob.java:153)
at
org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.main(HoodieTestSuiteJob.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.hudi.sync.common.AbstractSyncTool
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 26 more
```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
