Amrish Lal created HUDI-6619:
--------------------------------
Summary: Fix hudi-integ-test-bundle dependency on jackson jsk310
package.
Key: HUDI-6619
URL: https://issues.apache.org/jira/browse/HUDI-6619
Project: Apache Hudi
Issue Type: Bug
Reporter: Amrish Lal
*Command being used to run integration test*
{code:java}
./bin/spark-submit \ --packages org.apache.spark:spark-avro_2.12:3.3.2 \ --jars
file:////Users/amrish/jars/jackson-datatype-jsr310-2.10.0.jar \ --conf
spark.task.cpus=1 \ --conf spark.executor.cores=1 \ --conf
spark.task.maxFailures=100 \ --conf spark.memory.fraction=0.4 \ --conf
spark.rdd.compress=true \ --conf spark.kryoserializer.buffer.max=2000m \ --conf
spark.serializer=org.apache.spark.serializer.KryoSerializer \ --conf
spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar \ --conf
spark.memory.storageFraction=0.1 \ --conf spark.shuffle.service.enabled=true \
--conf spark.sql.hive.convertMetastoreParquet=false \ --conf
spark.driver.maxResultSize=12g \ --conf spark.executor.heartbeatInterval=120s \
--conf spark.network.timeout=600s \ --conf spark.yarn.max.executor.failures=10
\ --conf spark.sql.catalogImplementation=hive \ --class
org.apache.hudi.integ.testsuite.HoodieTestSuiteJob
/Users/amrish/code/amrish-hudi-3/packaging/hudi-integ-test-bundle/target/hudi-integ-test-bundle-0.14\
.0-SNAPSHOT.jar \ --source-ordering-field test_suite_source_ordering_field \
--use-deltastreamer \ --target-base-path /tmp/hudi/output \ --input-base-path
/tmp/hudi/input \ --target-table table1 \ --props
file:///Users/amrish/junk/test-aggressive-clean-archival.properties \
--workload-yaml-path
file:///Users/amrish/code/amrish-hudi-3/docker/demo/config/test-suite/simple-deltastreamer.yaml
\ --schemaprovider-class
org.apache.hudi.integ.testsuite.schema.TestSuiteFileBasedSchemaProvider \
--source-class org.apache.hudi.utilities.sources.AvroDFSSource \
--input-file-size 125829120 \ --workload-generator-classname
org.apache.hudi.integ.testsuite.dag.WorkflowDagGenerator \ --table-type
COPY_ON_WRITE \ --compact-scheduling-minshare 1 \ --clean-input \
--clean-output{code}
{{ }}
{*}Exception being thrown{*}{{{}{}}}
{code:java}
Caused by: java.lang.NoClassDefFoundError:
org/apache/hudi/com/fasterxml/jackson/datatype/jsr310/JavaTimeModule at
org.apache.hudi.common.util.JsonUtils.registerModules(JsonUtils.java:63) at
org.apache.spark.sql.adapter.BaseSpark3Adapter.<init>(BaseSpark3Adapter.scala:51)
at
org.apache.spark.sql.adapter.Spark3_3Adapter.<init>(Spark3_3Adapter.scala:46)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
java.lang.Class.newInstance(Class.java:442) at
org.apache.hudi.SparkAdapterSupport$.sparkAdapter$lzycompute(SparkAdapterSupport.scala:49){code}
{{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
