[
https://issues.apache.org/jira/browse/HUDI-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-6619:
---------------------------------
Labels: pull-request-available (was: )
> Fix hudi-integ-test-bundle dependency on jackson jsk310 package.
> ----------------------------------------------------------------
>
> Key: HUDI-6619
> URL: https://issues.apache.org/jira/browse/HUDI-6619
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Amrish Lal
> Priority: Major
> Labels: pull-request-available
>
> *Command being used to run integration test*
> {code:java}
> ./bin/spark-submit \ --packages org.apache.spark:spark-avro_2.12:3.3.2 \
> --jars file:////Users/amrish/jars/jackson-datatype-jsr310-2.10.0.jar \ --conf
> spark.task.cpus=1 \ --conf spark.executor.cores=1 \ --conf
> spark.task.maxFailures=100 \ --conf spark.memory.fraction=0.4 \ --conf
> spark.rdd.compress=true \ --conf spark.kryoserializer.buffer.max=2000m \
> --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ --conf
> spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar \ --conf
> spark.memory.storageFraction=0.1 \ --conf spark.shuffle.service.enabled=true
> \ --conf spark.sql.hive.convertMetastoreParquet=false \ --conf
> spark.driver.maxResultSize=12g \ --conf spark.executor.heartbeatInterval=120s
> \ --conf spark.network.timeout=600s \ --conf
> spark.yarn.max.executor.failures=10 \ --conf
> spark.sql.catalogImplementation=hive \ --class
> org.apache.hudi.integ.testsuite.HoodieTestSuiteJob
> /Users/amrish/code/amrish-hudi-3/packaging/hudi-integ-test-bundle/target/hudi-integ-test-bundle-0.14\
> .0-SNAPSHOT.jar \ --source-ordering-field test_suite_source_ordering_field \
> --use-deltastreamer \ --target-base-path /tmp/hudi/output \ --input-base-path
> /tmp/hudi/input \ --target-table table1 \ --props
> file:///Users/amrish/junk/test-aggressive-clean-archival.properties \
> --workload-yaml-path
> file:///Users/amrish/code/amrish-hudi-3/docker/demo/config/test-suite/simple-deltastreamer.yaml
> \ --schemaprovider-class
> org.apache.hudi.integ.testsuite.schema.TestSuiteFileBasedSchemaProvider \
> --source-class org.apache.hudi.utilities.sources.AvroDFSSource \
> --input-file-size 125829120 \ --workload-generator-classname
> org.apache.hudi.integ.testsuite.dag.WorkflowDagGenerator \ --table-type
> COPY_ON_WRITE \ --compact-scheduling-minshare 1 \ --clean-input \
> --clean-output{code}
> {{ }}
> {*}Exception being thrown{*}{{{}{}}}
> {code:java}
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hudi/com/fasterxml/jackson/datatype/jsr310/JavaTimeModule at
> org.apache.hudi.common.util.JsonUtils.registerModules(JsonUtils.java:63) at
> org.apache.spark.sql.adapter.BaseSpark3Adapter.<init>(BaseSpark3Adapter.scala:51)
> at
> org.apache.spark.sql.adapter.Spark3_3Adapter.<init>(Spark3_3Adapter.scala:46)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> java.lang.Class.newInstance(Class.java:442) at
> org.apache.hudi.SparkAdapterSupport$.sparkAdapter$lzycompute(SparkAdapterSupport.scala:49){code}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
