soumilshah1995 commented on issue #351:
URL:
https://github.com/apache/incubator-xtable/issues/351#issuecomment-1975180157
I added this two config now its throwing different error
```
--hoodie-conf 'hoodie.datasource.write.row.writer.enable=false' \
--hoodie-conf
'hoodie.avro.write.support.class=io.onetable.hudi.extensions.HoodieAvroWriteSupportWithFieldIds'
\
--hoodie-conf
'hoodie.client.init.callback.classes=io.onetable.hudi.extensions.AddFieldIdsClientInitCallback'
```
# Spark Submit
```
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer \
--packages 'org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0' \
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
--jars
'/Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/new_jars/hudi-extensions-0.1.0-SNAPSHOT.jar'
\
/Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar
\
--table-type COPY_ON_WRITE \
--target-base-path
'file:///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/hudi/bronze_orders'
\
--target-table bronze_orders \
--op UPSERT \
--enable-sync \
--sync-tool-classes 'io.onetable.hudi.sync.OneTableSyncTool' \
--source-limit 4000000 \
--source-ordering-field ts \
--source-class org.apache.hudi.utilities.sources.CsvDFSSource \
--hoodie-conf 'hoodie.datasource.write.recordkey.field=order_id' \
--hoodie-conf 'hoodie.datasource.write.partitionpath.field=order_date' \
--hoodie-conf 'hoodie.datasource.write.precombine.field=ts' \
--hoodie-conf
'hoodie.streamer.source.dfs.root=file://///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/sampledata/orders'
\
--hoodie-conf 'hoodie.deltastreamer.csv.header=true' \
--hoodie-conf 'hoodie.deltastreamer.csv.sep=\t' \
--hoodie-conf 'hoodie.onetable.formats.to.sync=DELTA' \
--hoodie-conf 'hoodie.onetable.target.metadata.retention.hr=168' \
--hoodie-conf 'hoodie.datasource.write.row.writer.enable=false' \
--hoodie-conf
'hoodie.avro.write.support.class=io.onetable.hudi.extensions.HoodieAvroWriteSupportWithFieldIds'
\
--hoodie-conf
'hoodie.client.init.callback.classes=io.onetable.hudi.extensions.AddFieldIdsClientInitCallback'
```
# Error
```
03 09:22:29 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
24/03/03 09:22:29 INFO MemoryStore: MemoryStore cleared
24/03/03 09:22:29 INFO BlockManager: BlockManager stopped
24/03/03 09:22:29 INFO BlockManagerMaster: BlockManagerMaster stopped
24/03/03 09:22:29 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
24/03/03 09:22:29 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoClassDefFoundError:
io/onetable/hudi/idtracking/IdTracker
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at
java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
at java.base/java.lang.Class.getConstructor0(Class.java:3342)
at java.base/java.lang.Class.newInstance(Class.java:556)
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:68)
at
org.apache.hudi.client.BaseHoodieClient.lambda$runClientInitCallbacks$0(BaseHoodieClient.java:152)
at
java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at
java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at
org.apache.hudi.client.BaseHoodieClient.runClientInitCallbacks(BaseHoodieClient.java:151)
at
org.apache.hudi.client.BaseHoodieClient.<init>(BaseHoodieClient.java:100)
at
org.apache.hudi.client.BaseHoodieWriteClient.<init>(BaseHoodieWriteClient.java:163)
at
org.apache.hudi.client.SparkRDDWriteClient.<init>(SparkRDDWriteClient.java:86)
at
org.apache.hudi.utilities.streamer.StreamSync.reInitWriteClient(StreamSync.java:988)
at
org.apache.hudi.utilities.streamer.StreamSync.setupWriteClient(StreamSync.java:961)
at
org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:414)
at
org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:840)
at
org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:205)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:584)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
io.onetable.hudi.idtracking.IdTracker
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
... 32 more
24/03/03 09:22:34 INFO ShutdownHookManager: Shutdown hook called
24/03/03 09:22:34 INFO ShutdownHookManager: Deleting directory
/private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-01d5445b-7bc1-49ad-aa5e-35f565e64adb
24/03/03 09:22:34 INFO ShutdownHookManager: Deleting directory
/private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-a5ebc47a-ea50-4c0a-a1ed-066e2761e6b5
soumilshah@Soumils-MBP DeltaStreamer %
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
