soumilshah1995 commented on issue #351:
URL:
https://github.com/apache/incubator-xtable/issues/351#issuecomment-1974905580
@the-other-tim-brown Still getting Error
```
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer \
--packages 'org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0' \
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
--jars
'/Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-extensions-0.1.0-SNAPSHOT-bundled.jar'
\
/Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar
\
--table-type COPY_ON_WRITE \
--target-base-path
'file:///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/hudi/bronze_orders'
\
--target-table bronze_orders \
--op UPSERT \
--enable-sync \
--sync-tool-classes 'io.onetable.hudi.sync.OneTableSyncTool' \
--source-limit 4000000 \
--source-ordering-field ts \
--source-class org.apache.hudi.utilities.sources.CsvDFSSource \
--hoodie-conf 'hoodie.datasource.write.recordkey.field=order_id' \
--hoodie-conf 'hoodie.datasource.write.partitionpath.field=order_date' \
--hoodie-conf 'hoodie.datasource.write.precombine.field=ts' \
--hoodie-conf
'hoodie.streamer.source.dfs.root=file://///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/sampledata/orders'
\
--hoodie-conf 'hoodie.deltastreamer.csv.header=true' \
--hoodie-conf 'hoodie.deltastreamer.csv.sep=\t' \
--hoodie-conf 'hoodie.onetable.formats.to.sync=DELTA'
```
# Error
```
4/03/02 15:47:34 INFO FileSystemViewManager: Creating in-memory based Table
View
24/03/02 15:47:34 WARN StreamSync: SyncTool class
io.onetable.hudi.sync.OneTableSyncTool failed with exception
org.apache.hudi.exception.HoodieMetaSyncException: Could not sync using the
meta sync class io.onetable.hudi.sync.OneTableSyncTool
at
org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:81)
at
org.apache.hudi.utilities.streamer.StreamSync.runMetaSync(StreamSync.java:938)
at
org.apache.hudi.utilities.streamer.StreamSync.writeToSink(StreamSync.java:851)
at
org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:446)
at
org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:840)
at
org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:205)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:584)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.ServiceConfigurationError:
io.onetable.spi.sync.TargetClient: io.onetable.hudi.HudiTargetClient Unable to
get public no-arg constructor
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
at
java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:673)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1233)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
at
io.onetable.client.TableFormatClientFactory.createTargetClientForName(TableFormatClientFactory.java:67)
at
io.onetable.client.TableFormatClientFactory.createForFormat(TableFormatClientFactory.java:51)
at
io.onetable.client.OneTableClient.lambda$sync$0(OneTableClient.java:99)
at
java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178)
at
java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at io.onetable.client.OneTableClient.sync(OneTableClient.java:95)
at
io.onetable.hudi.sync.OneTableSyncTool.syncHoodieTable(OneTableSyncTool.java:80)
at
org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:79)
... 20 more
Caused by: java.lang.NoClassDefFoundError:
org/apache/hudi/client/common/HoodieJavaEngineContext
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at
java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
at java.base/java.lang.Class.getConstructor0(Class.java:3342)
at java.base/java.lang.Class.getConstructor(Class.java:2151)
at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:660)
at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:657)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at
java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:668)
... 38 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hudi.client.common.HoodieJavaEngineContext
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
... 46 more
24/03/02 15:47:34 INFO StreamSync: Shutting down embedded timeline server
24/03/02 15:47:34 INFO EmbeddedTimelineService: Closing Timeline server
24/03/02 15:47:34 INFO TimelineService: Closing Timeline Service
24/03/02 15:47:34 INFO Javalin: Stopping Javalin ...
24/03/02 15:47:34 INFO Javalin: Javalin has stopped
24/03/02 15:47:34 INFO TimelineService: Closed Timeline Service
24/03/02 15:47:34 INFO EmbeddedTimelineService: Closed Timeline server
24/03/02 15:47:34 INFO SparkContext: SparkContext is stopping with exitCode
0.
24/03/02 15:47:34 INFO SparkUI: Stopped Spark web UI at
http://soumils-mbp:8090
24/03/02 15:47:34 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
24/03/02 15:47:34 INFO MemoryStore: MemoryStore cleared
24/03/02 15:47:34 INFO BlockManager: BlockManager stopped
24/03/02 15:47:34 INFO BlockManagerMaster: BlockManagerMaster stopped
24/03/02 15:47:34 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
24/03/02 15:47:34 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main"
org.apache.hudi.exception.HoodieMetaSyncException: Could not sync using the
meta sync class io.onetable.hudi.sync.OneTableSyncTool
at
org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:81)
at
org.apache.hudi.utilities.streamer.StreamSync.runMetaSync(StreamSync.java:938)
at
org.apache.hudi.utilities.streamer.StreamSync.writeToSink(StreamSync.java:851)
at
org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:446)
at
org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:840)
at
org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:205)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:584)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.ServiceConfigurationError:
io.onetable.spi.sync.TargetClient: io.onetable.hudi.HudiTargetClient Unable to
get public no-arg constructor
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
at
java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:673)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1233)
at
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
at
io.onetable.client.TableFormatClientFactory.createTargetClientForName(TableFormatClientFactory.java:67)
at
io.onetable.client.TableFormatClientFactory.createForFormat(TableFormatClientFactory.java:51)
at
io.onetable.client.OneTableClient.lambda$sync$0(OneTableClient.java:99)
at
java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178)
at
java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at io.onetable.client.OneTableClient.sync(OneTableClient.java:95)
at
io.onetable.hudi.sync.OneTableSyncTool.syncHoodieTable(OneTableSyncTool.java:80)
at
org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:79)
... 20 more
Caused by: java.lang.NoClassDefFoundError:
org/apache/hudi/client/common/HoodieJavaEngineContext
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at
java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
at java.base/java.lang.Class.getConstructor0(Class.java:3342)
at java.base/java.lang.Class.getConstructor(Class.java:2151)
at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:660)
at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:657)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at
java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:668)
... 38 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hudi.client.common.HoodieJavaEngineContext
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
... 46 more
24/03/02 15:47:34 INFO ShutdownHookManager: Shutdown hook called
24/03/02 15:47:34 INFO ShutdownHookManager: Deleting directory
/private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-4590b9e3-daae-4317-9782-c7126533de25
24/03/02 15:47:34 INFO ShutdownHookManager: Deleting directory
/private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-ea3fcde5-0a6c-46e7-aad7-264fd4c4c8f6
soumilshah@Soumils-MBP DeltaStreamer %
```
would it be best for quick call to resolve it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]