soumilshah1995 commented on issue #351:
URL:
https://github.com/apache/incubator-xtable/issues/351#issuecomment-1971814812
Thanks for the clarification. I understand now that I can't achieve the
table format directly. What I meant to convey is that I'd like to synchronize
data into all three formats (Hudi, Delta, and Iceberg) simultaneously during
write operations from the Hudi streamer.
I'm not familiar with Java or Maven, so I'm unsure about packaging the JAR.
Could you assist me with creating the JAR file for Spark 3.4?
To clarify further, my goal is to sync all three tables (Hudi, Delta, and
Iceberg) with every consecutive write operation via the delta streamer. I'm
hoping to avoid running separate processes for each format.
Regarding the JAR execution, would the following command suffice?
```
java -jar ./jar/utilities-0.1.0-beta1-bundled.jar --dataset ./my_config.yaml
```
Please confirm if achieving this synchronization within the Hudi Streamer is
feasible, and if so, how I can properly pass the JAR and what changes or
medication to below spark-submit job is required
```
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer \
--packages 'org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0'\
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
/Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar
\
--table-type COPY_ON_WRITE \
--op UPSERT \
--source-limit 4000000 \
--source-ordering-field ts \
--source-class org.apache.hudi.utilities.sources.CsvDFSSource \
--target-base-path
'file:///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/hudi/bronze_orders'
\
--target-table bronze_orders \
--props hudi_tbl.props
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
