This is an automated email from the ASF dual-hosted git repository.
vbalaji pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 66893bf fix spark-shell add jar problem
66893bf is described below
commit 66893bfef21ef4644536688595f685f4e1a95051
Author: Thinking <[email protected]>
AuthorDate: Sun Jun 2 21:25:21 2019 +0800
fix spark-shell add jar problem
jira link https://issues.apache.org/jira/browse/HUDI-101
issue link
https://github.com/apache/incubator-hudi/issues/516#issue-386048519
when using spark-shell with hoodie save data like :
```
./spark-shell --master yarn --jars
/home/hdfs/software/spark/hoodie/hoodie-spark-bundle-0.4.8-SNAPSHOT.jar --conf
spark.sql.hive.convertMetastoreParquet=false --packages
com.databricks:spark-avro_2.11:4.0.0
```
and
```
inputDF.write.format("com.uber.hoodie")
.option("hoodie.insert.shuffle.parallelism", "1") // any hoodie
client config can be passed like this
.option("hoodie.upsert.shuffle.parallelism", "1") // full list in
HoodieWriteConfig & its package
.option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY,
HoodieTableType.COPY_ON_WRITE.name())
.option(DataSourceWriteOptions.OPERATION_OPT_KEY,
DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL) // insert
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "_row_key")
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY,
"partition")
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY,
"extend_deal_date")
.option(HoodieWriteConfig.TABLE_NAME, "c_upload_code")
.mode(SaveMode.Overwrite)
.save("/tmp/test/hoodie")
```
It also report error `Invalid signature file digest for Manifest main
attributes`. Need to scan all infected dependency.
---
packaging/hoodie-spark-bundle/pom.xml | 1 +
1 file changed, 1 insertion(+)
diff --git a/packaging/hoodie-spark-bundle/pom.xml
b/packaging/hoodie-spark-bundle/pom.xml
index 023e480..906c323 100644
--- a/packaging/hoodie-spark-bundle/pom.xml
+++ b/packaging/hoodie-spark-bundle/pom.xml
@@ -179,6 +179,7 @@
</artifactSet>
<filters>
<filter>
+ <artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
