lyogev commented on issue #625: fix(spark): fix shading for spark bundle to 
include all hive's libraries unshaded
URL: https://github.com/apache/incubator-hudi/pull/625#issuecomment-478719303
 
 
   @n3nash hi, so you can check out this branch here:
   https://github.com/YotpoLtd/metorikku/pull/164
   
   Or more specifically the docker compose I used for my E2E spark test:
   https://github.com/YotpoLtd/metorikku/blob/hudi/e2e/hudi/docker-compose.yml
   
   The way we're running our project is via spark-submit.
   I tried adding your original hoodie-spark-bundle by using the 
```--packages``` parameter and the ```--jars``` parameter.
   
   But I had trouble with both.
   In some cases I was getting:
   ```
   java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
hoodie-file:... does not have a open stream. Cannot get the bytes written on 
the stream
   ```
   
   Once I got passed that, when it reached the hive-sync stage I got:
   ```
   java.lang.ClassNotFoundException: 
com.uber.hoodie.org.apache.hadoop_hive.conf.HiveConf
   ```
   
   Upon looking at the bundle I saw that hive wasn't actually bundled at all 
(since it inherited ```provided``` scoping from the parent).
   I changed the scoping to ```compile```  but then I got:
   ```
   java.lang.ClassCastException: 
org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore cannot 
be cast to 
com.uber.hoodie.org.apache.hadoop_hive.metastore.PartitionExpressionProxy
   ```
   
   So in the end I had to both bundle hive into the jar and also make sure it's 
not shaded, since something internal in hive was not handling the shading well.
   
   There is a big chance I'm missing something here, but my end goal is to 
provide an easy access to **Hudi with Metorikku**  by only adding a single JAR 
(2 is also OK :) ) to the spark-submit.
   
   Makes sense?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to