lyogev commented on issue #625: fix(spark): fix shading for spark bundle to include all hive's libraries unshaded URL: https://github.com/apache/incubator-hudi/pull/625#issuecomment-478719303 @n3nash hi, so you can check out this branch here: https://github.com/YotpoLtd/metorikku/pull/164 Or more specifically the docker compose I used for my E2E spark test: https://github.com/YotpoLtd/metorikku/blob/hudi/e2e/hudi/docker-compose.yml The way we're running our project is via spark-submit. I tried adding your original hoodie-spark-bundle by using the ```--packages``` parameter and the ```--jars``` parameter. But I had trouble with both. In some cases I was getting: ``` java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: hoodie-file:... does not have a open stream. Cannot get the bytes written on the stream ``` Once I got passed that, when it reached the hive-sync stage I got: ``` java.lang.ClassNotFoundException: com.uber.hoodie.org.apache.hadoop_hive.conf.HiveConf ``` Upon looking at the bundle I saw that hive wasn't actually bundled at all (since it inherited ```provided``` scoping from the parent). I changed the scoping to ```compile``` but then I got: ``` java.lang.ClassCastException: org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore cannot be cast to com.uber.hoodie.org.apache.hadoop_hive.metastore.PartitionExpressionProxy ``` So in the end I had to both bundle hive into the jar and also make sure it's not shaded, since something internal in hive was not handling the shading well. There is a big chance I'm missing something here, but my end goal is to provide an easy access to **Hudi with Metorikku** by only adding a single JAR (2 is also OK :) ) to the spark-submit. Makes sense?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
