umehrot2 commented on issue #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types URL: https://github.com/apache/incubator-hudi/pull/1005#issuecomment-555259043 > @umehrot2 : Sorry for the back-and-forth on this. Issue 1 (as mentioned in [#1005 (comment)](https://github.com/apache/incubator-hudi/pull/1005#issuecomment-554089712)) is due to fat jar hive-exec. @n3nash proposed a solution in Uber which wont require moving to spark-hive. Instead of the test dependency : hive-exec, can you try depending on the non-fat version of the jar called : hive-exec-core. Hopefully, we can control parquet/avro versions getting loaded for the tests. @bvaradar That's fine, we should take time and solve the right way. In `hudi-utilities` it is a test dependency, but not in `hudi-spark`. `hudi-spark` depends on `hive-service` which is bringing in `hive-exec` as a transitive dependency. And that is the reason its ending up in tests classpath. We would have to get rid of `hive-service` if we were to do that. Also I don't see any artifact like `hive-exec-core`. Maven does not recognize it. But besides that, when we are depending on runtime Spark's version of Hive, is there any reason why we are wanting to build it with hive 2.x instead ? May be `hudi-hive` in itself since that is an independent module makes sense to build with hive 2.x. But anything running within spark, why are we inclined to building with version of hive not supported by spark ?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
