bvaradar edited a comment on issue #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types URL: https://github.com/apache/incubator-hudi/pull/1005#issuecomment-555765815 Thanks @n3nash for your thoughts. @umehrot2 : If it is possible to achieve spark 2.4 upgrade cleanly without moving to spark-hive version, it makes sense to me to retain native hive version. I think it is better to not get locked down on spark-version of hive. As we are using custom code (non-spark) to do hive syncing, theoretically speaking - we may run into some hive issues which would need upgrade but as the issue is not seen in spark, they may be unwilling to patch their hive jars. we can use spark-hive as a last resort if we cannot upgrade to Spark 2.4 any other way :) In that spirit, To your concern related to transitive dependencies in hudi-spark module - As maven honors dependency ordering, can we list hive-exec (with classifier as "core") in the dependency section before hive-service and add exclusions in the dependency section for hive-service to exclude hive-exec. **I am not sure if this would work but don't have time to try this out myself.** Something along the lines of : ``` + <dependency> + <groupId>${hive.groupid}</groupId> + <artifactId>hive-exec</artifactId> + <version>${hive.version}</version> + <classifier>core</classifier> + </dependency> + <dependency> <groupId>${hive.groupid}</groupId> <artifactId>hive-service</artifactId> <version>${hive.version}</version> + <exclusions> + <exclusion> + <groupId>${hive.groupid}</groupId> + <artifactId>hive-exec</artifactId> + </exclusion> + </exclusions> </dependency> <dependency> ``` @umehrot2 : If we cannot make it to work any other way, I am ok with using spark-hive.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
