umehrot2 commented on issue #961: [HUDI-306] Support Glue catalog and other hive metastore implementations URL: https://github.com/apache/incubator-hudi/pull/961#issuecomment-549577195 @vinothchandar @bvaradar I would need your input here. Finally the tests got past the `being stuck` issue, but now the integration test fails with: ``` ###### Stderr ####### Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.get(Lorg/apache/hudi/org/apache/hadoop_hive/conf/HiveConf;)Lorg/apache/hadoop/hive/ql/metadata/Hive; at org.apache.hudi.hive.HoodieHiveClient.<init>(HoodieHiveClient.java:111) at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:60) at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncHive(DeltaSync.java:443) at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:385) at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:227) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:121) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:298) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ``` As you can see from the stack trace, and I didn't know this until now that even `hudi-utilities-bundle` shades Hive. I had removed the shading of Hive from `hudi-spark-bundle` to unblock this PR, but not from `hudi-utilities-bundle` and thats why this issue occurs through DeltaStreamer which uses shaded Hive. The reason for this happening is I have added `Hive.get` in this code, and `Hive` class exists in `hive-exec` which has not been shaded in Hudi until now, and ideally should not be because its a fat jar itself. But `HiveConf` is shaded and relocated. I think we need to take a larger call here, whether its fine to remove Hive shading from `hudi-utilities-bundle` or not. And with this change of introducing `Hive.get` even if Hive is shaded it will still break, because `hive-exec` is not being shaded while other modules are being shaded. So it will cause this `NoSuchMethodError`. Need your suggestions on whether we should completely move out of shading Hive.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
