[
https://issues.apache.org/jira/browse/HUDI-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284866#comment-17284866
]
sivabalan narayanan edited comment on HUDI-1609 at 2/15/21, 5:29 PM:
---------------------------------------------------------------------
hive sync w/ metastore approach fails locally with docker set up.
Tried to follow the docker demo here.
When trying to do the hive sync, running into ClassNotFoundError w/ CommonTree
in antlr.
{code:java}
2021-02-15 17:05:33,033 INFO [main] hive.metastore
(HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore,
current connections: 0 Exception in thread "main"
java.lang.NoClassDefFoundError: org/antlr/runtime/tree/CommonTree at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLs(HoodieHiveClient.java:401)
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLUsingHiveDriver(HoodieHiveClient.java:384)
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:374)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:122) at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:94) at
org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:231) Caused by:
java.lang.ClassNotFoundException: org.antlr.runtime.tree.CommonTree at
java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 11 more
{code}
maven dep tree for antlr-runtime
[https://gist.github.com/nsivabalan/4a308f3a55b1520edee067996b373215]
Looks like, when this jar is transitively fetched via hive-metastore, scope it
chosen as "provided". Eg: *hudi-spark-bundle_2.11*
But if this jar is transitively fetched via "org.antlr:stringtemplate", scope
is chosen as "compile".
But here is the excerpt from spark bundle and utilities bundle
{code:java}
//
[INFO] -------------< org.apache.hudi:hudi-utilities-bundle_2.11 >-------------
[INFO] Building hudi-utilities-bundle_2.11 0.8.0-SNAPSHOT [16/42]
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from uber-artifactory:
http://artifactory.uber.internal:4587/artifactory/repo/org/apache/hudi/hudi-utilities_2.11/0.8.0-SNAPSHOT/maven-metadata.xml
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @
hudi-utilities-bundle_2.11 ---
Downloading from uber-artifactory:
http://artifactory.uber.internal:4587/artifactory/repo/org/apache/hudi/hudi-utilities_2.11/0.8.0-SNAPSHOT/hudi-utilities_2.11-0.8.0-SNAPSHOT.jar
[INFO] org.apache.hudi:hudi-utilities-bundle_2.11:jar:0.8.0-SNAPSHOT
[INFO] \- org.apache.hive:hive-metastore:jar:2.3.1:provided
[INFO] \- org.antlr:antlr-runtime:jar:3.5.2:compile
[INFO] ---------------< org.apache.hudi:hudi-spark-bundle_2.11 >---------------
[INFO] Building hudi-spark-bundle_2.11 0.8.0-SNAPSHOT [24/42]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @
hudi-spark-bundle_2.11 ---
[INFO] org.apache.hudi:hudi-spark-bundle_2.11:jar:0.8.0-SNAPSHOT
[INFO] \- org.apache.hive:hive-metastore:jar:2.3.1:provided
[INFO] \- org.antlr:antlr-runtime:jar:3.5.2:provided
{code}
May be this could the issue.
was (Author: shivnarayan):
hive sync w/ metastore approach fails locally with docker set up.
Tried to follow the docker demo here.
When trying to do the hive sync, running into ClassNotFoundError w/ CommonTree
in antlr.
{code:java}
2021-02-15 17:05:33,033 INFO [main] hive.metastore
(HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore,
current connections: 0 Exception in thread "main"
java.lang.NoClassDefFoundError: org/antlr/runtime/tree/CommonTree at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLs(HoodieHiveClient.java:401)
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLUsingHiveDriver(HoodieHiveClient.java:384)
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:374)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:122) at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:94) at
org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:231) Caused by:
java.lang.ClassNotFoundException: org.antlr.runtime.tree.CommonTree at
java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 11 more
{code}
maven dep tree for antlr-runtime
[https://gist.github.com/nsivabalan/4a308f3a55b1520edee067996b373215]
Looks like, when this jar is transitively fetched via hive-metastore, scope it
chosen as "provided". Eg: *hudi-spark-bundle_2.11*
But if this jar is transitively fetched via "org.antlr:stringtemplate", scope
is chosen as "compile".
May be this could the issue.
> Issues w/ using hive metastore by disabling jdbc
> ------------------------------------------------
>
> Key: HUDI-1609
> URL: https://issues.apache.org/jira/browse/HUDI-1609
> Project: Apache Hudi
> Issue Type: Task
> Components: Hive Integration
> Reporter: sivabalan narayanan
> Priority: Major
> Labels: pull-request-available, sev:critical, user-support-issues
> Fix For: 0.8.0
>
>
> Ref: https://github.com/apache/hudi/issues/1679
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
