[
https://issues.apache.org/jira/browse/ATLAS-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936552#comment-16936552
]
cdmikechen edited comment on ATLAS-3172 at 9/24/19 8:34 AM:
------------------------------------------------------------
I use Hive2.3.3 to run `import-hive.sh`, It also has some errors.
I checked jackson jars and found that Hive use jackson-2.6.5 and ranger-1.2.0
use 2.9.9. So that there are some conflicts between jar packages.
{code:sh}
[atlas@bdcluster1 hook-bin]$ sh import-hive.sh
Using Hive configuration directory [/opt/soft/hive/conf]
Log file for import is /home/atlas/software/atlas/logs/import-hive.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/opt/soft/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/hdfs/software/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2019-09-24T15:56:17,631 INFO [main] org.apache.atlas.ApplicationProperties -
Looking for atlas-application.properties in classpath
2019-09-24T15:56:17,633 INFO [main] org.apache.atlas.ApplicationProperties -
Loading atlas-application.properties from
file:/home/opt/soft/apache-hive-2.3.3-bin/conf/atlas-application.properties
2019-09-24T15:56:17,669 INFO [main] org.apache.atlas.ApplicationProperties -
Property (set to default) atlas.graph.cache.db-cache = true
2019-09-24T15:56:17,669 INFO [main] org.apache.atlas.ApplicationProperties -
Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20
2019-09-24T15:56:17,669 INFO [main] org.apache.atlas.ApplicationProperties -
Property (set to default) atlas.graph.cache.db-cache-size = 0.5
2019-09-24T15:56:17,669 INFO [main] org.apache.atlas.ApplicationProperties -
Property (set to default) atlas.graph.cache.tx-cache-size = 15000
2019-09-24T15:56:17,669 INFO [main] org.apache.atlas.ApplicationProperties -
Property (set to default) atlas.graph.cache.tx-dirty-size = 120
Enter username for atlas :- admin
Enter password for atlas :-
2019-09-24T15:56:25,071 INFO [main] org.apache.atlas.AtlasBaseClient - Client
has only one service URL, will use that for all actions:
http://192.168.11.60:21000
2019-09-24T15:56:25,093 INFO [main] org.apache.hadoop.hive.conf.HiveConf -
Found configuration file
file:/home/opt/soft/apache-hive-2.3.3-bin/conf/hive-site.xml
2019-09-24T15:56:25,649 WARN [main] org.apache.hadoop.hive.conf.HiveConf -
HiveConf of name hive.server2.custom.authentication.service.username does not
exist
2019-09-24T15:56:25,649 WARN [main] org.apache.hadoop.hive.conf.HiveConf -
HiveConf of name hive.server2.custom.authentication.service.password does not
exist
2019-09-24T15:56:25,650 WARN [main] org.apache.hadoop.hive.conf.HiveConf -
HiveConf of name hive.server2.custom.authentication.cas.url does not exist
2019-09-24T15:56:25,650 WARN [main] org.apache.hadoop.hive.conf.HiveConf -
HiveConf of name hive.server2.custom.authentication.service.jdbcurl does not
exist
2019-09-24T15:56:26,278 WARN [main] org.apache.hadoop.util.NativeCodeLoader -
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2019-09-24T15:56:26,314 INFO [main] hive.metastore - Trying to connect to
metastore with URI thrift://bdcluster1:10001
2019-09-24T15:56:26,335 INFO [main] hive.metastore - Opened a connection to
metastore, current connections: 1
2019-09-24T15:56:26,369 INFO [main] hive.metastore - Connected to metastore.
2019-09-24T15:56:26,522 INFO [main]
org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing Hive metadata
2019-09-24T15:56:26,531 INFO [main]
org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 7 databases
2019-09-24T15:56:26,576 INFO [main] org.apache.atlas.AtlasBaseClient -
method=GET path=api/atlas/v2/entity/uniqueAttribute/type/
contentType=application/json; charset=UTF-8 accept=application/json status=404
Exception in thread "main" java.lang.NoSuchMethodError:
com.fasterxml.jackson.databind.util.BeanUtil.okNameForGetter(Lcom/fasterxml/jackson/databind/introspect/AnnotatedMethod;)Ljava/lang/String;
at
com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.findNameForSerialization(JaxbAnnotationIntrospector.java:824)
at
com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findNameForSerialization(AnnotationIntrospectorPair.java:583)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addGetterMethod(POJOPropertiesCollector.java:577)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:533)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:309)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueAccessor(POJOPropertiesCollector.java:196)
at
com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueAccessor(BasicBeanDescription.java:252)
at
com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:346)
at
com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:216)
at
com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)
at
com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1388)
at
com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1336)
at
com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:510)
at
com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:713)
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)
at
com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)
at
com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913)
at
com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:648)
at
com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204)
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
at
com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at
com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
at
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:382)
at
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:353)
at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:229)
at org.apache.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:320)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:450)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:402)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:281)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:251)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:168)
Failed to import Hive Meta Data!!!
{code}
ranger will use `jackson-module-jaxb-annotations-2.9.9.jar` but doesn't find,
so it use Hive's `jackson-module-jaxb-annotations-2.4.6.jar` .
was (Author: chenxiang):
I use Hive2.3.3 to run `import-hive.sh`, It also has some errors.
I checked jackson jars and found that Hive use jackson-2.6.5 and ranger-1.2.0
use 2.9.9. So that there are some conflicts between jar packages.
```
[atlas@bdcluster1 hook-bin]$ sh import-hive.sh
Using Hive configuration directory [/opt/soft/hive/conf]
Log file for import is /home/atlas/software/atlas/logs/import-hive.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/opt/soft/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/hdfs/software/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2019-09-24T15:56:17,631 INFO [main] org.apache.atlas.ApplicationProperties -
Looking for atlas-application.properties in classpath
2019-09-24T15:56:17,633 INFO [main] org.apache.atlas.ApplicationProperties -
Loading atlas-application.properties from
file:/home/opt/soft/apache-hive-2.3.3-bin/conf/atlas-application.properties
2019-09-24T15:56:17,669 INFO [main] org.apache.atlas.ApplicationProperties -
Property (set to default) atlas.graph.cache.db-cache = true
2019-09-24T15:56:17,669 INFO [main] org.apache.atlas.ApplicationProperties -
Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20
2019-09-24T15:56:17,669 INFO [main] org.apache.atlas.ApplicationProperties -
Property (set to default) atlas.graph.cache.db-cache-size = 0.5
2019-09-24T15:56:17,669 INFO [main] org.apache.atlas.ApplicationProperties -
Property (set to default) atlas.graph.cache.tx-cache-size = 15000
2019-09-24T15:56:17,669 INFO [main] org.apache.atlas.ApplicationProperties -
Property (set to default) atlas.graph.cache.tx-dirty-size = 120
Enter username for atlas :- admin
Enter password for atlas :-
2019-09-24T15:56:25,071 INFO [main] org.apache.atlas.AtlasBaseClient - Client
has only one service URL, will use that for all actions:
http://192.168.11.60:21000
2019-09-24T15:56:25,093 INFO [main] org.apache.hadoop.hive.conf.HiveConf -
Found configuration file
file:/home/opt/soft/apache-hive-2.3.3-bin/conf/hive-site.xml
2019-09-24T15:56:25,649 WARN [main] org.apache.hadoop.hive.conf.HiveConf -
HiveConf of name hive.server2.custom.authentication.service.username does not
exist
2019-09-24T15:56:25,649 WARN [main] org.apache.hadoop.hive.conf.HiveConf -
HiveConf of name hive.server2.custom.authentication.service.password does not
exist
2019-09-24T15:56:25,650 WARN [main] org.apache.hadoop.hive.conf.HiveConf -
HiveConf of name hive.server2.custom.authentication.cas.url does not exist
2019-09-24T15:56:25,650 WARN [main] org.apache.hadoop.hive.conf.HiveConf -
HiveConf of name hive.server2.custom.authentication.service.jdbcurl does not
exist
2019-09-24T15:56:26,278 WARN [main] org.apache.hadoop.util.NativeCodeLoader -
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2019-09-24T15:56:26,314 INFO [main] hive.metastore - Trying to connect to
metastore with URI thrift://bdcluster1:10001
2019-09-24T15:56:26,335 INFO [main] hive.metastore - Opened a connection to
metastore, current connections: 1
2019-09-24T15:56:26,369 INFO [main] hive.metastore - Connected to metastore.
2019-09-24T15:56:26,522 INFO [main]
org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing Hive metadata
2019-09-24T15:56:26,531 INFO [main]
org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 7 databases
2019-09-24T15:56:26,576 INFO [main] org.apache.atlas.AtlasBaseClient -
method=GET path=api/atlas/v2/entity/uniqueAttribute/type/
contentType=application/json; charset=UTF-8 accept=application/json status=404
Exception in thread "main" java.lang.NoSuchMethodError:
com.fasterxml.jackson.databind.util.BeanUtil.okNameForGetter(Lcom/fasterxml/jackson/databind/introspect/AnnotatedMethod;)Ljava/lang/String;
at
com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.findNameForSerialization(JaxbAnnotationIntrospector.java:824)
at
com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findNameForSerialization(AnnotationIntrospectorPair.java:583)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addGetterMethod(POJOPropertiesCollector.java:577)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:533)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:309)
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueAccessor(POJOPropertiesCollector.java:196)
at
com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueAccessor(BasicBeanDescription.java:252)
at
com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:346)
at
com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:216)
at
com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)
at
com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1388)
at
com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1336)
at
com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:510)
at
com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:713)
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)
at
com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)
at
com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913)
at
com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:648)
at
com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204)
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
at
com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at
com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
at
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:382)
at
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:353)
at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:229)
at org.apache.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:320)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:450)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:402)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:281)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:251)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:168)
Failed to import Hive Meta Data!!!
```
ranger will use `jackson-module-jaxb-annotations-2.9.9.jar` but doesn't find,
so it use Hive's `jackson-module-jaxb-annotations-2.4.6.jar` .
> Atlas Hive Hook lack of Jackson dependency
> ------------------------------------------
>
> Key: ATLAS-3172
> URL: https://issues.apache.org/jira/browse/ATLAS-3172
> Project: Atlas
> Issue Type: Bug
> Reporter: jianxiong zhao
> Priority: Major
>
> After copying Hive Hook to ${ATLAS_HOME}, executing `hook-bin/import-hive.sh`
> always fails with lack of Jackson.XXX dependencies
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
