[
https://issues.apache.org/jira/browse/RANGER-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sekiro updated RANGER-3198:
---------------------------
Environment:
Ranger Version: 2.1
Atlas Version: 2.1
Hive Version: 3.1
was:
Ranger Version: 2.1
Altas Version: 2.1
Hive Version: 3.1
> Ranger and Atlas are incompatible on Hive 3.1
> ---------------------------------------------
>
> Key: RANGER-3198
> URL: https://issues.apache.org/jira/browse/RANGER-3198
> Project: Ranger
> Issue Type: Bug
> Components: admin, audit, plugins, Ranger
> Affects Versions: 2.1.0
> Environment: Ranger Version: 2.1
> Atlas Version: 2.1
> Hive Version: 3.1
> Reporter: Sekiro
> Priority: Critical
>
> I configured Ranger and Atlas in Hive , the Hive service started
> successfully, but when I excute DDL SQL such as "Create Table", it seems that
> the Atlas kakfa can not receive any messages.
> Then I removed Ranger configurtion in Hive , the Atlas kakfa can receive
> messages.
>
> Although I granted all database permissions to Hive or Hadoop user in
> Ranger,but nothing help.
> I found some logs in Hive ,it seems that Atlas Hook was created successfully.
> This is "including ranger configurtion" logs , When the statement finishes,
> Kafka's log is not printed :
> 2021-03-08T08:44:10,510 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: ql.Driver
> (Driver.java:compile(554)) - Compiling
> command(queryId=hive_20210308084410_0089cdc7-45b6-46b3-9921-05c71e950cf4):
> USE `default`
> 2021-03-08T08:44:10,572 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:get(119)) - Looking for
> atlas-application.properties in classpath
> 2021-03-08T08:44:10,572 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:get(132)) - Loading
> atlas-application.propertiesfrom
> file:/etc/hive/conf.dist/atlas-application.properties
> 2021-03-08T08:44:10,591 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefaults(284)) - Using graphdb backend 'janus'
> 2021-03-08T08:44:10,591 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefaults(295)) - Using storage backend 'hbase2'
> 2021-03-08T08:44:10,591 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefaults(306)) - Using index backend 'solr'
> 2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefaults(310)) - Atlas is running in MODE:
> PROD.
> 2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefaults(316)) - Setting solr-wait-searcher
> property 'true'
> 2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefaults(320)) - Setting index.search.map-name
> property 'false'
> 2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefault(334)) - Property (set to default)
> atlas.graph.cache.db-cache = true
> 2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefault(334)) - Property (set to default)
> atlas.graph.cache.db-cache-clean-wait = 20
> 2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefault(334)) - Property (set to default)
> atlas.graph.cache.db-cache-size = 0.5
> 2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefault(334)) - Property (set to default)
> atlas.graph.cache.tx-cache-size = 15000
> 2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties
> (ApplicationProperties.java:setDefault(334)) - Property (set to default)
> atlas.graph.cache.tx-dirty-size = 120
> 2021-03-08T08:44:10,613 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: kafka.KafkaNotification
> (KafkaNotification.java:<init>(105)) - ==> KafkaNotification()
> 2021-03-08T08:44:10,616 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: security.SecurityUtil
> (SecurityUtil.java:getPassword(48)) - Attempting to retrieve password for key
> truststore.password from configured credential provider path
> jceks://file/etc/hive/conf/atlas.jceks
> 2021-03-08T08:44:10,720 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: kafka.KafkaNotification
> (KafkaNotification.java:<init>(137)) - <== KafkaNotification()
> 2021-03-08T08:44:10,734 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc
> HiveServer2-Handler-Pool: Thread-100([])]: hook.AtlasHook
> (AtlasHook.java:<clinit>(147)) - Created Atlas Hook
>
> And this is "removed ranger config" logs ,you can see that at the end of the
> logs, Kafka successfully created and sent the message .
> 2021-03-08T09:33:01,560 INFO [aaff2ff0-f790-4ec3-8add-b4b84d235ba6
> HiveServer2-Handler-Pool: Thread-86([])]: conf.HiveConf
> (HiveConf.java:getLogIdVar(5070)) - Using the default value passed in for log
> id: aaff2ff0-f790-4ec3-8add-b4b84d235ba6
> 2021-03-08T09:33:01,560 INFO [aaff2ff0-f790-4ec3-8add-b4b84d235ba6
> HiveServer2-Handler-Pool: Thread-86([])]: session.SessionState
> (SessionState.java:resetThreadName(452)) - Resetting thread name to
> HiveServer2-Handler-Pool: Thread-86
> 2021-03-08T09:33:01,747 INFO [HiveServer2-Background-Pool: Thread-84([])]:
> metadata.Hive (Hive.java:logDumpPhase(4669)) - Dumping metastore api call
> timing information for : execution phase
> 2021-03-08T09:33:01,747 INFO [HiveServer2-Background-Pool: Thread-84([])]:
> metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(4657)) - Total time spent
> in this metastore function was greater than 1000ms : dropTable_(String,
> String, boolean, boolean, boolean, )=3864
> 2021-03-08T09:33:01,748 INFO [HiveServer2-Background-Pool: Thread-84([])]:
> ql.Driver (Driver.java:execute(2531)) - Completed executing
> command(queryId=hive_20210308093257_3cb85365-3a9f-4984-aa29-1bc2c6ac7926);
> Timetaken: 3.913 seconds
> 2021-03-08T09:33:01,748 INFO [HiveServer2-Background-Pool: Thread-84([])]:
> ql.Driver (SessionState.java:printInfo(1227)) - OK
> 2021-03-08T09:33:01,748 INFO [HiveServer2-Background-Pool: Thread-84([])]:
> ql.Driver (Driver.java:checkConcurrency(285)) - Concurrency mode is disabled,
> not creating a lock manager
> 2021-03-08T09:33:01,827 INFO [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logAll(279)) - ProducerConfig values:
> acks = 1
> batch.size = 16384
> bootstrap.servers = [172.30.198.23:9092]
> buffer.memory = 33554432
> client.id =
> compression.type = none
> connections.max.idle.ms = 540000
> enable.idempotence = false
> interceptor.classes = []
> key.serializer = class org.apache.kafka.common.serialization.StringSerializer
> linger.ms = 0
> max.block.ms = 60000
> max.in.flight.requests.per.connection = 5
> max.request.size = 1048576
> metadata.max.age.ms = 300000
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 30000
> partitioner.class = class
> org.apache.kafka.clients.producer.internals.DefaultPartitioner
> receive.buffer.bytes = 32768
> reconnect.backoff.max.ms = 1000
> reconnect.backoff.ms = 50
> request.timeout.ms = 30000
> retries = 0
> retry.backoff.ms = 100
> sasl.client.callback.handler.class = null
> sasl.jaas.config = null
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.min.time.before.relogin = 60000
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> sasl.kerberos.ticket.renew.window.factor = 0.8
> sasl.login.callback.handler.class = null
> sasl.login.class = null
> sasl.login.refresh.buffer.seconds = 300
> sasl.login.refresh.min.period.seconds = 60
> sasl.login.refresh.window.factor = 0.8
> sasl.login.refresh.window.jitter = 0.05
> sasl.mechanism = GSSAPI
> security.protocol = PLAINTEXT
> send.buffer.bytes = 131072
> ssl.cipher.suites = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.endpoint.identification.algorithm = https
> ssl.key.password = null
> ssl.keymanager.algorithm = SunX509
> ssl.keystore.location = null
> ssl.keystore.password = null
> ssl.keystore.type = JKS
> ssl.protocol = TLS
> ssl.provider = null
> ssl.secure.random.implementation = null
> ssl.trustmanager.algorithm = PKIX
> ssl.truststore.location = null
> ssl.truststore.password = [hidden]
> ssl.truststore.type = JKS
> transaction.timeout.ms = 60000
> transactional.id = null
> value.serializer = class
> org.apache.kafka.common.serialization.StringSerializer
> 2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration 'key.deserializer'
> was supplied but isn't a known config.
> 2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration 'value.deserializer'
> was supplied but isn't a known config.
> 2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration 'hook.group.id' was
> supplied but isn't a known config.
> 2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration 'max.poll.records'
> was supplied but isn't a known config.
> 2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration 'data' was supplied
> but isn't a known config.
> 2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration
> 'auto.commit.interval.ms' was supplied but isn't a known config.
> 2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration
> 'offsets.topic.replication.factor' was supplied but isn't a known config.
> 2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration
> 'zookeeper.connection.timeout.ms' was supplied but isn't a known config.
> 2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration
> 'zookeeper.session.timeout.ms' was supplied but isn't a known config.
> 2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration 'enable.auto.commit'
> was supplied but isn't a known config.
> 2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration 'zookeeper.connect'
> was supplied but isn't a known config.
> 2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration 'poll.timeout.ms'
> was supplied but isn't a known config.
> 2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration
> 'zookeeper.sync.time.ms' was supplied but isn't a known config.
> 2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration 'session.timeout.ms'
> was supplied but isn't a known config.
> 2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig
> (AbstractConfig.java:logUnused(287)) - The configuration 'auto.offset.reset'
> was supplied but isn't a known config.
> 2021-03-08T09:33:01,874 INFO [Atlas Notifier 0([])]: utils.AppInfoParser
> (AppInfoParser.java:<init>(109)) - Kafka version : 2.0.0
> 2021-03-08T09:33:01,874 INFO [Atlas Notifier 0([])]: utils.AppInfoParser
> (AppInfoParser.java:<init>(110)) - Kafka commitId : 3402a8361b734732
> 2021-03-08T09:33:01,946 INFO [kafka-producer-network-thread |
> producer-1([])]: clients.Metadata (Metadata.java:update(273)) - Cluster ID:
> ytZ5Kgb9RXOOp0KEZL6iXw
> 2021-03-08T09:33:02,731 INFO [HiveServer2-Handler-Pool: Thread-77([])]:
> conf.HiveConf (HiveConf.java:getLogIdVar(5070)) - Using the default value
> passed in for log id: aaff2ff0-f790-4ec3-8add-b4b84d235ba6
> 2021-03-08T09:33:02,731 INFO [HiveServer2-Handler-Pool: Thread-77([])]:
> session.SessionState (SessionState.java:updateThreadName(441)) - Updating
> thread name to aaff2ff0-f790-4ec3-8add-b4b84d235ba6 HiveServer2-Handler-Pool:
> Thread-77
>
>
> Can someone give me some advice? Thanks very much!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
