-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60049/
-----------------------------------------------------------
Review request for Ambari and Vitalyi Brodetskyi.
Bugs: AMBARI-21239
https://issues.apache.org/jira/browse/AMBARI-21239
Repository: ambari
Description
-------
I had a running Ambari instance on version 2.5.1.0-105. Unfortunately i lost
the database VM storing the metadata after the cluster had been runing for
almost 2 weeks without issues. What I observed is that the instance in a few
hours got really slow. Upon investigation found hundreds of Beeline processes
like this:
ambari-+ 3439770 3439769 0 06:19 ? 00:00:00 -bash -c export
PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
; ! beeline -u
'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
-e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
ambari-+ 3439814 3439770 0 06:19 ? 00:00:42
/usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa
-Dhadoop.root.logger=INFO,console
-Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
-Xmx1024m -Xmx1024m
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties
-Dlog4j.configuration=beeline-log4j.properties
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar
/usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar
org.apache.hive.beeline.BeeLine -u
jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/
;transportMode=binary -e
ambari-+ 3439816 3439770 0 06:19 ? 00:00:00 awk {print}
ambari-+ 3439817 3439770 0 06:19 ? 00:00:00 grep -i -e Connection
refused -e Invalid URL
ambari-+ 3440645 3440644 0 06:22 ? 00:00:00 -bash -c export
PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
; ! beeline -u
'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
-e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
ambari-+ 3440685 3440645 0 06:22 ? 00:00:42
/usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa
-Dhadoop.root.logger=INFO,console
-Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
-Xmx1024m -Xmx1024m
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties
-Dlog4j.configuration=beeline-log4j.properties
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar
/usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar
org.apache.hive.beeline.BeeLine -u
jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/
;transportMode=binary -e
ambari-+ 3440686 3440645 0 06:22 ? 00:00:00 awk {print}
ambari-+ 3440687 3440645 0 06:22 ? 00:00:00 grep -i -e Connection
refused -e Invalid URL
ambari-+ 3441185 3441184 0 06:25 ? 00:00:00 -bash -c export
PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
; ! beeline -u
'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
-e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
ambari-+ 3441224 3441185 0 06:25 ? 00:00:46
/usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa
-Dhadoop.root.logger=INFO,console
-Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
-Xmx1024m -Xmx1024m
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties
-Dlog4j.configuration=beeline-log4j.properties
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar
/usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar
org.apache.hive.beeline.BeeLine -u
jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/
;transportMode=binary -e
Upon taking a count of the processes with org.apache.hive.beeline.BeeLine in
them and owned by ambari found 309 processes.
The cluster is accessible at 172.27.15.4, unfortunately the Ambari UI isn't
accessible as the DB host serving the metadata went down.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/hive_check.py
92d16a2
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_service_check.py
52da83a
Diff: https://reviews.apache.org/r/60049/diff/1/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
