----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60049/#review177738 -----------------------------------------------------------
Ship it! Ship It! - Vitalyi Brodetskyi On Червень 13, 2017, 1:57 після полудня, Andrew Onischuk wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60049/ > ----------------------------------------------------------- > > (Updated Червень 13, 2017, 1:57 після полудня) > > > Review request for Ambari and Vitalyi Brodetskyi. > > > Bugs: AMBARI-21239 > https://issues.apache.org/jira/browse/AMBARI-21239 > > > Repository: ambari > > > Description > ------- > > I had a running Ambari instance on version 2.5.1.0-105. Unfortunately i lost > the database VM storing the metadata after the cluster had been runing for > almost 2 weeks without issues. What I observed is that the instance in a few > hours got really slow. Upon investigation found hundreds of Beeline processes > like this: > > > > ambari-+ 3439770 3439769 0 06:19 ? 00:00:00 -bash -c export > PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' > ; ! beeline -u > 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' > -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' > ambari-+ 3439814 3439770 0 06:19 ? 00:00:42 > /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 > -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 > -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa > -Dhadoop.root.logger=INFO,console > -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native > -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true > -Xmx1024m -Xmx1024m > -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties > -Dlog4j.configuration=beeline-log4j.properties > -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar > /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar > org.apache.hive.beeline.BeeLine -u > jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:1050 0/;transportMode=binary -e > ambari-+ 3439816 3439770 0 06:19 ? 00:00:00 awk {print} > ambari-+ 3439817 3439770 0 06:19 ? 00:00:00 grep -i -e Connection > refused -e Invalid URL > ambari-+ 3440645 3440644 0 06:22 ? 00:00:00 -bash -c export > PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' > ; ! beeline -u > 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' > -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' > ambari-+ 3440685 3440645 0 06:22 ? 00:00:42 > /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 > -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 > -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa > -Dhadoop.root.logger=INFO,console > -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native > -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true > -Xmx1024m -Xmx1024m > -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties > -Dlog4j.configuration=beeline-log4j.properties > -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar > /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar > org.apache.hive.beeline.BeeLine -u > jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:1050 0/;transportMode=binary -e > ambari-+ 3440686 3440645 0 06:22 ? 00:00:00 awk {print} > ambari-+ 3440687 3440645 0 06:22 ? 00:00:00 grep -i -e Connection > refused -e Invalid URL > ambari-+ 3441185 3441184 0 06:25 ? 00:00:00 -bash -c export > PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' > ; ! beeline -u > 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' > -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' > ambari-+ 3441224 3441185 0 06:25 ? 00:00:46 > /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 > -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 > -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa > -Dhadoop.root.logger=INFO,console > -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native > -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true > -Xmx1024m -Xmx1024m > -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties > -Dlog4j.configuration=beeline-log4j.properties > -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar > /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar > org.apache.hive.beeline.BeeLine -u > jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:1050 0/;transportMode=binary -e > > > Upon taking a count of the processes with org.apache.hive.beeline.BeeLine in > them and owned by ambari found 309 processes. > The cluster is accessible at 172.27.15.4, unfortunately the Ambari UI isn't > accessible as the DB host serving the metadata went down. > > > Diffs > ----- > > > ambari-common/src/main/python/resource_management/libraries/functions/hive_check.py > 92d16a2 > ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_service_check.py > 52da83a > > > Diff: https://reviews.apache.org/r/60049/diff/1/ > > > Testing > ------- > > mvn clean test > > > Thanks, > > Andrew Onischuk > >
