[ https://issues.apache.org/jira/browse/AMBARI-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Onischuk updated AMBARI-21239: ------------------------------------- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk and branch-2.5 > Ambari leaves a lot of processes from Hive alert on the machine > --------------------------------------------------------------- > > Key: AMBARI-21239 > URL: https://issues.apache.org/jira/browse/AMBARI-21239 > Project: Ambari > Issue Type: Bug > Reporter: Andrew Onischuk > Assignee: Andrew Onischuk > Fix For: 2.5.2 > > Attachments: AMBARI-21239.patch > > > I had a running Ambari instance on version 2.5.1.0-105. Unfortunately i lost > the database VM storing the metadata after the cluster had been runing for > almost 2 weeks without issues. What I observed is that the instance in a few > hours got really slow. Upon investigation found hundreds of Beeline processes > like this: > > > ambari-+ 3439770 3439769 0 06:19 ? 00:00:00 -bash -c export > PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' > ; ! beeline -u > 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' > -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' > ambari-+ 3439814 3439770 0 06:19 ? 00:00:42 > /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 > -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 > -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa > -Dhadoop.root.logger=INFO,console > -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native > -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true > -Xmx1024m -Xmx1024m > -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties > -Dlog4j.configuration=beeline-log4j.properties > -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar > /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar > org.apache.hive.beeline.BeeLine -u > jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary > -e > ambari-+ 3439816 3439770 0 06:19 ? 00:00:00 awk {print} > ambari-+ 3439817 3439770 0 06:19 ? 00:00:00 grep -i -e Connection > refused -e Invalid URL > ambari-+ 3440645 3440644 0 06:22 ? 00:00:00 -bash -c export > PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' > ; ! beeline -u > 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' > -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' > ambari-+ 3440685 3440645 0 06:22 ? 00:00:42 > /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 > -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 > -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa > -Dhadoop.root.logger=INFO,console > -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native > -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true > -Xmx1024m -Xmx1024m > -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties > -Dlog4j.configuration=beeline-log4j.properties > -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar > /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar > org.apache.hive.beeline.BeeLine -u > jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary > -e > ambari-+ 3440686 3440645 0 06:22 ? 00:00:00 awk {print} > ambari-+ 3440687 3440645 0 06:22 ? 00:00:00 grep -i -e Connection > refused -e Invalid URL > ambari-+ 3441185 3441184 0 06:25 ? 00:00:00 -bash -c export > PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' > ; ! beeline -u > 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' > -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' > ambari-+ 3441224 3441185 0 06:25 ? 00:00:46 > /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 > -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 > -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa > -Dhadoop.root.logger=INFO,console > -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native > -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true > -Xmx1024m -Xmx1024m > -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties > -Dlog4j.configuration=beeline-log4j.properties > -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar > /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar > org.apache.hive.beeline.BeeLine -u > jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary > -e > > Upon taking a count of the processes with org.apache.hive.beeline.BeeLine in > them and owned by ambari found 309 processes. > The cluster is accessible at 172.27.15.4, unfortunately the Ambari UI isn't > accessible as the DB host serving the metadata went down. -- This message was sent by Atlassian JIRA (v6.4.14#64029)