[ 
https://issues.apache.org/jira/browse/AMBARI-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Onischuk updated AMBARI-21239:
-------------------------------------
    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Committed to trunk and branch-2.5

> Ambari leaves a lot of processes from Hive alert on the machine
> ---------------------------------------------------------------
>
>                 Key: AMBARI-21239
>                 URL: https://issues.apache.org/jira/browse/AMBARI-21239
>             Project: Ambari
>          Issue Type: Bug
>            Reporter: Andrew Onischuk
>            Assignee: Andrew Onischuk
>             Fix For: 2.5.2
>
>         Attachments: AMBARI-21239.patch
>
>
> I had a running Ambari instance on version 2.5.1.0-105. Unfortunately i lost
> the database VM storing the metadata after the cluster had been runing for
> almost 2 weeks without issues. What I observed is that the instance in a few
> hours got really slow. Upon investigation found hundreds of Beeline processes
> like this:
>     
>     
>     ambari-+ 3439770 3439769  0 06:19 ?        00:00:00 -bash -c export  
> PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
>  ; ! beeline -u 
> 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
>   -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
>     ambari-+ 3439814 3439770  0 06:19 ?        00:00:42 
> /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 
> -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 
> -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log 
> -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa 
> -Dhadoop.root.logger=INFO,console 
> -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
>  -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
> -Xmx1024m -Xmx1024m 
> -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties
>  -Dlog4j.configuration=beeline-log4j.properties 
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar 
> /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar 
> org.apache.hive.beeline.BeeLine -u 
> jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary
>  -e
>     ambari-+ 3439816 3439770  0 06:19 ?        00:00:00 awk {print}
>     ambari-+ 3439817 3439770  0 06:19 ?        00:00:00 grep -i -e Connection 
> refused -e Invalid URL
>     ambari-+ 3440645 3440644  0 06:22 ?        00:00:00 -bash -c export  
> PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
>  ; ! beeline -u 
> 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
>   -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
>     ambari-+ 3440685 3440645  0 06:22 ?        00:00:42 
> /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 
> -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 
> -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log 
> -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa 
> -Dhadoop.root.logger=INFO,console 
> -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
>  -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
> -Xmx1024m -Xmx1024m 
> -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties
>  -Dlog4j.configuration=beeline-log4j.properties 
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar 
> /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar 
> org.apache.hive.beeline.BeeLine -u 
> jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary
>  -e
>     ambari-+ 3440686 3440645  0 06:22 ?        00:00:00 awk {print}
>     ambari-+ 3440687 3440645  0 06:22 ?        00:00:00 grep -i -e Connection 
> refused -e Invalid URL
>     ambari-+ 3441185 3441184  0 06:25 ?        00:00:00 -bash -c export  
> PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
>  ; ! beeline -u 
> 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary'
>   -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
>     ambari-+ 3441224 3441185  0 06:25 ?        00:00:46 
> /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 
> -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 
> -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log 
> -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa 
> -Dhadoop.root.logger=INFO,console 
> -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
>  -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
> -Xmx1024m -Xmx1024m 
> -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties
>  -Dlog4j.configuration=beeline-log4j.properties 
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar 
> /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar 
> org.apache.hive.beeline.BeeLine -u 
> jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary
>  -e
>     
> Upon taking a count of the processes with org.apache.hive.beeline.BeeLine in
> them and owned by ambari found 309 processes.  
> The cluster is accessible at 172.27.15.4, unfortunately the Ambari UI isn't
> accessible as the DB host serving the metadata went down.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to