-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46427/
-----------------------------------------------------------
(Updated April 20, 2016, 8:55 p.m.)
Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
Changes
-------
Comments added for parameters.
Bugs: AMBARI-15985
https://issues.apache.org/jira/browse/AMBARI-15985
Repository: ambari
Description
-------
Service Checks for Hive Server Interactive and LLAP
- Checks Port for HSI.
- Issues DB queries to ascertaing liveness of LLAP.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/files/hiveLlapSmoke.sh
PRE-CREATION
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
2414e8b
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/service_check.py
251e71f
Diff: https://reviews.apache.org/r/46427/diff/
Testing
-------
- Tested with HSI installed and not installed.
- new serviece checks added pass.
- Python UT passes.
----------------------------------------------------------------------
Ran 261 tests in 6.542s
OK
----------------------------------------------------------------------
Total run:994
Total errors:0
Total failures:0
OK
Service Check o/p:
=================
2016-04-20 19:30:27,256 - Using hadoop conf dir:
/usr/hdp/current/hadoop-client/conf
2016-04-20 19:30:27,308 - call['ambari-python-wrap /usr/bin/hdp-select status
hive-server2'] {'timeout': 20}
2016-04-20 19:30:27,336 - call returned (0, 'hive-server2 - 2.5.0.0-157')
2016-04-20 19:30:27,341 - Running Hive Server checks
2016-04-20 19:30:27,342 - --------------------------
2016-04-20 19:30:27,342 - Server Address List : ['c6402.ambari.apache.org'],
Port : 10000
2016-04-20 19:30:27,342 - Waiting for the Hive Server to start...
2016-04-20 19:30:27,343 - Execute['! beeline -u
'jdbc:hive2://c6402.ambari.apache.org:10000/;transportMode=binary' -e '' 2>&1|
awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL''] {'path':
['/bin/', '/usr/bin/', '/usr/lib/hive/bin/', '/usr/sbin/'], 'user':
'ambari-qa', 'timeout': 30}
2016-04-20 19:30:30,016 - Successfully connected to c6402.ambari.apache.org on
port 10000
2016-04-20 19:30:30,016 - Successfully stayed connected to 'Hive Server' on
host: c6402.ambari.apache.org and port 10000 after 2.6735098362 seconds
2016-04-20 19:30:30,016 - Running Hive Server2 checks
2016-04-20 19:30:30,016 - --------------------------
2016-04-20 19:30:30,017 - Server Address List : ['c6402.ambari.apache.org'],
Port : 10500
2016-04-20 19:30:30,017 - Waiting for the Hive Server2 to start...
2016-04-20 19:30:30,018 - Execute['! beeline -u
'jdbc:hive2://c6402.ambari.apache.org:10500/;transportMode=binary' -e '' 2>&1|
awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL''] {'path':
['/bin/', '/usr/bin/', '/usr/lib/hive/bin/', '/usr/sbin/'], 'user':
'ambari-qa', 'timeout': 30}
2016-04-20 19:30:34,219 - Successfully connected to c6402.ambari.apache.org on
port 10500
2016-04-20 19:30:34,219 - Successfully stayed connected to 'Hive Server2' on
host: c6402.ambari.apache.org and port 10500 after 4.2018828392 seconds
2016-04-20 19:30:34,219 - Running LLAP checks
2016-04-20 19:30:34,219 - -------------------
2016-04-20 19:30:34,221 - File['/var/lib/ambari-agent/tmp/hiveLlapSmoke.sh']
{'content': StaticFile('hiveLlapSmoke.sh'), 'mode': 0755}
2016-04-20 19:30:34,224 - checked_call['hostid'] {}
2016-04-20 19:30:34,235 - checked_call returned (0, 'a8c06640')
2016-04-20 19:30:34,236 - Execute['env JAVA_HOME=/usr/jdk64/jdk1.8.0_60
/var/lib/ambari-agent/tmp/hiveLlapSmoke.sh /usr/hdp
llap_smoke_ida8c06640_date302016 prepare'] {'logoutput': True, 'try_sleep': 5,
'wait_for_finish': True, 'tries': 1, 'user': 'hive', 'stderr': -1, 'path':
['/usr/sbin', '/usr/local/bin', '/bin', '/usr/bin',
'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin']}
which: no hbase in
(/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/sbin:/usr/local/bin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin)
WARNING: Use "yarn jar" to launch YARN applications.
Java HotSpot(TM) 64-Bit Server VM warning: Using the ParNew young collector
with the Serial old collector is deprecated and will likely be removed in a
future release
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/hdp/2.5.0.0-157/hive2/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/hdp/2.5.0.0-157/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in
jar:file:/usr/hdp/2.5.0.0-157/hive2/lib/hive-common-2.1.0.2.5.0.0-157.jar!/hive-log4j2.properties
OK
Time taken: 21.683 seconds
Query ID = hive_20160420193104_47af1752-fbeb-4d7e-afd3-f680c12a0f5d
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id
application_1461173999701_0012)
Map 1: -/- Reducer 2: 0/2
Map 1: 0/1 Reducer 2: 0/2
Map 1: 0(+1)/1 Reducer 2: 0/2
Map 1: 1/1 Reducer 2: 0/2
Map 1: 1/1 Reducer 2: 0(+1)/2
Map 1: 1/1 Reducer 2: 1(+1)/2
Map 1: 1/1 Reducer 2: 2/2
Status: DAG finished successfully in 4.81 seconds
Query Execution Summary
----------------------------------------------------------------------------------------------
[2K[36;1mOPERATION DURATION
[22;0m----------------------------------------------------------------------------------------------
Compile Query 3.50s
Prepare Plan 0.99s
Submit Plan 1.67s
Start 0.76s
Finish 4.81s
----------------------------------------------------------------------------------------------
Task Execution Summary
----------------------------------------------------------------------------------------------
[2K[36;1m VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
OUTPUT_RECORDS
[22;0m----------------------------------------------------------------------------------------------
Map 1 1082.00 0 0 2
2
Reducer 2 2403.00 0 0 2
0
----------------------------------------------------------------------------------------------
LLAP IO Summary
----------------------------------------------------------------------------------------------
[2K[36;1m VERTICES ROWGROUPS META_HIT META_MISS DATA_HIT DATA_MISS
ALLOCATION USED TOTAL_IO
[22;0m----------------------------------------------------------------------------------------------
Map 1 0 0 0 0B 0B 0B
0B 0.00s
----------------------------------------------------------------------------------------------
Loading data to table default.llap_smoke_ida8c06640_date302016
Table default.llap_smoke_ida8c06640_date302016 stats: [numFiles=2, numRows=2,
totalSize=672, rawDataSize=204]
OK
Time taken: 13.035 seconds
OK
2
Time taken: 2.177 seconds, Fetched: 1 row(s)
2016-04-20 19:31:21,178 - Running HCAT checks
2016-04-20 19:31:21,178 - -------------------
2016-04-20 19:31:21,179 - checked_call['hostid'] {}
2016-04-20 19:31:21,208 - checked_call returned (0, 'a8c06640')
2016-04-20 19:31:21,209 - File['/var/lib/ambari-agent/tmp/hcatSmoke.sh']
{'content': StaticFile('hcatSmoke.sh'), 'mode': 0755}
2016-04-20 19:31:21,224 - Execute['env JAVA_HOME=/usr/jdk64/jdk1.8.0_60
/var/lib/ambari-agent/tmp/hcatSmoke.sh hcatsmokeida8c06640_date312016 prepare
true'] {'logoutput': True, 'path': ['/usr/sbin', '/usr/local/bin', '/bin',
'/usr/bin',
'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin'],
'tries': 3, 'user': 'ambari-qa', 'try_sleep': 5}
WARNING: Use "yarn jar" to launch YARN applications.
OK
t2
t3
Time taken: 5.182 seconds
WARNING: Use "yarn jar" to launch YARN applications.
OK
Time taken: 2.044 seconds
WARNING: Use "yarn jar" to launch YARN applications.
OK
Time taken: 2.817 seconds
2016-04-20 19:31:58,584 - ExecuteHadoop['fs -test -e
/apps/hive/warehouse/hcatsmokeida8c06640_date312016'] {'logoutput': True,
'bin_dir':
'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin',
'user': 'hdfs', 'conf_dir': '/usr/hdp/current/hadoop-client/conf'}
2016-04-20 19:31:58,590 - Execute['hadoop --config
/usr/hdp/current/hadoop-client/conf fs -test -e
/apps/hive/warehouse/hcatsmokeida8c06640_date312016'] {'logoutput': True,
'try_sleep': 0, 'environment': {}, 'tries': 1, 'user': 'hdfs', 'path':
['/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin']}
2016-04-20 19:32:01,625 - Execute[' /var/lib/ambari-agent/tmp/hcatSmoke.sh
hcatsmokeida8c06640_date312016 cleanup true'] {'logoutput': True, 'path':
['/usr/sbin', '/usr/local/bin', '/bin', '/usr/bin',
'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin'],
'tries': 3, 'user': 'ambari-qa', 'try_sleep': 5}
WARNING: Use "yarn jar" to launch YARN applications.
OK
Time taken: 2.468 seconds
2016-04-20 19:32:08,730 - Running WEBHCAT checks
2016-04-20 19:32:08,730 - ---------------------
2016-04-20 19:32:08,731 - File['/var/lib/ambari-agent/tmp/templetonSmoke.sh']
{'content': StaticFile('templetonSmoke.sh'), 'mode': 0755}
2016-04-20 19:32:08,738 -
File['/var/lib/ambari-agent/tmp/idtest.ambari-qa.1461180728.73.pig'] {'owner':
'hdfs', 'content': Template('templeton_smoke.pig.j2')}
2016-04-20 19:32:08,739 - Writing
File['/var/lib/ambari-agent/tmp/idtest.ambari-qa.1461180728.73.pig'] because it
doesn't exist
2016-04-20 19:32:08,743 - Changing owner for
/var/lib/ambari-agent/tmp/idtest.ambari-qa.1461180728.73.pig from 0 to hdfs
2016-04-20 19:32:08,744 -
HdfsResource['/tmp/idtest.ambari-qa.1461180728.73.pig'] {'security_enabled':
False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab':
[EMPTY], 'source':
'/var/lib/ambari-agent/tmp/idtest.ambari-qa.1461180728.73.pig', 'dfs_type': '',
'default_fs': 'hdfs://c6402.ambari.apache.org:8020',
'hdfs_resource_ignore_file':
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ...,
'kinit_path_local': 'kinit', 'principal_name': 'missing_principal', 'user':
'hdfs', 'owner': 'ambari-qa', 'hadoop_conf_dir':
'/usr/hdp/current/hadoop-client/conf', 'type': 'file', 'action':
['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse',
u'/mr-history/done', u'/app-logs', u'/tmp']}
2016-04-20 19:32:08,751 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl
-sS -L -w '"'"'%{http_code}'"'"' -X GET
'"'"'http://c6402.ambari.apache.org:50070/webhdfs/v1/tmp/idtest.ambari-qa.1461180728.73.pig?op=GETFILESTATUS&user.name=hdfs'"'"'
1>/tmp/tmpMdR_jN 2>/tmp/tmpJLxau4''] {'logoutput': None, 'quiet': False}
2016-04-20 19:32:08,795 - call returned (0, '')
2016-04-20 19:32:08,796 - Creating new file
/tmp/idtest.ambari-qa.1461180728.73.pig in DFS
2016-04-20 19:32:08,797 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl
-sS -L -w '"'"'%{http_code}'"'"' -X PUT -T
/var/lib/ambari-agent/tmp/idtest.ambari-qa.1461180728.73.pig
'"'"'http://c6402.ambari.apache.org:50070/webhdfs/v1/tmp/idtest.ambari-qa.1461180728.73.pig?op=CREATE&user.name=hdfs&overwrite=True'"'"'
1>/tmp/tmpgRJGcJ 2>/tmp/tmpavM5TI''] {'logoutput': None, 'quiet': False}
2016-04-20 19:32:08,960 - call returned (0, '')
2016-04-20 19:32:08,968 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl
-sS -L -w '"'"'%{http_code}'"'"' -X PUT
'"'"'http://c6402.ambari.apache.org:50070/webhdfs/v1/tmp/idtest.ambari-qa.1461180728.73.pig?op=SETOWNER&user.name=hdfs&owner=ambari-qa&group='"'"'
1>/tmp/tmpgRGtrp 2>/tmp/tmpD5KTED''] {'logoutput': None, 'quiet': False}
2016-04-20 19:32:09,041 - call returned (0, '')
2016-04-20 19:32:09,042 -
HdfsResource['/tmp/idtest.ambari-qa.1461180728.73.in'] {'security_enabled':
False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab':
[EMPTY], 'source': '/etc/passwd', 'dfs_type': '', 'default_fs':
'hdfs://c6402.ambari.apache.org:8020', 'hdfs_resource_ignore_file':
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ...,
'kinit_path_local': 'kinit', 'principal_name': 'missing_principal', 'user':
'hdfs', 'owner': 'ambari-qa', 'hadoop_conf_dir':
'/usr/hdp/current/hadoop-client/conf', 'type': 'file', 'action':
['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse',
u'/mr-history/done', u'/app-logs', u'/tmp']}
2016-04-20 19:32:09,043 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl
-sS -L -w '"'"'%{http_code}'"'"' -X GET
'"'"'http://c6402.ambari.apache.org:50070/webhdfs/v1/tmp/idtest.ambari-qa.1461180728.73.in?op=GETFILESTATUS&user.name=hdfs'"'"'
1>/tmp/tmpT7ncU1 2>/tmp/tmpBy8iMr''] {'logoutput': None, 'quiet': False}
2016-04-20 19:32:09,104 - call returned (0, '')
2016-04-20 19:32:09,105 - Creating new file
/tmp/idtest.ambari-qa.1461180728.73.in in DFS
2016-04-20 19:32:09,106 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl
-sS -L -w '"'"'%{http_code}'"'"' -X PUT -T /etc/passwd
'"'"'http://c6402.ambari.apache.org:50070/webhdfs/v1/tmp/idtest.ambari-qa.1461180728.73.in?op=CREATE&user.name=hdfs&overwrite=True'"'"'
1>/tmp/tmpnxRUA_ 2>/tmp/tmpNBhXEx''] {'logoutput': None, 'quiet': False}
2016-04-20 19:32:09,244 - call returned (0, '')
2016-04-20 19:32:09,246 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl
-sS -L -w '"'"'%{http_code}'"'"' -X PUT
'"'"'http://c6402.ambari.apache.org:50070/webhdfs/v1/tmp/idtest.ambari-qa.1461180728.73.in?op=SETOWNER&user.name=hdfs&owner=ambari-qa&group='"'"'
1>/tmp/tmpn_Z7pG 2>/tmp/tmpL668wn''] {'logoutput': None, 'quiet': False}
2016-04-20 19:32:09,273 - call returned (0, '')
2016-04-20 19:32:09,273 - HdfsResource[None] {'security_enabled': False,
'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY],
'dfs_type': '', 'default_fs': 'hdfs://c6402.ambari.apache.org:8020',
'hdfs_resource_ignore_file':
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ...,
'kinit_path_local': 'kinit', 'principal_name': 'missing_principal', 'user':
'hdfs', 'action': ['execute'], 'hadoop_conf_dir':
'/usr/hdp/current/hadoop-client/conf', 'immutable_paths':
[u'/apps/hive/warehouse', u'/mr-history/done', u'/app-logs', u'/tmp']}
2016-04-20 19:32:09,275 - Execute['/var/lib/ambari-agent/tmp/templetonSmoke.sh
c6402.ambari.apache.org ambari-qa 50111 idtest.ambari-qa.1461180728.73.pig
no_keytab false kinit no_principal'] {'logoutput': True, 'path':
['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3, 'try_sleep': 5}
Thanks,
Swapan Shridhar
