Hi Dale, good find. Feel free to submit a patch for this for branch-2.5 and
trunk.
I'm in favor or appending ".0" for ms if last_run_time doesn't already contain
ms.
>>> from datetime import datetime
>>> last_run_time = "2016-12-19 22:37:33"
>>> x = datetime.strptime(last_run_time, "%Y-%m-%d %H:%M:%S.%f")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_strptime.py",
line 325, in _strptime
(data_string, format))
ValueError: time data '2016-12-19 22:37:33' does not match format '%Y-%m-%d
%H:%M:%S.%f'
>>> last_run_time_with_ms = last_run_time if "." in last_run_time else
>>> last_run_time + ".0"
>>> print last_run_time_with_ms
2016-12-19 22:37:33.0
>>> x = datetime.strptime(last_run_time_with_ms, "%Y-%m-%d %H:%M:%S.%f")
>>> print x
2016-12-19 22:37:33
Thanks,
Alejandro
From: Dale Bradman <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Thursday, April 20, 2017 at 9:45 AM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Express Upgrade to HDP 2.5 fails on restarting the Hive Metastore
Hi.
I'm on Ambari 2.4.2.0-136 and HDP 2.4.2.0 with a view to doing an EXPRESS
Upgrade to HDP-2.5.3.0.
It finishes the service checks and the starts restarting Hive where it fails on
"Restarting Hive Metastore"
The error is:
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
line 259, in <module>
HiveMetastore().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 280, in execute
method(env)
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 696, in restart
self.pre_upgrade_restart(env, upgrade_type=upgrade_type)
File
"/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
line 114, in pre_upgrade_restart
self.upgrade_schema(env)
File
"/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
line 198, in upgrade_schema
status_params.tmp_dir)
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py",
line 208, in cached_kinit_executor
if (now - datetime.strptime(last_run_time, "%Y-%m-%d %H:%M:%S.%f") >
timedelta(minutes=expiration_time)):
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data '2016-12-19 22:37:33' does not match format '%Y-%m-%d
%H:%M:%S.%f'
Doing some digging, the
/usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py
does not handle non millisecond timestamps (line 205).
The timestamp it is trying to analyse here is the
/var/lib/ambari-agent/tmp/kinit_executor_cache/<<hash(hive.service.keytab|hive.principal.name)>>.
The timestamp within this file matches that of the error: "'2016-12-19
22:37:33'" - which contains no miliseconds. There are other files in this
directory all of which have timesstamps with miliseconds, so I'm going to
assume this is 1 in a million. Could you confirm this please?
Options for fixing in my opinion are:
1. Adding an additional conditional statement before line 205 that
determines if the timestamp contains miliseconds
2. Manually changing '2016-12-19 22:37:33' to '2016-12-19 22:37:33.000000'
This is an Express Upgrade, so downtime has been arranged but it's reaching
critical point, would option 2 work for immediate impact?
I've also upgraded to Ambari 2.5.0.3-7 on a test cluster and the code remains
as it is.
Let me know if JIRA required.
Cheers,
Dale
