Hi.
I'm on Ambari 2.4.2.0-136 and HDP 2.4.2.0 with a view to doing an EXPRESS
Upgrade to HDP-2.5.3.0.
It finishes the service checks and the starts restarting Hive where it fails on
"Restarting Hive Metastore"
The error is:
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
line 259, in <module>
HiveMetastore().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 280, in execute
method(env)
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 696, in restart
self.pre_upgrade_restart(env, upgrade_type=upgrade_type)
File
"/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
line 114, in pre_upgrade_restart
self.upgrade_schema(env)
File
"/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
line 198, in upgrade_schema
status_params.tmp_dir)
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py",
line 208, in cached_kinit_executor
if (now - datetime.strptime(last_run_time, "%Y-%m-%d %H:%M:%S.%f") >
timedelta(minutes=expiration_time)):
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data '2016-12-19 22:37:33' does not match format '%Y-%m-%d
%H:%M:%S.%f'
Doing some digging, the
/usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py
does not handle non millisecond timestamps (line 205).
The timestamp it is trying to analyse here is the
/var/lib/ambari-agent/tmp/kinit_executor_cache/<<hash(hive.service.keytab|hive.principal.name)>>.
The timestamp within this file matches that of the error: "'2016-12-19
22:37:33'" - which contains no miliseconds. There are other files in this
directory all of which have timesstamps with miliseconds, so I'm going to
assume this is 1 in a million. Could you confirm this please?
Options for fixing in my opinion are:
1. Adding an additional conditional statement before line 205 that
determines if the timestamp contains miliseconds
2. Manually changing '2016-12-19 22:37:33' to '2016-12-19 22:37:33.000000'
This is an Express Upgrade, so downtime has been arranged but it's reaching
critical point, would option 2 work for immediate impact?
I've also upgraded to Ambari 2.5.0.3-7 on a test cluster and the code remains
as it is.
Let me know if JIRA required.
Cheers,
Dale
