[Yahoo-eng-team] [Bug 1775140] [NEW] Keystoneauth does not consistently add the collect-timing parameter

[Andras Kovi]

Public bug reported:

Mistral devstack tests started to fail due to the following error [1]

2018-06-04 19:23:37.223185 Server-side error: "no such option collect_timing in 
group [keystone_authtoken]". Detail: 
2018-06-04 19:23:37.223221 Traceback (most recent call last):
2018-06-04 19:23:37.223226 
2018-06-04 19:23:37.223231   File 
"/usr/local/lib/python2.7/dist-packages/wsmeext/pecan.py", line 85, in 
callfunction
2018-06-04 19:23:37.223235 result = f(self, *args, **kwargs)
2018-06-04 19:23:37.223239 
2018-06-04 19:23:37.223243   File 
"/opt/stack/mistral/mistral/api/controllers/v2/event_trigger.py", line 81, in 
post
2018-06-04 19:23:37.223247 workflow_params=values.get('workflow_params'),
2018-06-04 19:23:37.223251 
2018-06-04 19:23:37.223255   File "/opt/stack/mistral/mistral/db/utils.py", 
line 88, in decorate
2018-06-04 19:23:37.223259 return retry.call(_with_auth_context, auth_ctx, 
func, *args, **kw)
2018-06-04 19:23:37.223263 
2018-06-04 19:23:37.223267   File 
"/opt/stack/mistral/mistral/utils/rest_utils.py", line 247, in call
2018-06-04 19:23:37.223271 return super(MistralRetrying, self).call(fn, 
*args, **kwargs)
2018-06-04 19:23:37.223275 
2018-06-04 19:23:37.223279   File 
"/usr/local/lib/python2.7/dist-packages/tenacity/__init__.py", line 330, in call
2018-06-04 19:23:37.223283 start_time=start_time)
2018-06-04 19:23:37.223287 
2018-06-04 19:23:37.223291   File 
"/usr/local/lib/python2.7/dist-packages/tenacity/__init__.py", line 279, in iter
2018-06-04 19:23:37.223295 return fut.result()
2018-06-04 19:23:37.223299 
2018-06-04 19:23:37.223303   File 
"/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 455, 
in result
2018-06-04 19:23:37.223307 return self.__get_result()
2018-06-04 19:23:37.223311 
2018-06-04 19:23:37.223315   File 
"/usr/local/lib/python2.7/dist-packages/tenacity/__init__.py", line 333, in call
2018-06-04 19:23:37.223319 result = fn(*args, **kwargs)
2018-06-04 19:23:37.223322 
2018-06-04 19:23:37.223326   File "/opt/stack/mistral/mistral/db/utils.py", 
line 45, in _with_auth_context
2018-06-04 19:23:37.223330 return func(*args, **kw)
2018-06-04 19:23:37.223334 
2018-06-04 19:23:37.223338   File 
"/opt/stack/mistral/mistral/services/triggers.py", line 186, in 
create_event_trigger
2018-06-04 19:23:37.223342 security.add_trust_id(values)
2018-06-04 19:23:37.223355 
2018-06-04 19:23:37.223360   File 
"/opt/stack/mistral/mistral/services/security.py", line 110, in add_trust_id
2018-06-04 19:23:37.223364 trust = create_trust()
2018-06-04 19:23:37.223368 
2018-06-04 19:23:37.223372   File 
"/opt/stack/mistral/mistral/services/security.py", line 45, in create_trust
2018-06-04 19:23:37.223376 trustee_id = 
keystone.client_for_admin().session.get_user_id()
2018-06-04 19:23:37.223380 
2018-06-04 19:23:37.223384   File 
"/opt/stack/mistral/mistral/utils/openstack/keystone.py", line 143, in 
client_for_admin
2018-06-04 19:23:37.223388 return _admin_client()
2018-06-04 19:23:37.223392 
2018-06-04 19:23:37.223396   File 
"/opt/stack/mistral/mistral/utils/openstack/keystone.py", line 136, in 
_admin_client
2018-06-04 19:23:37.223400 auth=auth
2018-06-04 19:23:37.223403 
2018-06-04 19:23:37.223408   File 
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/session.py", line 
270, in load_from_conf_options
2018-06-04 19:23:37.223412 return Session().load_from_conf_options(*args, 
**kwargs)
2018-06-04 19:23:37.223416 
2018-06-04 19:23:37.223420   File 
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/session.py", line 
251, in load_from_conf_options
2018-06-04 19:23:37.223424 kwargs.setdefault('collect_timing', 
c.collect_timing)
2018-06-04 19:23:37.223428 
2018-06-04 19:23:37.223432   File 
"/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3409, in 
__getattr__
2018-06-04 19:23:37.223436 return self._conf._get(name, self._group)
2018-06-04 19:23:37.223440 
2018-06-04 19:23:37.223444   File 
"/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2950, in _get
2018-06-04 19:23:37.223448 value, loc = self._do_get(name, group, namespace)
2018-06-04 19:23:37.223452 
2018-06-04 19:23:37.223456   File 
"/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2968, in 
_do_get
2018-06-04 19:23:37.223460 info = self._get_opt_info(name, group)
2018-06-04 19:23:37.223464 
2018-06-04 19:23:37.223468   File 
"/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3130, in 
_get_opt_info
2018-06-04 19:23:37.223474 raise NoSuchOptError(opt_name, group)
2018-06-04 19:23:37.223481 
2018-06-04 19:23:37.223487 NoSuchOptError: no such option collect_timing in 
group [keystone_authtoken]

The issue seems to be related to the release of the patch "Collect
timing information for API calls" [2]

[1] 
http://logs.openstack.org/85/527085/29/check/mistral-devstack/56e57ce/controller/logs/apache/mistral_api_log.txt.gz#_2018-06-04_19_23_37_223487

[Yahoo-eng-team] [Bug 1504725] Re: rabbitmq-server restart twice, log is crazy increasing until service restart

[Andras Kovi]

In Mistral, kombu 4.1.0 and amqp 2.2.2, the same issue is present.

** Also affects: mistral
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1504725

Title:
  rabbitmq-server restart twice, log is crazy increasing until service
  restart

Status in Mistral:
  New
Status in neutron:
  Expired
Status in oslo.messaging:
  Won't Fix

Bug description:
  After I restart the rabbitmq-server for the second time, the service log(such 
as nova,neutron and so on) is increasing crazy, log is such as " TypeError: 
'NoneType' object has no attribute '__getitem__'".
  It seems that the channel is setted to None. 

  trace log:

  2015-10-10 15:20:59.413 29515 TRACE root Traceback (most recent call last):
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 95, in 
inner_func
  2015-10-10 15:20:59.413 29515 TRACE root return infunc(*args, **kwargs)
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/oslo_messaging/_executors/impl_eventlet.py", 
line 96, in _executor_thread
  2015-10-10 15:20:59.413 29515 TRACE root incoming = self.listener.poll()
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 
122, in poll
  2015-10-10 15:20:59.413 29515 TRACE root self.conn.consume(limit=1, 
timeout=timeout)
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 
1202, in consume
  2015-10-10 15:20:59.413 29515 TRACE root six.next(it)
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 
1100, in iterconsume
  2015-10-10 15:20:59.413 29515 TRACE root error_callback=_error_callback)
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 
868, in ensure
  2015-10-10 15:20:59.413 29515 TRACE root ret, channel = autoretry_method()
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 458, in _ensured
  2015-10-10 15:20:59.413 29515 TRACE root return fun(*args, **kwargs)
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 545, in __call__
  2015-10-10 15:20:59.413 29515 TRACE root self.revive(create_channel())
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 251, in channel
  2015-10-10 15:20:59.413 29515 TRACE root chan = 
self.transport.create_channel(self.connection)
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in 
create_channel
  2015-10-10 15:20:59.413 29515 TRACE root return connection.channel()
  2015-10-10 15:20:59.413 29515 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 289, in channel
  2015-10-10 15:20:59.413 29515 TRACE root return self.channels[channel_id]
  2015-10-10 15:20:59.413 29515 TRACE root TypeError: 'NoneType' object has no 
attribute '__getitem__'
  2015-10-10 15:20:59.413 29515 TRACE root

To manage notifications about this bug go to:
https://bugs.launchpad.net/mistral/+bug/1504725/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp