As Matt found, it the problem was with out of date requirements. Going forward I would advise any third party CI that is not spinning up a new VM for every job to purge all python packages after each run. This will make devstack reinstall everything, avoid this type of problem. Though the problem was with global requirements, only our CI was failing because everyone else was getting the newest version of each package each time.
We are still failing on one test (test_ec2_instance_run.InstanceRunTest) and we are not sure what the cause is. Here is a log from a recent run: http://openstack-ci.tintri.com/tintri/refs-changes-88-210588-1/ Here is the failing test: ft1.280: setUpClass (tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest)_StringException: Traceback (most recent call last): File "tempest/test.py", line 272, in setUpClass six.reraise(etype, value, trace) File "tempest/test.py", line 265, in setUpClass cls.resource_setup() File "tempest/thirdparty/boto/test_ec2_instance_run.py", line 91, in resource_setup state = wait.state_wait(_state, "available") File "tempest/thirdparty/boto/utils/wait.py", line 51, in state_wait (dtime, final_set, status)) AssertionError: State change timeout exceeded!(196s) While waitingfor set(['available']) at "failed" >From n-crt log: 2015-08-07 15:21:58.237 ERROR oslo_messaging.rpc.dispatcher [req-d4ce0001-0754-461f-8fdc-57908baf88f7 tempest-InstanceRunTest-1110235717 tempest-InstanceRunTest-357311946] Exception during message handling: 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher executor_callback)) 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher executor_callback) 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/cert/manager.py", line 70, in decrypt_text 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher return crypto.decrypt_text(project_id, base64.b64decode(text)) 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/crypto.py", line 200, in decrypt_text 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher return priv_key.decrypt(text, padding.PKCS1v15()) 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 536, in decrypt 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher return _enc_dec_rsa(self._backend, self, ciphertext, padding) 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 76, in _enc_dec_rsa 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher return _enc_dec_rsa_pkey_ctx(backend, key, data, padding_enum) 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 105, in _enc_dec_rsa_pkey_ctx 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher _handle_rsa_enc_dec_error(backend, key) 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 145, in _handle_rsa_enc_dec_error 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher assert errors[0].reason in decoding_errors 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher AssertionError 2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher 2 The 08/07/2015 09:01, Matt Riedemann wrote: > > > On 8/7/2015 8:38 AM, Matt Riedemann wrote: > > > > > >On 8/6/2015 3:30 PM, Skyler Berg wrote: > >>After the change "cleanup NovaObjectDictCompat from virtual_interface" > >>[1] was merged into Nova on the morning of August 5th, Tintri's CI for > >>Cinder started failing 13 test cases that involve a volume being > >>attached to an instance [2]. > >> > >>I have verified that the tests fail with the above mentioned change and > >>pass when running against the previous commit. > >> > >>If anyone knows why this patch is causing an issue or is experiencing > >>similar problems, please let me know. > >> > >>In the meantime, expect Tintri's CI to be either down or reporting > >>failures until a solution is found. > >> > >>[1] https://review.openstack.org/#/c/200823/ > >>[2] http://openstack-ci.tintri.com/tintri/refs-changes-06-201406-35/ > >> > > > >It looks like you're missing this change in whatever version of > >oslo.versionedobjects you have in your CI: > > > >https://review.openstack.org/#/c/202200/ > > > >That should be in o.vo 0.6.0, latest is 0.7.0. What version of > >oslo.versionedobjects is on the this system? It would be helpful to > >have pip freeze output. > > > > I proposed a change to global-requirements to raise the minimum required > oslo.versionedobjects to >= 0.6.0 here: > > https://review.openstack.org/#/c/210445/ > > -- > > Thanks, > > Matt Riedemann -- __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
