On Wed, Nov 20, 2013 at 9:43 PM, Ken'ichi Ohmichi <ken1ohmi...@gmail.com> wrote: > Hi Joe, > > 2013/11/20 Joe Gordon <joe.gord...@gmail.com>: >> Hi All, >> >> As many of you have noticed the gate has been in very bad shape over the >> past few days. Here is a list of some of the top open bugs (without pending >> patches, and many recent hits) that we are hitting. Gate won't be stable, >> and it will be hard to get your code merged, until we fix these bugs. >> >> 1) https://bugs.launchpad.net/bugs/1251920 >> nova >> 468 Hits > > Can we know the frequency of each failure? > I'm trying 1251920 and putting the investigation tempest patch. > https://review.openstack.org/#/c/57193/ > > The patch can avoid this problem 4 times, but I am not sure this is > worth or not. > > > Thanks > Ken'ichi Ohmichi > > --- >> 2) https://bugs.launchpad.net/bugs/1251784 >> neutron, Nova >> 328 Hits >> 3) https://bugs.launchpad.net/bugs/1249065 >> neutron >> 122 hits >> 4) https://bugs.launchpad.net/bugs/1251448 >> neutron >> 65 Hits >> >> Raw Data: >> >> >> Note: If a bug has any hits for anything besides failure, it means the >> fingerprint isn't perfect. >> >> Elastic recheck known issues >> Bug: https://bugs.launchpad.net/bugs/1251920 => message:"assertionerror: >> console output was empty" AND filename:"console.html" Title: Tempest >> failures due to failure to return console logs from an instance Project: >> Status nova: Confirmed Hits FAILURE: 468 Bug: >> https://bugs.launchpad.net/bugs/1251784 => message:"Connection to neutron >> failed: Maximum attempts reached" AND filename:"logs/screen-n-cpu.txt" >> Title: nova+neutron scheduling error: Connection to neutron failed: Maximum >> attempts reached Project: Status neutron: New nova: New Hits FAILURE: 328 >> UNSTABLE: 13 SUCCESS: 275 Bug: https://bugs.launchpad.net/bugs/1240256 => >> message:" 503" AND filename:"logs/syslog.txt" AND >> syslog_program:"proxy-server" Title: swift proxy-server returning 503 during >> tempest run Project: Status openstack-ci: Incomplete swift: New tempest: New >> Hits FAILURE: 136 SUCCESS: 83 >> Pending Patch Bug: https://bugs.launchpad.net/bugs/1249065 => message:"No >> nw_info cache associated with instance" AND filename:"logs/screen-n-api.txt" >> Title: Tempest failure: tempest/scenario/test_snapshot_pattern.py Project: >> Status neutron: New nova: Confirmed Hits FAILURE: 122 Bug: >> https://bugs.launchpad.net/bugs/1252514 => message:"Got error from Swift: >> put_object" AND filename:"logs/screen-g-api.txt" Title: glance doesn't >> recover if Swift returns an error Project: Status devstack: New glance: New >> swift: New Hits FAILURE: 95 >> Pending Patch Bug: https://bugs.launchpad.net/bugs/1244255 => >> message:"NovaException: Unexpected vif_type=binding_failed" AND >> filename:"logs/screen-n-cpu.txt" Title: binding_failed because of l2 agent >> assumed down Project: Status neutron: Fix Committed Hits FAILURE: 92 >> SUCCESS: 29 Bug: https://bugs.launchpad.net/bugs/1251448 => message:" >> possible networks found, use a Network ID to be more specific. (HTTP 400)" >> AND filename:"console.html" Title: BadRequest: Multiple possible networks >> found, use a Network ID to be more specific. Project: Status neutron: New >> Hits FAILURE: 65 Bug: https://bugs.launchpad.net/bugs/1239856 => >> message:"tempest/services" AND message:"/images_client.py" AND >> message:"wait_for_image_status" AND filename:"console.html" Title: >> "TimeoutException: Request timed out" on >> tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestXML >> Project: Status glance: New Hits FAILURE: 62 Bug: >> https://bugs.launchpad.net/bugs/1235435 => message:"One or more ports have >> an IP allocation from this subnet" AND message:" SubnetInUse: Unable to >> complete operation on subnet" AND filename:"logs/screen-q-svc.txt" Title: >> 'SubnetInUse: Unable to complete operation on subnet UUID. One or more ports >> have an IP allocation from this subnet.' Project: Status neutron: Incomplete >> nova: Fix Committed tempest: New Hits FAILURE: 48 Bug: >> https://bugs.launchpad.net/bugs/1224001 => >> message:"tempest.scenario.test_network_basic_ops AssertionError: Timed out >> waiting for" AND filename:"console.html" Title: test_network_basic_ops fails >> waiting for network to become available Project: Status neutron: In Progress >> swift: Invalid tempest: Invalid Hits FAILURE: 42 Bug: >> https://bugs.launchpad.net/bugs/1218391 => message:"Cannot 'createImage'" >> AND filename:"console.html" Title: >> tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestXML.test_delete_image_that_is_not_yet_active >> spurious failure Project: Status nova: Confirmed swift: Confirmed tempest: >> Confirmed Hits FAILURE: 25 >> >> >> >> best, >> Joe Gordon >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Joe seemed to be on the same track with https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:57578,n,z but went far enough to revert the change that introduced that test. A couple people were going to keep hitting those changes to run them through more tests and see if 1251920 goes away. I don't quite understand why this test is problematic (Joe indicated it went in at about the time 1251920 became a problem). I would be very interested in finding out why this caused a problem. You can see frequencies for bugs with known signatures at http://status.openstack.org/elastic-recheck/ Hope this helps. Clark _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
