No, we haven't put this document in etherpad. But it's a great idea. This way other folks can jump in with ideas. I'll put it up in eteherpad over the next few days
On Wed, Jun 19, 2013 at 10:10 AM, Sean Dague <[email protected]> wrote: > This is great stuff! You guys have this up in etherpad somewhere? > > Really great to see people organizing around this. I'd reviewed a bunch of > Jordan's code, so I knew he was in the mix, sorry for not realizing there > were other folks working on this as well. > > -Sean > > > On 06/19/2013 10:27 AM, Miguel Lavalle wrote: > >> Hi, >> >> As of last week, we have organized a team to deal with the Quantum in >> full gate issues. There are 3 people working on this: Jordan Pittier, >> Ala.Rezmerita and myself. Below you will find a document describing all >> the identified issues and who is assigned to fix them. Of cource, we can >> use more hands / heads. If anyone wants to help or has identified other >> issues not listed below, please get in touch with us. >> >> Regards >> >> Fix Jenkins gate-tempest-devstack-vm-**quantum-full >> >> >> This document lists all the Tempest tests that are failing on a Devstack >> + Quantum Trunk setup. The goal is to make these tests pass for the >> Havana Milestone 3. Alla, Miguel and Jordan are working on this. >> >> >> For each test : >> >> * >> >> Provide the full path of the test. If the test has a double >> interface (JSON and XML), it’s enough to provide only the JSON path. >> >> * >> >> Provide a small stacktrace or a link to a Paste service such as >> http://paste.openstack.org/ >> >> * >> >> Would be great to provide the CURL URL that triggers the bug >> >> * >> >> Provide a small analysis of what the bug may be and the components >> involved (Quantum, Quantum API on Nova’s side, Nova-network etc.) >> >> * >> >> As far as possible, provide the file and the line number where the >> Python exception is raised or translated (“casted”) >> >> * >> >> If already filed, an URL to the bug report in Launchpad >> >> >> Tests related to the Fixed IPs Compute API extension (Miguel) >> >> >> 1) >> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_list_* >> *fixed_ip_details >> >> * >> >> Trace: >> http://paste.openstack.org/**show/38363/<http://paste.openstack.org/show/38363/> >> >> * >> >> curl-H "X-Auth-Token:$TOKEN" -X GET >> http://$IP:8774/v2/$TENANT_ID/**os-fixed-ips/10.0.0.3 < >> http://10.0.0.3> >> >> * >> >> API: >> http://api.openstack.org/api-**ref.html#ext-os-fixed-ips<http://api.openstack.org/api-ref.html#ext-os-fixed-ips> >> >> >> * >> >> Explanation: Call to >> nova.api.openstack.compute.**contrib.fixed_ips.** >> FixedIPController::show() >> This file doesn’t use the Quantum API nor the Nova-Network API. It >> interacts directly with the DB, which is bad. >> >> * >> >> A possible fix would be to : >> >> 1. >> >> >> Change >> nova.api.openstack.compute.**contrib.fixed_ips.** >> FixedIPController::show() >> to use either nova.network.api.API::get_**fixed_ip() (for Nova >> Network) or nova.network.quantumv2.api.**API:get_fixed_ip() (for >> Nova Network) >> >> 2. >> >> >> Implement nova.network.quantumv2.api.**API:get_fixed_ip() which >> currently raises a NotImplementedError exception >> >> 2) >> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_set_** >> reserve >> >> * >> >> Trace: >> http://paste.openstack.org/**show/38372/<http://paste.openstack.org/show/38372/> >> >> * >> >> curl-H "X-Auth-Token:$TOKEN" -X POST >> http://$IP:8774/v2/$TENANT_ID/**os-fixed-ips/10.0.0.3/action >> <http://10.0.0.3/action> >> >> * >> >> API: >> http://api.openstack.org/api-**ref.html#ext-os-fixed-ips<http://api.openstack.org/api-ref.html#ext-os-fixed-ips> >> >> >> * >> >> Possible Fix: Should call >> nova.api.openstack.compute.**contrib.fixed_ips.** >> FixedIPController::_set_**reserved() >> (once the result of db.fixed_ip_get_by_address() is made through the >> API). Here the problem is that neither Nova-Network nor Quatum API >> implement an equivalent of db.fixed_ip_update() >> >> 3) >> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_set_** >> unreserve >> >> * >> >> Same as 2) >> >> >> >> Tests related to Quotas Admin(Miguel) >> >> >> 4)tempest.api.compute.admin.**test_quotas:**QuotasAdminTestJSON.test_** >> security_groups_exceed_limit >> >> 5)tempest.api.compute.admin.**test_quotas:**QuotasAdminTestJSON.test_** >> security_groups_rules_exceed_**limit >> >> >> Tests related to Floating IPs >> >> >> 6)tempest.api.compute.**floating_ips.test_floating_**ips_actions:** >> FloatingIPsTestJSON.test_**associate_ip_to_server_** >> without_passing_floating_ip >> >> * >> >> Trace Tempest >> http://paste.openstack.org/**show/38431/<http://paste.openstack.org/show/38431/> >> >> * >> >> Bug : >> https://bugs.launchpad.net/**quantum/+bug/1190242<https://bugs.launchpad.net/quantum/+bug/1190242> >> >> * >> >> curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X >> POST http://$IP:8774/v2/$TENANT_ID/**servers/$SERVER_ID/action >> <http://10.1.59.157:8774/v2/**de6c5fbc55a34cbcaa3d79eb6b21a7** >> 84/servers/0b2ad3b6-c14a-4d89-**b2a0-c015f0a88a1f/action<http://10.1.59.157:8774/v2/de6c5fbc55a34cbcaa3d79eb6b21a784/servers/0b2ad3b6-c14a-4d89-b2a0-c015f0a88a1f/action> >> >-d >> ‘{"addFloatingIp": {"address": ""}}’ >> >> >> * >> >> Notice that the address of the FloatingIP is empty. >> >> * >> >> API : >> http://api.openstack.org/api-**ref.html#ext-floating-ips<http://api.openstack.org/api-ref.html#ext-floating-ips> >> >> >> * >> >> Explanation : Mismatch in the 2 API. If there is only one IP in the >> pool, Quantum allows the floating IP to be empty and returns the one >> and only IP in the pool. Nova-network doesn’t allow this and returns >> a 404 >> >> * >> >> Review >> https://review.openstack.org/#**/c/32740/<https://review.openstack.org/#/c/32740/> >> >> >> 7)tempest.api.compute.**floating_ips.test_floating_**ips_actions:** >> FloatingIPsTestJSON.test_**delete_floating_ip >> >> * >> >> Trace Tempest : >> http://paste.openstack.org/**show/38505/<http://paste.openstack.org/show/38505/> >> >> * >> >> Bug : >> https://bugs.launchpad.net/**tempest/+bug/1160309(see<https://bugs.launchpad.net/tempest/+bug/1160309(see>comment >> #2) >> >> >> * >> >> This is related to bug 9) >> >> >> 8)tempest.api.compute.**floating_ips.test_floating_**ips_actions:** >> FloatingIPsTestJSON.test_**delete_nonexistant_floating_ip >> >> >> * >> >> Related to 9) >> >> >> 9)tempest.api.compute.**floating_ips.test_list_**floating_ips:** >> FloatingIPDetailsTestJSON.**test_get_nonexistant_floating_**ip_details >> >> * >> >> Bug >> :https://bugs.launchpad.net/**tempest/+bug/1160309<https://bugs.launchpad.net/tempest/+bug/1160309> >> >> * >> >> Trace >> tempest:http://paste.**openstack.org/show/38430/<http://paste.openstack.org/show/38430/> >> >> * >> >> Trace nova: >> http://paste.openstack.org/**show/38433/<http://paste.openstack.org/show/38433/> >> >> >> * >> >> Curl:curl -H "X-Auth-Token:$TOKEN" -X GET >> http://$IP:8774/v2/$PROJECT_**ID/os-floating-ips/**99987878787878 >> >> Proposed Fix: >> https://review.openstack.org/**33024<https://review.openstack.org/33024> >> >> >> Tests Related to Security Groups (Jordan) >> >> >> 10)tempest.api.compute.**security_groups.test_security_**group_rules:** >> SecurityGroupRulesTestJSON.**test_security_group_rules_** >> create_with_invalid_id >> >> * >> >> TRACE: >> http://paste.openstack.org/**show/38373/<http://paste.openstack.org/show/38373/> >> >> * >> >> curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X >> >> POST http://$IP:8774/v2/$TENANT_ID/**os-security-group-rules -d >> ‘{"security_group_rule": {"from_port": 22, "ip_protocol": "tcp", >> "to_port": 22, "parent_group_id": "9991393497170", "cidr": null, >> "group_id": null}}’ >> >> * >> >> Explanation: Notice that the parent_goup_id is a numerical ID and >> not a UUID. Quantum has an additional check to validate that the ID >> is an UUID (see >> nova/network/security_group/**quantum_driver.py::validate_**id()) >> >> * >> >> API: >> http://api.openstack.org/api-**ref.html#ext-os-security-**groups<http://api.openstack.org/api-ref.html#ext-os-security-groups> >> >> * >> >> Bug: >> https://bugs.launchpad.net/**tempest/+bug/1182384<https://bugs.launchpad.net/tempest/+bug/1182384> >> >> >> * >> >> Review: >> https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/> >> >> >> 11)tempest.api.compute.**security_groups.test_security_**group_rules:** >> SecurityGroupRulesTestJSON.**test_security_group_rules_** >> delete_with_invalid_id >> >> * >> >> >> http://paste.openstack.org/**show/38424/<http://paste.openstack.org/show/38424/> >> >> >> * >> >> curl -H "X-Auth-Token:$TOKEN" -X DELETE >> http://$IP:8774/v2$TENANT_ID/**os-security-group-rules/** >> 9991407551273 >> >> * >> >> Explanation : Same bug as 10 >> >> * >> >> API: >> http://api.openstack.org/api-**ref.html#ext-os-security-**groups<http://api.openstack.org/api-ref.html#ext-os-security-groups> >> >> >> * >> >> Review: >> https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/> >> >> 12)tempest.api.compute.**security_groups.test_security_** >> groups:SecurityGroupsTestJSON.**test_delete_nonexistant_**security_group >> >> * >> >> Same as bug 10 >> >> >> * >> >> Review: >> https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/> >> >> >> 13)tempest.api.compute.**security_groups.test_security_** >> groups:SecurityGroupsTestJSON.**test_security_group_get_** >> nonexistant_group >> >> * >> >> Same as bug 10 >> >> >> * >> >> Review: >> https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/> >> >> >> 14)tempest.api.compute.**security_groups.test_security_** >> groups:SecurityGroupsTestJSON.**test_security_group_create_** >> with_duplicate_name >> >> Security Group with duplicate name should not be created, but two groups >> with the same name can be created in quantum. We have here the same >> problem as in 15 and in 16. With Quantum, there is no validation that a >> group with given name exists already or if the given SG name is empty >> or is composed of white spaces or is more than 255 chars. >> >> >> In the description of bug >> https://bugs.launchpad.net/**nova/+bug/1161411this<https://bugs.launchpad.net/nova/+bug/1161411this>issue >> is generally >> >> discussed. SecurityGroup API are based on the ID and not the names, >> except for adding an instance to a security group. In order to solve >> the last problem the bug https://bugs.launchpad.net/** >> nova/+bug/1161473was <https://bugs.launchpad.net/nova/+bug/1161473was> >> added. >> >> >> The major question is if these 3 tests (14, 15, 16) : does the name of a >> security group is really that important? If so, we must add some >> validation methods. If not the test suit concerning this part must be >> disable in tempest. What do you think Miguel? >> >> >> >> 15)tempest.api.compute.**security_groups.test_security_** >> groups:SecurityGroupsTestJSON.**test_security_group_create_** >> with_invalid_group_description >> >> 16)tempest.api.compute.**security_groups.test_security_** >> groups:SecurityGroupsTestJSON.**test_security_group_create_** >> with_invalid_group_name >> >> BUG:https://bugs.launchpad.**net/nova/+bug/1161411+<https://bugs.launchpad.net/nova/+bug/1161411+> >> https://bugs.launchpad.net/**nova/+bug/1161473<https://bugs.launchpad.net/nova/+bug/1161473> >> >> Traceback(tempest) >> :http://paste.openstack.org/**show/38423/<http://paste.openstack.org/show/38423/> >> >> The Security Group should not be created with group name an empty string >> or with white spaces/chars more than 255 >> >> CURL:curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" >> -X POST http://$IP:8774/v2/$PROJECT_**ID/os-security-groups -d >> '{"security_group": {"name": " ", "description": >> "description-1554950088"}}' >> >> curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X >> POST http://$IP:8774/v2/$PROJECT_**ID/os-security-groups -d >> '{"security_group": {"name": " ", "description": >> "description-1554950088"}}' >> >> >> 17)tempest.api.compute.**security_groups.test_security_** >> groups:SecurityGroupsTestJSON.**test_server_security_groups >> >> * >> >> Tempest Trace : >> http://paste.openstack.org/**show/38427/<http://paste.openstack.org/show/38427/> >> >> * >> >> Fixed by >> https://review.openstack.org/#**/c/32288/(merged<https://review.openstack.org/#/c/32288/(merged>on >> June,12th) >> >> >> >> >> Tests related to servers(Ala) >> >> >> 18)tempest.api.compute.**servers.test_list_server_**filters:** >> ListServerFiltersTestXML.test_**list_servers_filtered_by_ip_**regex >> >> * >> >> BUG: >> https://bugs.launchpad.net/**quantum/+bug/1182883<https://bugs.launchpad.net/quantum/+bug/1182883> >> >> * >> >> BP: >> https://blueprints.launchpad.**net/quantum/+spec/like-op-list<https://blueprints.launchpad.net/quantum/+spec/like-op-list> >> >> >> * >> >> CURL : GET http://$IP:8774/v2/$PROJECT_**ID/servers?ip=10. >> >> * >> >> Explanation: The regex search is not supported by Quantum. Thus >> Quantum returns a 404 Not Found (0 server match) where Tempest >> expects one server to be found. >> >> * >> >> Possible Fix : For "search port by IP with regex" feature, I think >> the best place to hack it would be in file >> quantum/db/db_base_plugin_v2.**py::_get_ports_query() >> >> >> 19)tempest.api.compute.**servers.test_servers_negative:** >> ServersNegativeTest.test_**create_with_nonexistent_**security_group >> >> FIXED : >> https://review.openstack.org/#**/c/30271/<https://review.openstack.org/#/c/30271/> >> >> >> 20)tempest.api.compute.**servers.test_virtual_**interfaces:** >> VirtualInterfacesTestXML.test_**list_virtual_interfaces >> >> * >> >> TRACE (NOVA): >> http://paste.openstack.org/**show/38371/<http://paste.openstack.org/show/38371/> >> >> * >> >> BUG: >> https://bugs.launchpad.net/**tempest/+bug/1183436<https://bugs.launchpad.net/tempest/+bug/1183436> >> >> >> * >> >> CURL: GET >> http://$IP:8774/v2/$PROJECT_**ID/servers/$SERVER/os-virtual-** >> interfaces >> >> * >> >> Explanation: This HTTP request calls the Quantum API >> (nova/nova/network/quantumv2/**api.py) and specifically the >> get_vifs_by_* methods which are not implemented (raise >> NotImplementedError()) >> >> * >> >> Possible Fix: >> >> >> o >> >> skip this test if Quantum is enabled as set in Tempest >> configuration. Or >> >> o >> >> Implement the get_vifs_by_* methods >> >> PATCH: (still not approved) >> https://review.openstack.org/#**/c/31755/<https://review.openstack.org/#/c/31755/> >> >> >> On Tue, Jun 18, 2013 at 5:56 PM, Sean Dague <[email protected] >> <mailto:[email protected]>> wrote: >> >> On 06/18/2013 03:32 PM, Monty Taylor wrote: >> >> >> >> On 06/18/2013 03:14 PM, David Ripton wrote: >> >> On 06/18/2013 12:43 PM, Martina Kollarova wrote: >> >> Jenkins keeps running all the tests, even if the basic >> pep8 test fails, >> and runs all of the (very slow) Tempest Quantum tests, >> even though >> almost all of them are failing. >> >> I propose that it should fail and stop all of the other >> tests once there >> is a failure in a voting test. For non-voting tests, it >> should stop only >> itself, not the others. >> >> This would decrease the feedback loop and we wouldn't >> have to wait for >> the non-voting Quantum tests to see that they failed as >> always. >> >> >> -1 >> >> In addition to the other objections, we currently get a lot >> of false >> positives (fail, retry, fail, retry, succeed), and it would >> be harder to >> debug these problems if the output was truncated differently >> each time. >> >> Is anyone working on fixing the perma-failing Quantum test? >> When the >> Postgres test was perma-failing, one of the infrastructure >> folks gave us >> an ultimatum that if nobody fixed it soon, it would be >> disabled. (Happy >> ending: Mauro fixed it before it got disabled.) >> >> >> That was brought up a little while ago, but we had already spent >> so much >> effort to get it working in the first place, none of us had the >> heart to >> put in such an ultimatum. But seriously- it might be time for an >> all-hands-on-deck dogpile to figure out what's up the the >> quantum gate. >> >> >> The biggest cause of the Quantum vs. Full Tempest runs is that a lot >> of the network api's in nova currently don't do translation of >> errors. So under nova-network certain data validation and error >> codes are returned, when quantum is used others are returned. >> >> This is a nova-api, so it needs to be consistent regardless of >> backend (i.e. we don't return different API responses on different >> databases). >> >> Issues like this one - >> >> https://bugs.launchpad.net/__**nova/+bug/1160309<https://bugs.launchpad.net/__nova/+bug/1160309> >> >> >> <https://bugs.launchpad.net/**nova/+bug/1160309<https://bugs.launchpad.net/nova/+bug/1160309> >> > >> >> Jordan Pittier has been working on some of these issues (he's the >> only one I've seen working them from a Tempest / nova side), and got >> to the crux of the problem. It could use more hands though to >> organize the rest of those and get them banged out. >> >> I'm sure there are other issues once we get past this class. But >> that would go a long way. >> >> -Sean >> >> -- >> Sean Dague >> http://dague.net >> >> ______________________________**___________________ >> OpenStack-dev mailing list >> [email protected].**__org >> >> <mailto:OpenStack-dev@lists.**openstack.org<[email protected]> >> > >> http://lists.openstack.org/__**cgi-bin/mailman/listinfo/__** >> openstack-dev<http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev>< >> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >> > >> >> >> >> >> >> ______________________________**_________________ >> OpenStack-dev mailing list >> [email protected].**org <[email protected]> >> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >> >> > > -- > Sean Dague > http://dague.net > > ______________________________**_________________ > OpenStack-dev mailing list > [email protected].**org <[email protected]> > http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
