Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
On 14:14 Thu 02 Apr , Marcus Vinícius Ramires do Nascimento wrote: Hi Mike, I'm working on test coverage improvement for HDS/Hitachi drivers. As I talked to you in #openstack-cinder channel, I'm facing troubles with 3 tests (apparently those fails are not related to the driver) and I'm trying to discover if it's a bug to report it or if it's a infrastructure/configuration problem. I'll switch back the CI to check mode and I'll continue working with this failures isolated to investigate the problem. Is it OK? The CI is running 300 tests now (http://177.84.241.119:1/27/164527/3/silent/). Yes. -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
We believe we have satisfied the required criteria [1] to have NetApp’s fibre channel drivers included in the Kilo release. We have submitted a revert patch [2] along with posting an ether pad [3] to provide more detail on our progress. Thanks for your consideration. [1] http://lists.openstack.org/pipermail/openstack-dev/2015-March/059990.html [2] https://review.openstack.org/#/c/169781/ [3] https://etherpad.openstack.org/p/NetApp-Kilo-Fibre-Channel Thanks so much, -Alex On Thu, Mar 26, 2015 at 8:48 PM, Ryan Hsu r...@vmware.com wrote: Thanks for clarifying! Ryan On Mar 26, 2015, at 5:29 PM, Mike Perez thin...@gmail.com wrote: On 00:24 Fri 27 Mar , Ryan Hsu wrote: Rightfully so, but it doesn't hurt to offer suggestions that might improve the community. It would just be nice to have exclusions reconsidered if there are legitimate bugs behind them. You see them all the time in the tempest tests ala SKIPPED: Skipped until Bug: 1373513 is resolved so it's hard to understand why we can't just apply the same principles to third-party CI. Your usage of exclusions is fine for fixing bugs in my opinion. My meaning of exclusion was not allowing these additional tests to be discovered. -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
Hi Mike, I'm working on test coverage improvement for HDS/Hitachi drivers. As I talked to you in #openstack-cinder channel, I'm facing troubles with 3 tests (apparently those fails are not related to the driver) and I'm trying to discover if it's a bug to report it or if it's a infrastructure/configuration problem. I'll switch back the CI to check mode and I'll continue working with this failures isolated to investigate the problem. Is it OK? The CI is running 300 tests now (http://177.84.241.119:1/27/164527/3/silent/). Thanks, On Thu, Apr 2, 2015 at 12:10 PM, Alex Meade mr.alex.me...@gmail.com wrote: We believe we have satisfied the required criteria [1] to have NetApp’s fibre channel drivers included in the Kilo release. We have submitted a revert patch [2] along with posting an ether pad [3] to provide more detail on our progress. Thanks for your consideration. [1] http://lists.openstack.org/pipermail/openstack-dev/2015-March/059990.html [2] https://review.openstack.org/#/c/169781/ [3] https://etherpad.openstack.org/p/NetApp-Kilo-Fibre-Channel Thanks so much, -Alex On Thu, Mar 26, 2015 at 8:48 PM, Ryan Hsu r...@vmware.com wrote: Thanks for clarifying! Ryan On Mar 26, 2015, at 5:29 PM, Mike Perez thin...@gmail.com wrote: On 00:24 Fri 27 Mar , Ryan Hsu wrote: Rightfully so, but it doesn't hurt to offer suggestions that might improve the community. It would just be nice to have exclusions reconsidered if there are legitimate bugs behind them. You see them all the time in the tempest tests ala SKIPPED: Skipped until Bug: 1373513 is resolved so it's hard to understand why we can't just apply the same principles to third-party CI. Your usage of exclusions is fine for fixing bugs in my opinion. My meaning of exclusion was not allowing these additional tests to be discovered. -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- *Marcus Vinícius Ramires do Nascimento* marcus...@gmail.com Cel: (11) 97396-4018 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
Hi Mike, The majority of the CIs don't run all 304 tests mostly because of these tempest problems. I remember there was a list in the Thirdparty CI Wiki with the common tests that used to fails to everyone, or at least most of people. IMO its better to have a more consistent CI than CIs with full coverage giving lots of false negatives. Don't how much of the tests from that list are fixed on tempest but we should bring that list up again and reconsider to have the list of 'known to fail' tests. Our HBSD drivers are only running 211 because we remove the snapshots tests that were failing due a patch that broken our driver. On Thu, Mar 26, 2015 at 2:21 PM, Mike Perez thin...@gmail.com wrote: On 09:39 Thu 26 Mar , Mike Perez wrote: As discussed in the last Cinder meeting [1], in order to have your volume driver readded into the Kilo release, you must have a CI reporting and stable for five days prior to 4/6. This includes: 1) Providing logs to screen sessions, etc configs, tempest output [2]. 2) You should be running around 304 tests if you're following instructions from the Cinder third party wiki [3]. If you're running less than that, your CI will be *NOT* be considered satisfactory for skipping tests. I will also be emailing individuals who have already asked for exceptions, just to make sure communication was clear. [1] - http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-03-25-16.00.log.html#l-173 [2] - http://ci.openstack.org/third_party.html#requirements [3] - https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F The current CI's not meeting the second requirement: * Cloudbyte * Dell EQL * Dell SC FC * Dell SC ISCSI * EMC VMAX FC * EMC VMX ISCSI * EMC VNX FC * EMC VNX ISCSI * EMC XIO FC * EMC XIO ISCSI * HDS NFS * HDS NAS * Hitachi HBSD FC * Hitach HBSD ISCSI * IBM Flash Systems FC * IBM Flash Systems ISCSI * IBM NAS * IBM XIV (couldn't find tempest results to verify) * IBM Storwize FC * IBM Storwize ISCSI * Nimble * OpenVStorage * Quobyte * XIO FC * XIO ISCSI * Vmware Pretty sure this is because the previous instructions in the wiki were incorrect and are now corrected [1]. This is not the fault of the CI maintainers. As mentioned, individual emails are being sent out to get this all sorted. [1] - https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
On 09:39 Thu 26 Mar , Mike Perez wrote: As discussed in the last Cinder meeting [1], in order to have your volume driver readded into the Kilo release, you must have a CI reporting and stable for five days prior to 4/6. This includes: 1) Providing logs to screen sessions, etc configs, tempest output [2]. 2) You should be running around 304 tests if you're following instructions from the Cinder third party wiki [3]. If you're running less than that, your CI will be *NOT* be considered satisfactory for skipping tests. I will also be emailing individuals who have already asked for exceptions, just to make sure communication was clear. [1] - http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-03-25-16.00.log.html#l-173 [2] - http://ci.openstack.org/third_party.html#requirements [3] - https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F The current CI's not meeting the second requirement: * Cloudbyte * Dell EQL * Dell SC FC * Dell SC ISCSI * EMC VMAX FC * EMC VMX ISCSI * EMC VNX FC * EMC VNX ISCSI * EMC XIO FC * EMC XIO ISCSI * HDS NFS * HDS NAS * Hitachi HBSD FC * Hitach HBSD ISCSI * IBM Flash Systems FC * IBM Flash Systems ISCSI * IBM NAS * IBM XIV (couldn't find tempest results to verify) * IBM Storwize FC * IBM Storwize ISCSI * Nimble * OpenVStorage * Quobyte * XIO FC * XIO ISCSI * Vmware Pretty sure this is because the previous instructions in the wiki were incorrect and are now corrected [1]. This is not the fault of the CI maintainers. As mentioned, individual emails are being sent out to get this all sorted. [1] - https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
On 14:38 Thu 26 Mar , Erlon Cruz wrote: snip Our HBSD drivers are only running 211 because we remove the snapshots tests that were failing due a patch that broken our driver. Whats being done about that in Kilo? That's a minimum feature required for all drivers in Cinder: http://docs.openstack.org/developer/cinder/devref/drivers.html#minimum-features -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
As discussed in the last Cinder meeting [1], in order to have your volume driver readded into the Kilo release, you must have a CI reporting and stable for five days prior to 4/6. This includes: 1) Providing logs to screen sessions, etc configs, tempest output [2]. 2) You should be running around 304 tests if you're following instructions from the Cinder third party wiki [3]. If you're running less than that, your CI will be *NOT* be considered satisfactory for skipping tests. I will also be emailing individuals who have already asked for exceptions, just to make sure communication was clear. [1] - http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-03-25-16.00.log.html#l-173 [2] - http://ci.openstack.org/third_party.html#requirements [3] - https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
Hi Mike, I'm working on it! The bug was fixed and now I'm working to get all tempest.api.volume tests back again on our CI, including also the tests that were missing. On Thu, Mar 26, 2015 at 3:00 PM, Mike Perez thin...@gmail.com wrote: On 14:38 Thu 26 Mar , Erlon Cruz wrote: snip Our HBSD drivers are only running 211 because we remove the snapshots tests that were failing due a patch that broken our driver. Whats being done about that in Kilo? That's a minimum feature required for all drivers in Cinder: http://docs.openstack.org/developer/cinder/devref/drivers.html#minimum-features -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- *Marcus Vinícius Ramires do Nascimento* marcus...@gmail.com Cel: (11) 97396-4018 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
Thanks Mike. The wiki still showing the old edit when I had sent out my earlier email but I see it's been updated now. I've tested running with tox -e all -- volume and that gets us to 291 tests now. -Ryan On Mar 26, 2015, at 4:28 PM, Mike Perez thin...@gmail.com wrote: On 21:45 Thu 26 Mar , Ryan Hsu wrote: Hmm, that's what I thought at first but when I looked at the What tests do I use FAQ, the tests that it says to use links to: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/volume, which is exactly what we're running. Even so, I ran the tox -e all -- volume command and that just runs 4 more tests. As the email that went out, it was mentioned the wiki has been corrected. It should not just be api.volume. There have been a number of drivers today in the #openstack-cinder that have reported to me that they made the adjustments from the wiki and are now reporting 294 tests. If this is not making a difference for you, I recommend you reach out to the liasons mentioned in the wiki and get help: https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#Questions -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
Rightfully so, but it doesn't hurt to offer suggestions that might improve the community. It would just be nice to have exclusions reconsidered if there are legitimate bugs behind them. You see them all the time in the tempest tests ala SKIPPED: Skipped until Bug: 1373513 is resolved so it's hard to understand why we can't just apply the same principles to third-party CI. Thanks, Ryan On Mar 26, 2015, at 4:42 PM, Anita Kuno ante...@anteaya.info wrote: On 03/26/2015 06:48 PM, Ryan Hsu wrote: Exclusions are legitimate and will always be necessary at some point. In the case of the linked bug, this was once a known issue for the VMware driver and we had excluded affected tests so that CI could continue to run. This is the same way we do it in Nova CI and oslo.vmware CI. This is Cinder, Ryan, and Mike is the PTL. It is his decision. Thank you, Anita. Thanks, Ryan On Mar 26, 2015, at 3:30 PM, Mike Perez thin...@gmail.com wrote: On 20:49 Thu 26 Mar , Ryan Hsu wrote: Hi Mike, We (VMware CI) run testr run tempest.api.volume for our Cinder CI and this runs about ~240 tests for us. I'm guessing that the rest of the ~60 tests are not being run due to skips and disabled features. For example, here is a sampling of tests that are skipped in a recent run (note that this is using tempest.conf with no explicit disabling of Cinder services): tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestJSON.test_iscsi_volume ... SKIPPED: Block Live migration not available setUpClass (tempest.api.orchestration.stacks.test_volumes.CinderResourcesTest) ... SKIPPED: Heat support is required setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV2Test) ... SKIPPED: Cinder multi-backend feature disabled tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume ... SKIPPED: SSH required for this test setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test) ... SKIPPED: Cinder backup feature disabled setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test) ... SKIPPED: Cinder backup feature disabled setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV1Test) ... SKIPPED: Cinder multi-backend feature disabled tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved. tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern ... SKIPPED: Skipped until Bug: 1205344 is resolved. setUpClass (tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest) ... SKIPPED: The EC2 API is not available setUpClass (tempest.thirdparty.boto.test_ec2_volumes.EC2VolumesTest) ... SKIPPED: The EC2 API is not available tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved. As we are actually running the volume suite according to the FAQ and the above skipped tests are documented by our CI, would it be possible to add an exception to the rule? I'm sure these numbers will be different for all CIs and as long as people are not abusing and hiding skipped tests, I don't see this as a problem. There will be no exceptions. Everyone must pass the same tests or you're not an approved volume driver for OpenStack Cinder. You should also take this bug [1] Vmware hit as a lesson of doing any excluding in your CI. The driver would've been seriously broken in Kilo if this wasn't caught. [1] - https://bugs.launchpad.net/cinder/+bug/1436603 -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
On 00:11 Fri 27 Mar , Ryan Hsu wrote: Thanks Mike. The wiki still showing the old edit when I had sent out my earlier email but I see it's been updated now. I've tested running with tox -e all -- volume and that gets us to 291 tests now. Ah gotcha. That's great to hear, thank you! -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
On 00:24 Fri 27 Mar , Ryan Hsu wrote: Rightfully so, but it doesn't hurt to offer suggestions that might improve the community. It would just be nice to have exclusions reconsidered if there are legitimate bugs behind them. You see them all the time in the tempest tests ala SKIPPED: Skipped until Bug: 1373513 is resolved so it's hard to understand why we can't just apply the same principles to third-party CI. Your usage of exclusions is fine for fixing bugs in my opinion. My meaning of exclusion was not allowing these additional tests to be discovered. -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
Mike, I am communicating this problem with my teams and will get it resolved asap. Jay On Mar 26, 2015 1:23 PM, Mike Perez thin...@gmail.com wrote: On 09:39 Thu 26 Mar , Mike Perez wrote: As discussed in the last Cinder meeting [1], in order to have your volume driver readded into the Kilo release, you must have a CI reporting and stable for five days prior to 4/6. This includes: 1) Providing logs to screen sessions, etc configs, tempest output [2]. 2) You should be running around 304 tests if you're following instructions from the Cinder third party wiki [3]. If you're running less than that, your CI will be *NOT* be considered satisfactory for skipping tests. I will also be emailing individuals who have already asked for exceptions, just to make sure communication was clear. [1] - http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-03-25-16.00.log.html#l-173 [2] - http://ci.openstack.org/third_party.html#requirements [3] - https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F The current CI's not meeting the second requirement: * Cloudbyte * Dell EQL * Dell SC FC * Dell SC ISCSI * EMC VMAX FC * EMC VMX ISCSI * EMC VNX FC * EMC VNX ISCSI * EMC XIO FC * EMC XIO ISCSI * HDS NFS * HDS NAS * Hitachi HBSD FC * Hitach HBSD ISCSI * IBM Flash Systems FC * IBM Flash Systems ISCSI * IBM NAS * IBM XIV (couldn't find tempest results to verify) * IBM Storwize FC * IBM Storwize ISCSI * Nimble * OpenVStorage * Quobyte * XIO FC * XIO ISCSI * Vmware Pretty sure this is because the previous instructions in the wiki were incorrect and are now corrected [1]. This is not the fault of the CI maintainers. As mentioned, individual emails are being sent out to get this all sorted. [1] - https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
Thanks for clarifying! Ryan On Mar 26, 2015, at 5:29 PM, Mike Perez thin...@gmail.com wrote: On 00:24 Fri 27 Mar , Ryan Hsu wrote: Rightfully so, but it doesn't hurt to offer suggestions that might improve the community. It would just be nice to have exclusions reconsidered if there are legitimate bugs behind them. You see them all the time in the tempest tests ala SKIPPED: Skipped until Bug: 1373513 is resolved so it's hard to understand why we can't just apply the same principles to third-party CI. Your usage of exclusions is fine for fixing bugs in my opinion. My meaning of exclusion was not allowing these additional tests to be discovered. -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
You want to run the volume tests and not tempest.api.volume tests. -Original Message- From: Ryan Hsu [mailto:r...@vmware.com] Sent: Thursday, March 26, 2015 3:50 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded Hi Mike, We (VMware CI) run testr run tempest.api.volume for our Cinder CI and this runs about ~240 tests for us. I'm guessing that the rest of the ~60 tests are not being run due to skips and disabled features. For example, here is a sampling of tests that are skipped in a recent run (note that this is using tempest.conf with no explicit disabling of Cinder services): tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestJSON.test_iscsi_volume ... SKIPPED: Block Live migration not available setUpClass (tempest.api.orchestration.stacks.test_volumes.CinderResourcesTest) ... SKIPPED: Heat support is required setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV2Test) ... SKIPPED: Cinder multi-backend feature disabled tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume ... SKIPPED: SSH required for this test setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test) ... SKIPPED: Cinder backup feature disabled setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test) ... SKIPPED: Cinder backup feature disabled setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV1Test) ... SKIPPED: Cinder multi-backend feature disabled tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern ... SKI PPED: Skipped until Bug: 1373513 is resolved. tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern ... SKIPPED: Skipped until Bug: 1205344 is resolved. setUpClass (tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest) ... SKIPPED: The EC2 API is not available setUpClass (tempest.thirdparty.boto.test_ec2_volumes.EC2VolumesTest) ... SKIPPED: The EC2 API is not available tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved. As we are actually running the volume suite according to the FAQ and the above skipped tests are documented by our CI, would it be possible to add an exception to the rule? I'm sure these numbers will be different for all CIs and as long as people are not abusing and hiding skipped tests, I don't see this as a problem. Thanks, Ryan On Mar 26, 2015, at 9:39 AM, Mike Perez thin...@gmail.com wrote: As discussed in the last Cinder meeting [1], in order to have your volume driver readded into the Kilo release, you must have a CI reporting and stable for five days prior to 4/6. This includes: 1) Providing logs to screen sessions, etc configs, tempest output [2]. 2) You should be running around 304 tests if you're following instructions from the Cinder third party wiki [3]. If you're running less than that, your CI will be *NOT* be considered satisfactory for skipping tests. I will also be emailing individuals who have already asked for exceptions, just to make sure communication was clear. [1] - http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-03-25- 16.00.log.html#l-173 [2] - http://ci.openstack.org/third_party.html#requirements [3] - https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_te sts_do_I_use.3F -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
Hi Mike, We (VMware CI) run testr run tempest.api.volume for our Cinder CI and this runs about ~240 tests for us. I'm guessing that the rest of the ~60 tests are not being run due to skips and disabled features. For example, here is a sampling of tests that are skipped in a recent run (note that this is using tempest.conf with no explicit disabling of Cinder services): tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestJSON.test_iscsi_volume ... SKIPPED: Block Live migration not available setUpClass (tempest.api.orchestration.stacks.test_volumes.CinderResourcesTest) ... SKIPPED: Heat support is required setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV2Test) ... SKIPPED: Cinder multi-backend feature disabled tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume ... SKIPPED: SSH required for this test setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test) ... SKIPPED: Cinder backup feature disabled setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test) ... SKIPPED: Cinder backup feature disabled setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV1Test) ... SKIPPED: Cinder multi-backend feature disabled tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved. tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern ... SKIPPED: Skipped until Bug: 1205344 is resolved. setUpClass (tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest) ... SKIPPED: The EC2 API is not available setUpClass (tempest.thirdparty.boto.test_ec2_volumes.EC2VolumesTest) ... SKIPPED: The EC2 API is not available tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved. As we are actually running the volume suite according to the FAQ and the above skipped tests are documented by our CI, would it be possible to add an exception to the rule? I'm sure these numbers will be different for all CIs and as long as people are not abusing and hiding skipped tests, I don't see this as a problem. Thanks, Ryan On Mar 26, 2015, at 9:39 AM, Mike Perez thin...@gmail.com wrote: As discussed in the last Cinder meeting [1], in order to have your volume driver readded into the Kilo release, you must have a CI reporting and stable for five days prior to 4/6. This includes: 1) Providing logs to screen sessions, etc configs, tempest output [2]. 2) You should be running around 304 tests if you're following instructions from the Cinder third party wiki [3]. If you're running less than that, your CI will be *NOT* be considered satisfactory for skipping tests. I will also be emailing individuals who have already asked for exceptions, just to make sure communication was clear. [1] - http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-03-25-16.00.log.html#l-173 [2] - http://ci.openstack.org/third_party.html#requirements [3] - https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
On 21:45 Thu 26 Mar , Ryan Hsu wrote: Hmm, that's what I thought at first but when I looked at the What tests do I use FAQ, the tests that it says to use links to: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/volume, which is exactly what we're running. Even so, I ran the tox -e all -- volume command and that just runs 4 more tests. As the email that went out, it was mentioned the wiki has been corrected. It should not just be api.volume. There have been a number of drivers today in the #openstack-cinder that have reported to me that they made the adjustments from the wiki and are now reporting 294 tests. If this is not making a difference for you, I recommend you reach out to the liasons mentioned in the wiki and get help: https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#Questions -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
On 03/26/2015 06:48 PM, Ryan Hsu wrote: Exclusions are legitimate and will always be necessary at some point. In the case of the linked bug, this was once a known issue for the VMware driver and we had excluded affected tests so that CI could continue to run. This is the same way we do it in Nova CI and oslo.vmware CI. This is Cinder, Ryan, and Mike is the PTL. It is his decision. Thank you, Anita. Thanks, Ryan On Mar 26, 2015, at 3:30 PM, Mike Perez thin...@gmail.com wrote: On 20:49 Thu 26 Mar , Ryan Hsu wrote: Hi Mike, We (VMware CI) run testr run tempest.api.volume for our Cinder CI and this runs about ~240 tests for us. I'm guessing that the rest of the ~60 tests are not being run due to skips and disabled features. For example, here is a sampling of tests that are skipped in a recent run (note that this is using tempest.conf with no explicit disabling of Cinder services): tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestJSON.test_iscsi_volume ... SKIPPED: Block Live migration not available setUpClass (tempest.api.orchestration.stacks.test_volumes.CinderResourcesTest) ... SKIPPED: Heat support is required setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV2Test) ... SKIPPED: Cinder multi-backend feature disabled tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume ... SKIPPED: SSH required for this test setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test) ... SKIPPED: Cinder backup feature disabled setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test) ... SKIPPED: Cinder backup feature disabled setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV1Test) ... SKIPPED: Cinder multi-backend feature disabled tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved. tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern ... SKIPPED: Skipped until Bug: 1205344 is resolved. setUpClass (tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest) ... SKIPPED: The EC2 API is not available setUpClass (tempest.thirdparty.boto.test_ec2_volumes.EC2VolumesTest) ... SKIPPED: The EC2 API is not available tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved. As we are actually running the volume suite according to the FAQ and the above skipped tests are documented by our CI, would it be possible to add an exception to the rule? I'm sure these numbers will be different for all CIs and as long as people are not abusing and hiding skipped tests, I don't see this as a problem. There will be no exceptions. Everyone must pass the same tests or you're not an approved volume driver for OpenStack Cinder. You should also take this bug [1] Vmware hit as a lesson of doing any excluding in your CI. The driver would've been seriously broken in Kilo if this wasn't caught. [1] - https://bugs.launchpad.net/cinder/+bug/1436603 -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
Exclusions are legitimate and will always be necessary at some point. In the case of the linked bug, this was once a known issue for the VMware driver and we had excluded affected tests so that CI could continue to run. This is the same way we do it in Nova CI and oslo.vmware CI. Thanks, Ryan On Mar 26, 2015, at 3:30 PM, Mike Perez thin...@gmail.com wrote: On 20:49 Thu 26 Mar , Ryan Hsu wrote: Hi Mike, We (VMware CI) run testr run tempest.api.volume for our Cinder CI and this runs about ~240 tests for us. I'm guessing that the rest of the ~60 tests are not being run due to skips and disabled features. For example, here is a sampling of tests that are skipped in a recent run (note that this is using tempest.conf with no explicit disabling of Cinder services): tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestJSON.test_iscsi_volume ... SKIPPED: Block Live migration not available setUpClass (tempest.api.orchestration.stacks.test_volumes.CinderResourcesTest) ... SKIPPED: Heat support is required setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV2Test) ... SKIPPED: Cinder multi-backend feature disabled tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume ... SKIPPED: SSH required for this test setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test) ... SKIPPED: Cinder backup feature disabled setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test) ... SKIPPED: Cinder backup feature disabled setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV1Test) ... SKIPPED: Cinder multi-backend feature disabled tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved. tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern ... SKIPPED: Skipped until Bug: 1205344 is resolved. setUpClass (tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest) ... SKIPPED: The EC2 API is not available setUpClass (tempest.thirdparty.boto.test_ec2_volumes.EC2VolumesTest) ... SKIPPED: The EC2 API is not available tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved. As we are actually running the volume suite according to the FAQ and the above skipped tests are documented by our CI, would it be possible to add an exception to the rule? I'm sure these numbers will be different for all CIs and as long as people are not abusing and hiding skipped tests, I don't see this as a problem. There will be no exceptions. Everyone must pass the same tests or you're not an approved volume driver for OpenStack Cinder. You should also take this bug [1] Vmware hit as a lesson of doing any excluding in your CI. The driver would've been seriously broken in Kilo if this wasn't caught. [1] - https://bugs.launchpad.net/cinder/+bug/1436603 -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Deadline For Volume Drivers to Be Readded
On 22:42 Thu 26 Mar , Ryan Hsu wrote: Like I mentioned earlier, these numbers are going to be different for everyone depending on their testbed set up or driver capabilities. Just by disabling Heat in devstack you're going to miss some tests. As long as people are transparent about this, I don't see the harm here. This has nothing to do with what services you're running (e.g. heat, sahara). If you're just running volume.api tests, you're verifying a limited number of tests due to your discovery settings. For your convenience, here's a diff between testing just volume.api versus tox -e all -- volume: http://paste.openstack.org/raw/196962/ As you'll notice there is a lot of snapshot/image tests missed that should be covered by all CI's. However Vmware is kicking off the tests, this can be corrected without enabling any additional OpenStack services by: tox -e all -- volume or if you're using devstack-gate, export this before running the tests: export DEVSTACK_GATE_TEMPEST_REGEX=volume All of this is explained in the Cinder wiki: https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
