[openstack-dev] [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
Hello Juilen, I forwarded your mail to the correct mailing list. Please do not use the qa list any longer. I am happy that you are interested in stress tests. All the tests in tempest/stress/actions are more or less deprecated. So what you should use instead is the stress decorator (e.g. https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_actions.py#L55). Unfortunately it's not yet used for scenarios like you describe. I'd suggest to build a scenario test in tempest/scenario and use this decorator on it. Any patch like that is welcome on gerrit. If you are planning to work in that area for more than just a patch, a blueprint would be nice. A good way to coordinate your efforts is also in the QA meeting (https://wiki.openstack.org/wiki/Meetings/QATeamMeeting) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Wednesday, January 15, 2014 5:57 PM To: openstack...@lists.openstack.org Subject: [openstack-qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi everyone, I’m quite new on OpenStack / Tempest and I’m actually working on stress tests. I want to suggest a new feature in a currently available stress test. Not sure if this email should be posted on the QA mailing list or the dev mailing list, but I give it a try here since it is about a Tempest stress test ☺ At the moment the “ssh_floating.py” stress test seems really interesting but I would like to improve it a bit. By now this script is simulating an SSH connection by binding a TCP socket on the newly created instance. But this test don’t allow us to check if this instance is really available. I’m mostly thinking about the metadata service unable to provide the SSH key pair to the instance, but surely other scenarios can lead to an instance considered “ACTIVE” but actually unusable. So I’m implementing a full SSH connection test using the “paramiko” SSH library and a key pair generated in the same way the other test resources are managed in this script : either one SSH key pair for every test runs or a new key pair for each run (depends on the JSON configuration file). I don’t plan to remove the old test (TCP socket binding), rather move this one on a separate test function and put the full SSH connection test code instead. Is this feature interesting for the OpenStack community ? Should I create a blueprint on the Tempest project on Launchpad in order to provide my code through Gerrit ? On a second time, I plan to overhaul improve this “ssh_floating.py” script by clean the code a little bit, and add more cleaning code in order to avoid leaving instances/security groups/floating IP behind : I do have this kind of behavior right now and I already improved the teardown() on this way. Should I consider this code as a new functionality (thus create a blueprint) or should I create a defect and assign it to myself ? Cordialement / Best Regards, Julien LELOUP RD 3DExperience Platform IaaS Factory Technology Engineer julien.lel...@3ds.commailto:julien.lel...@3ds.com [cid:image003.gif@01CF1216.D43ECE20] 3DS.COMhttp://www.3ds.com/ Dassault Systèmes | 10 rue Marcel Dassault, CS 40501 | 78946 Vélizy-Villacoublay Cedex | France This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
Hello Marc, Thanks for your answer. At the moment I'm willing to spend some time on this kind of scenario so I will see how to use the stress decorator inside a scenario test. Does this means that all stress tests available in tempest/stress should be ported as scenario tests with this decorator ? I do have some ideas about features on stress test that I find useful for my own use case : like adding more statistics on stress test runs in order to use them as benchmarks. I don't know if this kind of feature was already discussed in the OpenStack community but since stress tests are a bit deprecated now, maybe there is some room for this kind of improvement on fresh stress tests. Best Regards, Julien LELOUP -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 9:45 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Juilen, I forwarded your mail to the correct mailing list. Please do not use the qa list any longer. I am happy that you are interested in stress tests. All the tests in tempest/stress/actions are more or less deprecated. So what you should use instead is the stress decorator (e.g. https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_actions.py#L55). Unfortunately it's not yet used for scenarios like you describe. I'd suggest to build a scenario test in tempest/scenario and use this decorator on it. Any patch like that is welcome on gerrit. If you are planning to work in that area for more than just a patch, a blueprint would be nice. A good way to coordinate your efforts is also in the QA meeting (https://wiki.openstack.org/wiki/Meetings/QATeamMeeting) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Wednesday, January 15, 2014 5:57 PM To: openstack...@lists.openstack.org Subject: [openstack-qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi everyone, I’m quite new on OpenStack / Tempest and I’m actually working on stress tests. I want to suggest a new feature in a currently available stress test. Not sure if this email should be posted on the QA mailing list or the dev mailing list, but I give it a try here since it is about a Tempest stress test ☺ At the moment the “ssh_floating.py” stress test seems really interesting but I would like to improve it a bit. By now this script is simulating an SSH connection by binding a TCP socket on the newly created instance. But this test don’t allow us to check if this instance is really available. I’m mostly thinking about the metadata service unable to provide the SSH key pair to the instance, but surely other scenarios can lead to an instance considered “ACTIVE” but actually unusable. So I’m implementing a full SSH connection test using the “paramiko” SSH library and a key pair generated in the same way the other test resources are managed in this script : either one SSH key pair for every test runs or a new key pair for each run (depends on the JSON configuration file). I don’t plan to remove the old test (TCP socket binding), rather move this one on a separate test function and put the full SSH connection test code instead. Is this feature interesting for the OpenStack community ? Should I create a blueprint on the Tempest project on Launchpad in order to provide my code through Gerrit ? On a second time, I plan to overhaul improve this “ssh_floating.py” script by clean the code a little bit, and add more cleaning code in order to avoid leaving instances/security groups/floating IP behind : I do have this kind of behavior right now and I already improved the teardown() on this way. Should I consider this code as a new functionality (thus create a blueprint) or should I create a defect and assign it to myself ? Cordialement / Best Regards, Julien LELOUP RD 3DExperience Platform IaaS Factory Technology Engineer julien.lel...@3ds.commailto:julien.lel...@3ds.com [cid:image003.gif@01CF1216.D43ECE20] 3DS.COMhttp://www.3ds.com/ Dassault Systèmes | 10 rue Marcel Dassault, CS 40501 | 78946 Vélizy-Villacoublay Cedex | France This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer This email and any attachments are intended solely for the use of the individual or entity
Re: [openstack-dev] [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
Hi Julien, most of the cases in tempest/stress are already covered by exiting tests in /api or /scenario. The only thing that is missing is the decorator on them. BTW here is the Etherpad from the summit talk that we had: https://etherpad.openstack.org/p/icehouse-summit-qa-stress-tests It possible help to understand the state. I didn't managed to work on the action items that are left. Your suggestions sound good. So I'd happy so see some patches :) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 11:52 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks for your answer. At the moment I'm willing to spend some time on this kind of scenario so I will see how to use the stress decorator inside a scenario test. Does this means that all stress tests available in tempest/stress should be ported as scenario tests with this decorator ? I do have some ideas about features on stress test that I find useful for my own use case : like adding more statistics on stress test runs in order to use them as benchmarks. I don't know if this kind of feature was already discussed in the OpenStack community but since stress tests are a bit deprecated now, maybe there is some room for this kind of improvement on fresh stress tests. Best Regards, Julien LELOUP -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 9:45 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Juilen, I forwarded your mail to the correct mailing list. Please do not use the qa list any longer. I am happy that you are interested in stress tests. All the tests in tempest/stress/actions are more or less deprecated. So what you should use instead is the stress decorator (e.g. https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_actions.py#L55). Unfortunately it's not yet used for scenarios like you describe. I'd suggest to build a scenario test in tempest/scenario and use this decorator on it. Any patch like that is welcome on gerrit. If you are planning to work in that area for more than just a patch, a blueprint would be nice. A good way to coordinate your efforts is also in the QA meeting (https://wiki.openstack.org/wiki/Meetings/QATeamMeeting) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Wednesday, January 15, 2014 5:57 PM To: openstack...@lists.openstack.org Subject: [openstack-qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi everyone, I’m quite new on OpenStack / Tempest and I’m actually working on stress tests. I want to suggest a new feature in a currently available stress test. Not sure if this email should be posted on the QA mailing list or the dev mailing list, but I give it a try here since it is about a Tempest stress test ☺ At the moment the “ssh_floating.py” stress test seems really interesting but I would like to improve it a bit. By now this script is simulating an SSH connection by binding a TCP socket on the newly created instance. But this test don’t allow us to check if this instance is really available. I’m mostly thinking about the metadata service unable to provide the SSH key pair to the instance, but surely other scenarios can lead to an instance considered “ACTIVE” but actually unusable. So I’m implementing a full SSH connection test using the “paramiko” SSH library and a key pair generated in the same way the other test resources are managed in this script : either one SSH key pair for every test runs or a new key pair for each run (depends on the JSON configuration file). I don’t plan to remove the old test (TCP socket binding), rather move this one on a separate test function and put the full SSH connection test code instead. Is this feature interesting for the OpenStack community ? Should I create a blueprint on the Tempest project on Launchpad in order to provide my code through Gerrit ? On a second time, I plan to overhaul improve this “ssh_floating.py” script by clean the code a little bit, and add more cleaning code in order to avoid leaving instances/security groups/floating IP behind : I do have this kind of behavior right now and I already improved the teardown() on this way. Should I consider this code as a new functionality (thus create a blueprint) or should I create a defect and assign it to myself ? Cordialement / Best Regards, Julien LELOUP RD 3DExperience Platform IaaS Factory Technology Engineer julien.lel...@3ds.commailto:julien.lel...@3ds.com [cid:image003.gif@01CF1216.D43ECE20] 3DS.COMhttp://www.3ds.com/ Dassault
Re: [openstack-dev] [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
On 01/17/2014 09:06 AM, Koderer, Marc wrote: Hi Julien, most of the cases in tempest/stress are already covered by exiting tests in /api or /scenario. The only thing that is missing is the decorator on them. BTW here is the Etherpad from the summit talk that we had: https://etherpad.openstack.org/p/icehouse-summit-qa-stress-tests It possible help to understand the state. I didn't managed to work on the action items that are left. Your suggestions sound good. So I'd happy so see some patches :) Regards Marc To clarify, it is still possible to have code for a stress test in tempest that is not a decorated scenario test. But such a stress test case would probably only be accepted if it were a good stress case and would be rejected as a regular scenario test for some reason. -David From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 11:52 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks for your answer. At the moment I'm willing to spend some time on this kind of scenario so I will see how to use the stress decorator inside a scenario test. Does this means that all stress tests available in tempest/stress should be ported as scenario tests with this decorator ? I do have some ideas about features on stress test that I find useful for my own use case : like adding more statistics on stress test runs in order to use them as benchmarks. I don't know if this kind of feature was already discussed in the OpenStack community but since stress tests are a bit deprecated now, maybe there is some room for this kind of improvement on fresh stress tests. Best Regards, Julien LELOUP -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 9:45 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Juilen, I forwarded your mail to the correct mailing list. Please do not use the qa list any longer. I am happy that you are interested in stress tests. All the tests in tempest/stress/actions are more or less deprecated. So what you should use instead is the stress decorator (e.g. https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_actions.py#L55). Unfortunately it's not yet used for scenarios like you describe. I'd suggest to build a scenario test in tempest/scenario and use this decorator on it. Any patch like that is welcome on gerrit. If you are planning to work in that area for more than just a patch, a blueprint would be nice. A good way to coordinate your efforts is also in the QA meeting (https://wiki.openstack.org/wiki/Meetings/QATeamMeeting) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Wednesday, January 15, 2014 5:57 PM To: openstack...@lists.openstack.org Subject: [openstack-qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi everyone, I’m quite new on OpenStack / Tempest and I’m actually working on stress tests. I want to suggest a new feature in a currently available stress test. Not sure if this email should be posted on the QA mailing list or the dev mailing list, but I give it a try here since it is about a Tempest stress test ☺ At the moment the “ssh_floating.py” stress test seems really interesting but I would like to improve it a bit. By now this script is simulating an SSH connection by binding a TCP socket on the newly created instance. But this test don’t allow us to check if this instance is really available. I’m mostly thinking about the metadata service unable to provide the SSH key pair to the instance, but surely other scenarios can lead to an instance considered “ACTIVE” but actually unusable. So I’m implementing a full SSH connection test using the “paramiko” SSH library and a key pair generated in the same way the other test resources are managed in this script : either one SSH key pair for every test runs or a new key pair for each run (depends on the JSON configuration file). I don’t plan to remove the old test (TCP socket binding), rather move this one on a separate test function and put the full SSH connection test code instead. Is this feature interesting for the OpenStack community ? Should I create a blueprint on the Tempest project on Launchpad in order to provide my code through Gerrit ? On a second time, I plan to overhaul improve this “ssh_floating.py” script by clean the code a little bit, and add more cleaning code in order to avoid leaving instances/security groups/floating IP behind : I do have this kind of behavior right now and I already improved the teardown() on this way. Should I consider this code as a new
