Re: Automatic Smoketests for the Cloud Images: What to Test?
Sandro \red\ Mathys r...@fedoraproject.org writes:
On Fri, Mar 7, 2014 at 8:20 PM, Vitaly Kuznetsov vi...@redhat.com wrote:
Sandro \red\ Mathys r...@fedoraproject.org writes:
On Fri, Mar 7, 2014 at 7:32 PM, Vitaly Kuznetsov vi...@redhat.com wrote:
So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone
of the people involved (at the Red Hat side, I guess) well me why we
have 3 systems for 1 task?
(my personal opinion) I think we rather have plenty of tasks, not
one. Afaict (after 5 min. of reading Taskotron's development plan
https://fedoraproject.org/wiki/User:Tflink/taskotron_development_plan)
Taskotron is designed to replace AutoQA in the first place.
RHEL's Cloud Image Validation was developed several years ago when the
following task was on the table: we have many AWS regions, many images,
different architectures, we need to try different hardware types and
AWS-specific features (e.g. attach EBS on the fly or test AWS-specific
content delivery) and finally we need to aggregate the result. Existing
test infrastructure was built around Beaker which is not that well
suited for the job and creating a separate tool was considered a
reasonable trade-off.
Well, one task as in do cloud image QA.
Thanks, for sharing that insights, really helpful to help my
understanding. So, do you currently test EC2 only? (Not saying that's
necessarily bad / too little).
Now it is EC2-only but Google's ComputeEngine was on the horizon.
Now, we do have the RHQE stuff in place and it's already used for
testing Fedora images...that's good. Is that fully automated? Or to
what extend?
You run the tool with the data (AMI IDs, region, arch) and get the
result in a meanwhile. It can be fully-automated once we have this data
announced via fedmsg or in any other automated way (now I just read
mailing list and if there are any images announced by Dennis I run the
tool).
When I took ownership of this external
need (for the Fedora cloud product) I was under the impression we
only just (are going to) have Taskotron and everyone knows it's THE
way to go.
I personally love collaboration. It would be awesome if we could avoid
spreading resources on '3 systems for 1 task'. I definitely want to know
more about Taskotron and its movement towards cloud image testing.
That's why I was a bit confused to find there's actually 3 systems.
Collaboration is certainly great, but that's not how it's done so
let's try to improve on this.
So, would you recommend to keep using your tools or rather go with
Taskotron? Or do we do some things in one and others in the other? Or
do we try to fully implement your tests in Taskotron and drop doing
the tests with your tools?
Well, it depends on what's our future plan. IMHO once we have images
announced via fedmsg we can have all basic things covered by the existing
tool (and I'm definitely in for integration and support process for the
tool) and it won't take us long to set everything up. With regards to
Taskotron I want to know more on how this 'cloud integration' is planned
as (if I'm not mistaken) there's no code written yet. If merging here
seems reasonable then I'm in. I'll try reaching out to Tim others on
fedora-qa-devel list.
So, what's the status here? Tim's responses to this thread show no
cloud integration code has been written yet and he's open to have
valid integrated in Taskotron, particularly if helping hands do most
of the work so he can keep focusing on other open tasks. Could you
work on that, Vitaly?
I'm still willing to but I have to confess this task ('Take a look at
Taskotron and figure out how me can merge / collaborate / integrate /
whatever') is still in my backlog. I'll try picking this up next week,
feel free to ping me on irc (vitty) any time.
Also, Karanbir, what's your (i.e. CentOS's) story? You say you already
have a CI system running but shared little other information. What CI
system? Did you already implement image tests? What kind of
collaboration would you suggest here?
--
Vitaly Kuznetsov
___
cloud mailing list
cloud@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/cloud
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
I said I would help with this but someone else took the lead. What's the 411 haha??? On Apr 4, 2014 12:34 AM, Sandro red Mathys r...@fedoraproject.org wrote: On Fri, Mar 7, 2014 at 8:20 PM, Vitaly Kuznetsov vi...@redhat.com wrote: Sandro \red\ Mathys r...@fedoraproject.org writes: On Fri, Mar 7, 2014 at 7:32 PM, Vitaly Kuznetsov vi...@redhat.com wrote: So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone of the people involved (at the Red Hat side, I guess) well me why we have 3 systems for 1 task? (my personal opinion) I think we rather have plenty of tasks, not one. Afaict (after 5 min. of reading Taskotron's development plan https://fedoraproject.org/wiki/User:Tflink/taskotron_development_plan) Taskotron is designed to replace AutoQA in the first place. RHEL's Cloud Image Validation was developed several years ago when the following task was on the table: we have many AWS regions, many images, different architectures, we need to try different hardware types and AWS-specific features (e.g. attach EBS on the fly or test AWS-specific content delivery) and finally we need to aggregate the result. Existing test infrastructure was built around Beaker which is not that well suited for the job and creating a separate tool was considered a reasonable trade-off. Well, one task as in do cloud image QA. Thanks, for sharing that insights, really helpful to help my understanding. So, do you currently test EC2 only? (Not saying that's necessarily bad / too little). Now it is EC2-only but Google's ComputeEngine was on the horizon. Now, we do have the RHQE stuff in place and it's already used for testing Fedora images...that's good. Is that fully automated? Or to what extend? You run the tool with the data (AMI IDs, region, arch) and get the result in a meanwhile. It can be fully-automated once we have this data announced via fedmsg or in any other automated way (now I just read mailing list and if there are any images announced by Dennis I run the tool). When I took ownership of this external need (for the Fedora cloud product) I was under the impression we only just (are going to) have Taskotron and everyone knows it's THE way to go. I personally love collaboration. It would be awesome if we could avoid spreading resources on '3 systems for 1 task'. I definitely want to know more about Taskotron and its movement towards cloud image testing. That's why I was a bit confused to find there's actually 3 systems. Collaboration is certainly great, but that's not how it's done so let's try to improve on this. So, would you recommend to keep using your tools or rather go with Taskotron? Or do we do some things in one and others in the other? Or do we try to fully implement your tests in Taskotron and drop doing the tests with your tools? Well, it depends on what's our future plan. IMHO once we have images announced via fedmsg we can have all basic things covered by the existing tool (and I'm definitely in for integration and support process for the tool) and it won't take us long to set everything up. With regards to Taskotron I want to know more on how this 'cloud integration' is planned as (if I'm not mistaken) there's no code written yet. If merging here seems reasonable then I'm in. I'll try reaching out to Tim others on fedora-qa-devel list. So, what's the status here? Tim's responses to this thread show no cloud integration code has been written yet and he's open to have valid integrated in Taskotron, particularly if helping hands do most of the work so he can keep focusing on other open tasks. Could you work on that, Vitaly? Also, Karanbir, what's your (i.e. CentOS's) story? You say you already have a CI system running but shared little other information. What CI system? Did you already implement image tests? What kind of collaboration would you suggest here? ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
On Fri, Apr 4, 2014 at 1:35 PM, Tim Ski marshy...@gmail.com wrote: I said I would help with this but someone else took the lead. What's the 411 haha??? Where did you say so? I haven't seen any such message and neither did the cloud list's archive. Anyway, what exactly would you like to help with? On Apr 4, 2014 12:34 AM, Sandro red Mathys r...@fedoraproject.org wrote: On Fri, Mar 7, 2014 at 8:20 PM, Vitaly Kuznetsov vi...@redhat.com wrote: Sandro \red\ Mathys r...@fedoraproject.org writes: On Fri, Mar 7, 2014 at 7:32 PM, Vitaly Kuznetsov vi...@redhat.com wrote: So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone of the people involved (at the Red Hat side, I guess) well me why we have 3 systems for 1 task? (my personal opinion) I think we rather have plenty of tasks, not one. Afaict (after 5 min. of reading Taskotron's development plan https://fedoraproject.org/wiki/User:Tflink/taskotron_development_plan) Taskotron is designed to replace AutoQA in the first place. RHEL's Cloud Image Validation was developed several years ago when the following task was on the table: we have many AWS regions, many images, different architectures, we need to try different hardware types and AWS-specific features (e.g. attach EBS on the fly or test AWS-specific content delivery) and finally we need to aggregate the result. Existing test infrastructure was built around Beaker which is not that well suited for the job and creating a separate tool was considered a reasonable trade-off. Well, one task as in do cloud image QA. Thanks, for sharing that insights, really helpful to help my understanding. So, do you currently test EC2 only? (Not saying that's necessarily bad / too little). Now it is EC2-only but Google's ComputeEngine was on the horizon. Now, we do have the RHQE stuff in place and it's already used for testing Fedora images...that's good. Is that fully automated? Or to what extend? You run the tool with the data (AMI IDs, region, arch) and get the result in a meanwhile. It can be fully-automated once we have this data announced via fedmsg or in any other automated way (now I just read mailing list and if there are any images announced by Dennis I run the tool). When I took ownership of this external need (for the Fedora cloud product) I was under the impression we only just (are going to) have Taskotron and everyone knows it's THE way to go. I personally love collaboration. It would be awesome if we could avoid spreading resources on '3 systems for 1 task'. I definitely want to know more about Taskotron and its movement towards cloud image testing. That's why I was a bit confused to find there's actually 3 systems. Collaboration is certainly great, but that's not how it's done so let's try to improve on this. So, would you recommend to keep using your tools or rather go with Taskotron? Or do we do some things in one and others in the other? Or do we try to fully implement your tests in Taskotron and drop doing the tests with your tools? Well, it depends on what's our future plan. IMHO once we have images announced via fedmsg we can have all basic things covered by the existing tool (and I'm definitely in for integration and support process for the tool) and it won't take us long to set everything up. With regards to Taskotron I want to know more on how this 'cloud integration' is planned as (if I'm not mistaken) there's no code written yet. If merging here seems reasonable then I'm in. I'll try reaching out to Tim others on fedora-qa-devel list. So, what's the status here? Tim's responses to this thread show no cloud integration code has been written yet and he's open to have valid integrated in Taskotron, particularly if helping hands do most of the work so he can keep focusing on other open tasks. Could you work on that, Vitaly? Also, Karanbir, what's your (i.e. CentOS's) story? You say you already have a CI system running but shared little other information. What CI system? Did you already implement image tests? What kind of collaboration would you suggest here? ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
On Fri, 7 Mar 2014 18:46:49 +0900 Sandro \red\ Mathys r...@fedoraproject.org wrote: snip So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone of the people involved (at the Red Hat side, I guess) well me why we have 3 systems for 1 task? When I took ownership of this external need (for the Fedora cloud product) I was under the impression we only just (are going to) have Taskotron and everyone knows it's THE way to go. Honestly, I think you're comparing apples and oranges here with Taskotron and valid. On the one hand you have systems like Taskotron which are (or will/can be) capable of running any number of things from package-level checks like rpmlint or depcheck, cloud image checks of the type that we're discussing here, graphical installation media checks with an anaconda install image, interface with beaker to run various other checks, run static analysis on SRPMs and just about anything else that you can think of. On the other hand, you have valid. As far as I can tell, it was designed to do one thing (test cloud images) and do that one thing well. I'm not saying there's anything wrong with that but saying that the two systems do the same thing isn't really accurate. As far as CentOS's CI system goes, I don't claim to know much about it. It sounds similar to what we're planning for Taskotron but I wasn't able to find many details about how it manages reporting, how jobs are triggered or what the interface between the tests and the jenkins runner is. At its core, Taskotron is a system for triggering somewhat-arbitrary tasks, executing those tasks and reporting the generated output. Our goal is to provide a framework with which Fedora developers can automate tasks to be run in specific situations. Is it capable of everything right now? No, it isn't. We're in the process of a major re-write and working off about 2 years of technical debt so as much as I'd like reality to be different, this is going to take some time and/or external help to get new features added and working. I'm not dead set on writing a bunch of custom code for running cloud image checks. In fact, I'd rather not do that if we don't have to. If it makes more sense to run RH QE's cloud stuff, then that sounds like a decent integration point with Taskotron unless there is other HW available for running them. I'm still assuming that there is no existing code for getting the generated cloud images from Koji into a runnable state which we can use (unless someone's got a bunch of usable EC2 credit that I don't know about), so some work will still be needed on that front. I purposely left the part of actually running any individual checks open-ended because I assume that you all will come up with some method for checking the things that you're interested in. If you all decide that you want to use valid for cloud image checks, let's start looking at what it would take to get that integrated. Tim signature.asc Description: PGP signature ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
On Fri, 7 Mar 2014 19:46:28 +0900 Sandro \red\ Mathys r...@fedoraproject.org wrote: snip When I took ownership of this external need (for the Fedora cloud product) I was under the impression we only just (are going to) have Taskotron and everyone knows it's THE way to go. I personally love collaboration. It would be awesome if we could avoid spreading resources on '3 systems for 1 task'. I definitely want to know more about Taskotron and its movement towards cloud image testing. That's why I was a bit confused to find there's actually 3 systems. Collaboration is certainly great, but that's not how it's done so let's try to improve on this. So, would you recommend to keep using your tools or rather go with Taskotron? Or do we do some things in one and others in the other? Or do we try to fully implement your tests in Taskotron and drop doing the tests with your tools? I still don't understand why this is an either/or. As I mentioned earlier, Taskotron and valid do not seem to be trying to accomplish the same goals. From what I can see, they should be able to be integrated without too much trouble and I strongly suspect that level of effort would be far less than setting up separate systems. Tim signature.asc Description: PGP signature ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
Sandro \red\ Mathys r...@fedoraproject.org writes: Heads-up: I've taken ownership for the external need Automatic Smoketests on Image Build [0] Testing an image takes time and resources, and with several images, it takes several times that. Since that simply doesn't scale - well, doesn't even work well without scaling - we want to automate the smoketests using Taskotron. Unfortunately, Taskotron is still in a very early development phase and having mentioned only just our (most) basic needs, Tim estimates it won't be good enough for us in the target timeframe (i.e. by F21 Alpha). There's a slight chance it will be ready, but even then we still need time to implement the smoketests. I just want to point out the fact that for RHEL/Fedora/etc image testing we have the following tool: https://github.com/RedHatQE/valid We were using it to test Fedora images since F18. To be honest I'm not that familiar with Taskotron but maybe we can collaborate somehow. So if we want to have Taskotron ready for us in time, we need to find some developer(s) committing some of their resources. Any takers? If you're interested, head to [1] to learn more, there's also a contribution guide linked from there. I'm sure tflink in #fedora-qa is happy to talk to people still interested after reading through the wiki if there's further questions. Also, we need to compile our actual needs. What kind of tests do we want to run? Most basic tests (things that we manually tested in the past) are probably: - Was the IP correctly set up? - Is SSH running and reachable? - Does the fedora user exist? - Is the ssh pubkey installed for the fedora user? - Is the fedora user able to use sudo? - Is the root account locked? - Is package installation working? - Is the firewall disabled? - Checking cloud-init and systemctl for bootup errors - does that make sense or is there too much noise? What else can / should we test? Obviously the tailored images should be tested for their specific stuff but lets first focus on the common ground. In addition to that current image validation has the following: - testing different instace types - empty bash history - selinux contexts test - running services - package set - EBS test - Ephemeral test - available cores/memory test - userdata handling (cloud-init) - several bugs test (regressions) - ... (full test list is here: https://github.com/RedHatQE/valid/tree/master/valid/testing_modules) Thanks, Sandro [0] https://fedoraproject.org/wiki/Cloud_Changelist#External_Need:_Automatic_Smoketests_on_Image_Build [1] https://fedoraproject.org/wiki/Taskotron ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- Vitaly Kuznetsov ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
Sandro \red\ Mathys r...@fedoraproject.org writes: On Fri, Mar 7, 2014 at 6:37 PM, Vitaly Kuznetsov vi...@redhat.com wrote: Sandro \red\ Mathys r...@fedoraproject.org writes: Heads-up: I've taken ownership for the external need Automatic Smoketests on Image Build [0] Testing an image takes time and resources, and with several images, it takes several times that. Since that simply doesn't scale - well, doesn't even work well without scaling - we want to automate the smoketests using Taskotron. Unfortunately, Taskotron is still in a very early development phase and having mentioned only just our (most) basic needs, Tim estimates it won't be good enough for us in the target timeframe (i.e. by F21 Alpha). There's a slight chance it will be ready, but even then we still need time to implement the smoketests. I just want to point out the fact that for RHEL/Fedora/etc image testing we have the following tool: https://github.com/RedHatQE/valid We were using it to test Fedora images since F18. To be honest I'm not that familiar with Taskotron but maybe we can collaborate somehow. So if we want to have Taskotron ready for us in time, we need to find some developer(s) committing some of their resources. Any takers? If you're interested, head to [1] to learn more, there's also a contribution guide linked from there. I'm sure tflink in #fedora-qa is happy to talk to people still interested after reading through the wiki if there's further questions. Also, we need to compile our actual needs. What kind of tests do we want to run? Most basic tests (things that we manually tested in the past) are probably: - Was the IP correctly set up? - Is SSH running and reachable? - Does the fedora user exist? - Is the ssh pubkey installed for the fedora user? - Is the fedora user able to use sudo? - Is the root account locked? - Is package installation working? - Is the firewall disabled? - Checking cloud-init and systemctl for bootup errors - does that make sense or is there too much noise? What else can / should we test? Obviously the tailored images should be tested for their specific stuff but lets first focus on the common ground. In addition to that current image validation has the following: - testing different instace types - empty bash history - selinux contexts test - running services - package set - EBS test - Ephemeral test - available cores/memory test - userdata handling (cloud-init) - several bugs test (regressions) - ... (full test list is here: https://github.com/RedHatQE/valid/tree/master/valid/testing_modules) Thanks, Sandro [0] https://fedoraproject.org/wiki/Cloud_Changelist#External_Need:_Automatic_Smoketests_on_Image_Build [1] https://fedoraproject.org/wiki/Taskotron So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone of the people involved (at the Red Hat side, I guess) well me why we have 3 systems for 1 task? (my personal opinion) I think we rather have plenty of tasks, not one. Afaict (after 5 min. of reading Taskotron's development plan https://fedoraproject.org/wiki/User:Tflink/taskotron_development_plan) Taskotron is designed to replace AutoQA in the first place. RHEL's Cloud Image Validation was developed several years ago when the following task was on the table: we have many AWS regions, many images, different architectures, we need to try different hardware types and AWS-specific features (e.g. attach EBS on the fly or test AWS-specific content delivery) and finally we need to aggregate the result. Existing test infrastructure was built around Beaker which is not that well suited for the job and creating a separate tool was considered a reasonable trade-off. When I took ownership of this external need (for the Fedora cloud product) I was under the impression we only just (are going to) have Taskotron and everyone knows it's THE way to go. I personally love collaboration. It would be awesome if we could avoid spreading resources on '3 systems for 1 task'. I definitely want to know more about Taskotron and its movement towards cloud image testing. -- Vitaly Kuznetsov ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
On Fri, Mar 7, 2014 at 3:50 AM, Frankie frankie.onuo...@gmail.com wrote: I would love to assist with this. Kindly let me know how I can be of best use. Frankie, if you feel up to it, helping with the Taskotron development as mentioned in the first post would be highly appreciated. See also Tim Flink's post (the one right after yours). -- Sandro ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
Sandro \red\ Mathys r...@fedoraproject.org writes: On Fri, Mar 7, 2014 at 7:32 PM, Vitaly Kuznetsov vi...@redhat.com wrote: So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone of the people involved (at the Red Hat side, I guess) well me why we have 3 systems for 1 task? (my personal opinion) I think we rather have plenty of tasks, not one. Afaict (after 5 min. of reading Taskotron's development plan https://fedoraproject.org/wiki/User:Tflink/taskotron_development_plan) Taskotron is designed to replace AutoQA in the first place. RHEL's Cloud Image Validation was developed several years ago when the following task was on the table: we have many AWS regions, many images, different architectures, we need to try different hardware types and AWS-specific features (e.g. attach EBS on the fly or test AWS-specific content delivery) and finally we need to aggregate the result. Existing test infrastructure was built around Beaker which is not that well suited for the job and creating a separate tool was considered a reasonable trade-off. Well, one task as in do cloud image QA. Thanks, for sharing that insights, really helpful to help my understanding. So, do you currently test EC2 only? (Not saying that's necessarily bad / too little). Now it is EC2-only but Google's ComputeEngine was on the horizon. Now, we do have the RHQE stuff in place and it's already used for testing Fedora images...that's good. Is that fully automated? Or to what extend? You run the tool with the data (AMI IDs, region, arch) and get the result in a meanwhile. It can be fully-automated once we have this data announced via fedmsg or in any other automated way (now I just read mailing list and if there are any images announced by Dennis I run the tool). When I took ownership of this external need (for the Fedora cloud product) I was under the impression we only just (are going to) have Taskotron and everyone knows it's THE way to go. I personally love collaboration. It would be awesome if we could avoid spreading resources on '3 systems for 1 task'. I definitely want to know more about Taskotron and its movement towards cloud image testing. That's why I was a bit confused to find there's actually 3 systems. Collaboration is certainly great, but that's not how it's done so let's try to improve on this. So, would you recommend to keep using your tools or rather go with Taskotron? Or do we do some things in one and others in the other? Or do we try to fully implement your tests in Taskotron and drop doing the tests with your tools? Well, it depends on what's our future plan. IMHO once we have images announced via fedmsg we can have all basic things covered by the existing tool (and I'm definitely in for integration and support process for the tool) and it won't take us long to set everything up. With regards to Taskotron I want to know more on how this 'cloud integration' is planned as (if I'm not mistaken) there's no code written yet. If merging here seems reasonable then I'm in. I'll try reaching out to Tim others on fedora-qa-devel list. Also, Karanbir, what's your (i.e. CentOS's) story? You say you already have a CI system running but shared little other information. What CI system? Did you already implement image tests? What kind of collaboration would you suggest here? -- Sandro ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- Vitaly Kuznetsov ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
On Fri, Mar 07, 2014 at 07:46:28PM +0900, Sandro red Mathys wrote: That's why I was a bit confused to find there's actually 3 systems. Collaboration is certainly great, but that's not how it's done so let's try to improve on this. Speaking without any sort of officialness, I think the split is largely the result of historical reasons for keeping high walls between Fedora, RHEL, and CentOS. Those reasons are changing, and we have the opportunity to work together where we were restricted before. -- Matthew Miller-- Fedora Project--mat...@fedoraproject.org ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I would love to assist with this. Kindly let me know how I can be of best use. Kind Regards, On 03/05/2014 02:58 PM, Sandro red Mathys wrote: Heads-up: I've taken ownership for the external need Automatic Smoketests on Image Build [0] Testing an image takes time and resources, and with several images, it takes several times that. Since that simply doesn't scale - well, doesn't even work well without scaling - we want to automate the smoketests using Taskotron. Unfortunately, Taskotron is still in a very early development phase and having mentioned only just our (most) basic needs, Tim estimates it won't be good enough for us in the target timeframe (i.e. by F21 Alpha). There's a slight chance it will be ready, but even then we still need time to implement the smoketests. So if we want to have Taskotron ready for us in time, we need to find some developer(s) committing some of their resources. Any takers? If you're interested, head to [1] to learn more, there's also a contribution guide linked from there. I'm sure tflink in #fedora-qa is happy to talk to people still interested after reading through the wiki if there's further questions. Also, we need to compile our actual needs. What kind of tests do we want to run? Most basic tests (things that we manually tested in the past) are probably: - Was the IP correctly set up? - Is SSH running and reachable? - Does the fedora user exist? - Is the ssh pubkey installed for the fedora user? - Is the fedora user able to use sudo? - Is the root account locked? - Is package installation working? - Is the firewall disabled? - Checking cloud-init and systemctl for bootup errors - does that make sense or is there too much noise? What else can / should we test? Obviously the tailored images should be tested for their specific stuff but lets first focus on the common ground. Thanks, Sandro [0] https://fedoraproject.org/wiki/Cloud_Changelist#External_Need:_Automatic_Smoketests_on_Image_Build [1] https://fedoraproject.org/wiki/Taskotron ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJTGMOJAAoJEJGS9+sdPu4ECvkP/Rdn+E9VN3ile16aIXbBvHq8 XF0AStwIvW96/0H8EwTnqxDvbDaxGmPHRyLyXR6e8giizdGb46NIX9xFb0+rIzSi 2/8wQ7mI4Q2DQQ5VPsU4OAU3upw1CZqoGGYZEB7RphTjX4kMqKY6uVQ83ZU660gl ANvDb5tSebtpd31p9Sii1e5TCuRncWJzvwXvtvE98RJaLWZZk1Rp8Zg22Imdvma6 KqzEbswNYMQC0ap6hKBKltIr622QaXFWhXES8EOK5P5eYE0OKZApDi053GyHNoZS 9i6MheJEcEhOl1MJ4bdh14LccQdk0RdxvxA6LdArLT/wZJK7GMeuFS9FSbtf+qLc c1hO9jQ2guApvB78Z+iteHEygRExeThCQtpp6FV52FvRjl3VqC7uJLYo+ozXbDTb mX/8eVOAhoVZn/EFHN73kWPb/46NQyjzrY2qNCLtksAj05SMxp7N2insREL7CjNS EPWKgmnFnrFaCYhLhPRpJDa9QnlIe6ZuBAM3FNSm6OSxtcc/sajGAY8aBtzVBwhW pwa45iRloIJResZ214+5guhj0JvQHsft1izWvpbSdjREqJ83s2cDFZ++o346BZjw OanE1LRhTBILZliByzROrAEWPuAeGnUDTrxlJmqYhRrCQrVpCQ2r1DsTH4seoQKO q86qF6u52XfCtIDSlfD4 =lwvT -END PGP SIGNATURE- ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
On Wed, 5 Mar 2014 20:58:32 +0900 Sandro \red\ Mathys r...@fedoraproject.org wrote: Heads-up: I've taken ownership for the external need Automatic Smoketests on Image Build [0] Testing an image takes time and resources, and with several images, it takes several times that. Since that simply doesn't scale - well, doesn't even work well without scaling - we want to automate the smoketests using Taskotron. Unfortunately, Taskotron is still in a very early development phase and having mentioned only just our (most) basic needs, Tim estimates it won't be good enough for us in the target timeframe (i.e. by F21 Alpha). There's a slight chance it will be ready, but even then we still need time to implement the smoketests. So if we want to have Taskotron ready for us in time, we need to find some developer(s) committing some of their resources. Any takers? If you're interested, head to [1] to learn more, there's also a contribution guide linked from there. I'm sure tflink in #fedora-qa is happy to talk to people still interested after reading through the wiki if there's further questions. To go into a bit more detail, the taskotron bits which would be required would depend a little bit on what comes out of the cloud image generation. I'm assuming the following: - the image creation process will generate a fedmsg upon completion that's relatively simple to differentiate from other fedmsgs. - this fedmsg contains some unique information which can be used to obtain a path to the generated image - the image is not already being loaded into a cloud-ish system - some cloud-ish system (ie openstack) is available for testing If my assumptions are correct, the scheduling component would be a relatively simple change. Taskotron would need one or two new modules to do the following: - take the fedmsg information, upload the desired image into the cloud-ish system and return the image id - launch an instance with a specified image id, flavor and key id. return the instance's IP address There would be a complication here in that Taskotron doesn't have a concept of shared secrets for the ssh private key or but I'm sure we could figure something out. Also, we need to compile our actual needs. What kind of tests do we want to run? Most basic tests (things that we manually tested in the past) are probably: - Was the IP correctly set up? - Is SSH running and reachable? - Does the fedora user exist? - Is the ssh pubkey installed for the fedora user? - Is the fedora user able to use sudo? - Is the root account locked? - Is package installation working? - Is the firewall disabled? - Checking cloud-init and systemctl for bootup errors - does that make sense or is there too much noise? The easiest way to write a task to check stuff like this would be to write it in python. Input would be a key id (see the shared secret comment above) and a user id. The task would check the items that you're interested in, collate and return the results. Taskotron would then report results and emit a fedmsg indicating said result (fedmsg code is yet to be completed but is slated for our first deliverable). I realize that the documentation for how to do this is non-existent at the moment. We've been focusing on getting the details figured out and a staging system up and running before getting howtos written. We have a trivial example written now (rpmlint) and will have a more complicated example (depcheck) mostly complete before the end of next week. While those are indeed package-level checks, they will have examples of the task definition format (YAML) and the interface needed for tasks written in python. I hope this is enough detail to at least outline what would need to be done. I'd love to see it done but I'm not sure what our (qa-devel) priorities are going to be once the base Taskotron system is in place and thus, not sure if it would otherwise be done before F21-alpha. Please let me know if you have any questions, I'm happy to help where I can. Tim signature.asc Description: PGP signature ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
On Thu, Mar 06, 2014 at 06:44:29PM -0700, Tim Flink wrote: To go into a bit more detail, the taskotron bits which would be required would depend a little bit on what comes out of the cloud image generation. I'm assuming the following: - the image creation process will generate a fedmsg upon completion that's relatively simple to differentiate from other fedmsgs. Yes, that's the plan. - this fedmsg contains some unique information which can be used to obtain a path to the generated image Ditto. - the image is not already being loaded into a cloud-ish system The plan currently is to upload it into EC2 and also make the qcow2 available via FTP. - some cloud-ish system (ie openstack) is available for testing I assume we can do that. - launch an instance with a specified image id, flavor and key id. return the instance's IP address There would be a complication here in that Taskotron doesn't have a concept of shared secrets for the ssh private key or but I'm sure we could figure something out. Generally, the public key is held by the cloud service (openstack or ec2 or whatever) and we'd need a way for Taskotron to keep the private key private. -- Matthew Miller-- Fedora Project--mat...@fedoraproject.org ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Automatic Smoketests for the Cloud Images: What to Test?
Heads-up: I've taken ownership for the external need Automatic Smoketests on Image Build [0] Testing an image takes time and resources, and with several images, it takes several times that. Since that simply doesn't scale - well, doesn't even work well without scaling - we want to automate the smoketests using Taskotron. Unfortunately, Taskotron is still in a very early development phase and having mentioned only just our (most) basic needs, Tim estimates it won't be good enough for us in the target timeframe (i.e. by F21 Alpha). There's a slight chance it will be ready, but even then we still need time to implement the smoketests. So if we want to have Taskotron ready for us in time, we need to find some developer(s) committing some of their resources. Any takers? If you're interested, head to [1] to learn more, there's also a contribution guide linked from there. I'm sure tflink in #fedora-qa is happy to talk to people still interested after reading through the wiki if there's further questions. Also, we need to compile our actual needs. What kind of tests do we want to run? Most basic tests (things that we manually tested in the past) are probably: - Was the IP correctly set up? - Is SSH running and reachable? - Does the fedora user exist? - Is the ssh pubkey installed for the fedora user? - Is the fedora user able to use sudo? - Is the root account locked? - Is package installation working? - Is the firewall disabled? - Checking cloud-init and systemctl for bootup errors - does that make sense or is there too much noise? What else can / should we test? Obviously the tailored images should be tested for their specific stuff but lets first focus on the common ground. Thanks, Sandro [0] https://fedoraproject.org/wiki/Cloud_Changelist#External_Need:_Automatic_Smoketests_on_Image_Build [1] https://fedoraproject.org/wiki/Taskotron ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
On 03/05/2014 05:55 PM, Matthew Miller wrote: - Checking cloud-init and systemctl for bootup errors - does that make sense or is there too much noise? I started a related list at https://fedorahosted.org/cloud/ticket/4 How can we work on this in a manner to sync with the CentOS effort on the same front ? we've started working on a lightweight branch of the functional test suite: specifically for this instance testing. Plus, we already have infra up to run a CI setup, and we run multiple on-premise cloud controllers with various hypervisors, as well as interface with multiple public cloud vendors. - KB -- Karanbir Singh +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh GnuPG Key : http://www.karan.org/publickey.asc ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Automatic Smoketests for the Cloud Images: What to Test?
On Wed, Mar 5, 2014 at 6:58 AM, Sandro red Mathys r...@fedoraproject.org wrote: Heads-up: I've taken ownership for the external need Automatic Smoketests on Image Build [0] Testing an image takes time and resources, and with several images, it takes several times that. Since that simply doesn't scale - well, doesn't even work well without scaling - we want to automate the smoketests using Taskotron. rpm-ostree has included in it an (unnamed) custom qemu-based testing hypervisor, which derives from gnome-continuous. Most of the code is in here: https://github.com/cgwalters/rpm-ostree/blob/master/src/autobuilder/js/tasks/testbase.js What makes the rpm-ostree testing hypervisor fundamentally different from a lot of other testing systems out there is it works without IP networking. Basically: 1) Use libguestfs to inject testing code into the guest 2) The host offers a dbus-over-virtio-serial channel for guests to request things like screenshots 3) Watch the systemd journal from the guest for events Again, operating without guest IP networking and having the ability to watch early boot events like the console.log and systemd journal are key. Where I'm going with this eventually is that this testing code acts as a basic smoketest, performed on the same physical host as the image (tree) composition servers. If that passes, then we have an OS version that can be sanely provisioned at a larger scale - say to OpenStack, or to things like AutoTest/Beaker physical hardware inventory/provisioning systems. For example the basic smoketest: https://github.com/cgwalters/rpm-ostree/blob/master/src/autobuilder/js/tasks/task-smoketest.js All it does is watch the systemd journal for two events. If we reached multi-user.target, we're good. If any service crashed, we failed. This testing code is rather deeply intertwined with the larger rpm-ostree/ostree deployment model - but I'd consider patches to help test traditional RPM-on-the-client OSes. (Really it's not even *that* much code, it wouldn't be too hard to fork/rewrite it) ___ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
