[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Marco Ceppi: Work items changed: Work items: [marcoceppi] Integration Testing includes framework that charm authors can write tests against (embedded in the charm) (amulet): INPROGRESS [marcoceppi] Jenkins testing on new merge proposal, on success it is a candidate for review: POSTPONED [marcoceppi] Develop Juju test plugin (ie juju test): DONE [marcoceppi] Modify charm tools to capture a stub integration test: INPROGRESS - Include charm-helper library for all charmers: TODO + Include charm-helper library for all charmers: DONE -- Juju Charm Testing https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-juju-charm-testing -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Marco Ceppi: Work items changed: Work items: [marcoceppi] Integration Testing includes framework that charm authors can write tests against (embedded in the charm) (amulet): INPROGRESS - [marcoceppi] Jenkins testing on new merge proposal, on success it is a candidate for review: TODO + [marcoceppi] Jenkins testing on new merge proposal, on success it is a candidate for review: POSTPONED [marcoceppi] Develop Juju test plugin (ie juju test): DONE [marcoceppi] Modify charm tools to capture a stub integration test: INPROGRESS Include charm-helper library for all charmers: TODO -- Juju Charm Testing https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-juju-charm-testing -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Marco Ceppi: Work items changed: Work items: - Integration Testing includes framework that charm authors can write tests against (embedded in the charm): TODO - Jenkins testing on new merge proposal, on success it is a candidate for review: TODO - Develop Juju test plugin (ie juju test): TODO - Modify charm tools to capture a stub integration test: TODO + [marcoceppi] Integration Testing includes framework that charm authors can write tests against (embedded in the charm) (amulet): INPROGRESS + [marcoceppi] Jenkins testing on new merge proposal, on success it is a candidate for review: TODO + [marcoceppi] Develop Juju test plugin (ie juju test): DONE + [marcoceppi] Modify charm tools to capture a stub integration test: INPROGRESS Include charm-helper library for all charmers: TODO -- Juju Charm Testing https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-juju-charm-testing -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Antonio Rosales: Whiteboard changed: Discussion: *Bug submission on charm failure. *Define a process around how charm maintainers respond to test failures and subsequent bugs. Can a user run a manual test and submit the test back to the bug report to update testing status to green. *Enable Autocharm tester to be more resilient against provider failures, and Jenkins usage. simulate provider failure, and be able to recover: $ juju ssh MACHINE sudo shutdown now * Define WIs to execute auto charm testing on Go. * Continuous Integration (also will help with gating on charm commits) * Juju Testing Blogging * Juju testing communication to Juju lists. * Work on integrating/fixing Charm runner (graph testing/ dependency/env set up testing). Add a Jenkins workflow to run a charm or a set of charms in the following LXC environments: -raring container on raring host -raring container on precise host -precise container on raring host -precise container on precise host Two modes of testing: -Unit (does the charm start, and report ready) -Workload (test the charms relations, and pushing data) Reference Links: *Charm Test Spec [html] https://juju.ubuntu.com/docs/charm-tests.html * Charm Test Spec [source] http://bazaar.launchpad.net/~juju/juju/docs/view/head:/source/charm-tests.rst * CharmTester Charm http://jujucharms.com/~mark-mims/oneiric/charmtester * Charm Runner: https://launchpad.net/charmrunner * Jenkins Charm Testing: https://jenkins.qa.ubuntu.com/view/Charms/ [USER STORIES] William is a juju user who wishes to know a charms current stability Saul is patching a charm and wants to in sure his changes are work with current tests Laura is a charm maintainer and wants to add tests to in sure her charm is stable Kara is a charm maintainer and needs to know when her charm is broken Lee is a charmer who, while reviewing charm submissions, needs to know if these changes break backwards compatibility with currently deployed services Gaius is a charm maintainer from an upstream project and needs an easy way to learn how to write tests for his charm [ASSUMPTIONS] - Charm tester/charm tester control will work with gojuju for at least graph testing [RISKS] - Relying soley on graph testing may result in inaccurate test results due to lack of embedded tests - Making tests too complicated may result in low adoption rate of embedded testing [IN SCOPE] [OUT OF SCOPE] [USER ACCEPTANCE] [RELEASE NOTE/BLOG] - (Needs spec and WI definition) -[a.rosales; 12-DEC-2012] + [Notes] + + UDS 1305 notes: http://pad.ubuntu.com/ep/pad/view/uds-1305-servercloud-s + -juju-charm-testing/latest === UDS 1303 Notes === Pad: http://pad.ubuntu.com/uds-1303-servercloud-r-juju-charm-testing Question: Is there a way in meta-data to explicitly state provider support. -Example: Ceph: Does cloud provider have block support -More broadly stated does the cloud provider have the capabilities the charm needs Idea: -In charm testing status be able to show that a charm failure can be a result of the provider not providing the needed capabilities, ie the Ceph charm fails on a provider because it does not support object store. -Make interface usage more verbose in the charm description. -Need a rule/spec on how a interface should be implemented -Need to investigate possible enforment of interfaces -**Have the testing framework iterate through the operational deployment requirments. Interfaces doc link broken: -Example: http://jujucharms.com/interfaces/ceph-client Interface doc link broken: https://juju.ubuntu.com/Interfaces/ceph-client -- broken Meta-language testing (http://paste.ubuntu.com/5588570/): Lanugage suggestions: http://lettuce.it/ http://cukes.info/ Charm-Runner integration: - https://launchpad.net/juju-deployer Wrap Go/Py juju client status: - https://launchpad.net/python-jujuclient - --- [notes from cloudsprint 2013-05] Topics to cover Current Testing Current todos Experiences from IS Ideas Review charm policy to include: Test must pass tests Charm must have tests We want embedded tests! All tests live in the charm Functional Tests /test (in charm) Integration /test.d (in charm) How to make it low entry for charmers to add tests charm create tests (charm-tools make a stub _simple_ test) leverage libraries, and possibly a deployment (dare I say declarative) testing lang. Sidnei mocks all the juju calls (U1 testing) have a library that stubs this for you. pull this into charm-helper library leverage Go-watch leverage charm testing with charm upgrade Story Points: (added to Blueprint work item) Integration Testing includes framework that charm authors can write tests against
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Antonio Rosales: Whiteboard changed: Discussion: *Bug submission on charm failure. *Define a process around how charm maintainers respond to test failures and subsequent bugs. Can a user run a manual test and submit the test back to the bug report to update testing status to green. *Enable Autocharm tester to be more resilient against provider failures, and Jenkins usage. simulate provider failure, and be able to recover: $ juju ssh MACHINE sudo shutdown now * Define WIs to execute auto charm testing on Go. * Continuous Integration (also will help with gating on charm commits) * Juju Testing Blogging * Juju testing communication to Juju lists. * Work on integrating/fixing Charm runner (graph testing/ dependency/env set up testing). Add a Jenkins workflow to run a charm or a set of charms in the following LXC environments: -raring container on raring host -raring container on precise host -precise container on raring host -precise container on precise host Two modes of testing: -Unit (does the charm start, and report ready) -Workload (test the charms relations, and pushing data) Reference Links: *Charm Test Spec [html] https://juju.ubuntu.com/docs/charm-tests.html * Charm Test Spec [source] http://bazaar.launchpad.net/~juju/juju/docs/view/head:/source/charm-tests.rst * CharmTester Charm http://jujucharms.com/~mark-mims/oneiric/charmtester * Charm Runner: https://launchpad.net/charmrunner * Jenkins Charm Testing: https://jenkins.qa.ubuntu.com/view/Charms/ [USER STORIES] William is a juju user who wishes to know a charms current stability Saul is patching a charm and wants to in sure his changes are work with current tests Laura is a charm maintainer and wants to add tests to in sure her charm is stable Kara is a charm maintainer and needs to know when her charm is broken Lee is a charmer who, while reviewing charm submissions, needs to know if these changes break backwards compatibility with currently deployed services Gaius is a charm maintainer from an upstream project and needs an easy way to learn how to write tests for his charm [ASSUMPTIONS] - Charm tester/charm tester control will work with gojuju for at least graph testing [RISKS] - Relying soley on graph testing may result in inaccurate test results due to lack of embedded tests - Making tests too complicated may result in low adoption rate of embedded testing [IN SCOPE] [OUT OF SCOPE] [USER ACCEPTANCE] [RELEASE NOTE/BLOG] - [Notes] + [NOTES] UDS 1305 notes: http://pad.ubuntu.com/ep/pad/view/uds-1305-servercloud-s -juju-charm-testing/latest === UDS 1303 Notes === Pad: http://pad.ubuntu.com/uds-1303-servercloud-r-juju-charm-testing Question: Is there a way in meta-data to explicitly state provider support. -Example: Ceph: Does cloud provider have block support -More broadly stated does the cloud provider have the capabilities the charm needs Idea: -In charm testing status be able to show that a charm failure can be a result of the provider not providing the needed capabilities, ie the Ceph charm fails on a provider because it does not support object store. -Make interface usage more verbose in the charm description. -Need a rule/spec on how a interface should be implemented -Need to investigate possible enforment of interfaces -**Have the testing framework iterate through the operational deployment requirments. Interfaces doc link broken: -Example: http://jujucharms.com/interfaces/ceph-client Interface doc link broken: https://juju.ubuntu.com/Interfaces/ceph-client -- broken Meta-language testing (http://paste.ubuntu.com/5588570/): Lanugage suggestions: http://lettuce.it/ http://cukes.info/ Charm-Runner integration: - https://launchpad.net/juju-deployer Wrap Go/Py juju client status: - https://launchpad.net/python-jujuclient --- [notes from cloudsprint 2013-05] Topics to cover Current Testing Current todos Experiences from IS Ideas Review charm policy to include: Test must pass tests Charm must have tests We want embedded tests! All tests live in the charm Functional Tests /test (in charm) Integration /test.d (in charm) How to make it low entry for charmers to add tests charm create tests (charm-tools make a stub _simple_ test) leverage libraries, and possibly a deployment (dare I say declarative) testing lang. Sidnei mocks all the juju calls (U1 testing) have a library that stubs this for you. pull this into charm-helper library leverage Go-watch leverage charm testing with charm upgrade -- Juju Charm Testing https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-juju-charm-testing -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Marco Ceppi: Whiteboard changed: Discussion: *Bug submission on charm failure. *Define a process around how charm maintainers respond to test failures and subsequent bugs. Can a user run a manual test and submit the test back to the bug report to update testing status to green. *Enable Autocharm tester to be more resilient against provider failures, and Jenkins usage. simulate provider failure, and be able to recover: $ juju ssh MACHINE sudo shutdown now * Define WIs to execute auto charm testing on Go. * Continuous Integration (also will help with gating on charm commits) * Juju Testing Blogging * Juju testing communication to Juju lists. * Work on integrating/fixing Charm runner (graph testing/ dependency/env set up testing). Add a Jenkins workflow to run a charm or a set of charms in the following LXC environments: -raring container on raring host -raring container on precise host -precise container on raring host -precise container on precise host Two modes of testing: -Unit (does the charm start, and report ready) -Workload (test the charms relations, and pushing data) Reference Links: *Charm Test Spec [html] https://juju.ubuntu.com/docs/charm-tests.html * Charm Test Spec [source] http://bazaar.launchpad.net/~juju/juju/docs/view/head:/source/charm-tests.rst * CharmTester Charm http://jujucharms.com/~mark-mims/oneiric/charmtester * Charm Runner: https://launchpad.net/charmrunner * Jenkins Charm Testing: https://jenkins.qa.ubuntu.com/view/Charms/ [USER STORIES] William is a juju user who wishes to know a charms current stability - Saul is patching a charm and wants to in sure his changes are work with + Saul is patching a charm and wants to insure his changes work with the current tests - Laura is a charm maintainer and wants to add tests to in sure her charm + Laura is a charm maintainer and wants to add tests to insure her charm is stable Kara is a charm maintainer and needs to know when her charm is broken Lee is a charmer who, while reviewing charm submissions, needs to know if these changes break backwards compatibility with currently deployed services Gaius is a charm maintainer from an upstream project and needs an easy way to learn how to write tests for his charm [ASSUMPTIONS] - Charm tester/charm tester control will work with gojuju for at least graph testing [RISKS] - Relying soley on graph testing may result in inaccurate test results due to lack of embedded tests - Making tests too complicated may result in low adoption rate of embedded testing [IN SCOPE] [OUT OF SCOPE] [USER ACCEPTANCE] [RELEASE NOTE/BLOG] [NOTES] UDS 1305 notes: http://pad.ubuntu.com/ep/pad/view/uds-1305-servercloud-s -juju-charm-testing/latest === UDS 1303 Notes === Pad: http://pad.ubuntu.com/uds-1303-servercloud-r-juju-charm-testing Question: Is there a way in meta-data to explicitly state provider support. -Example: Ceph: Does cloud provider have block support -More broadly stated does the cloud provider have the capabilities the charm needs Idea: -In charm testing status be able to show that a charm failure can be a result of the provider not providing the needed capabilities, ie the Ceph charm fails on a provider because it does not support object store. -Make interface usage more verbose in the charm description. -Need a rule/spec on how a interface should be implemented -Need to investigate possible enforment of interfaces -**Have the testing framework iterate through the operational deployment requirments. Interfaces doc link broken: -Example: http://jujucharms.com/interfaces/ceph-client Interface doc link broken: https://juju.ubuntu.com/Interfaces/ceph-client -- broken Meta-language testing (http://paste.ubuntu.com/5588570/): Lanugage suggestions: http://lettuce.it/ http://cukes.info/ Charm-Runner integration: - https://launchpad.net/juju-deployer Wrap Go/Py juju client status: - https://launchpad.net/python-jujuclient --- [notes from cloudsprint 2013-05] Topics to cover Current Testing Current todos Experiences from IS Ideas Review charm policy to include: Test must pass tests Charm must have tests We want embedded tests! All tests live in the charm Functional Tests /test (in charm) Integration /test.d (in charm) How to make it low entry for charmers to add tests charm create tests (charm-tools make a stub _simple_ test) leverage libraries, and possibly a deployment (dare I say declarative) testing lang. Sidnei mocks all the juju calls (U1 testing) have a library that stubs this for you. pull this into charm-helper library leverage Go-watch leverage charm testing with charm upgrade -- Juju Charm Testing
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Antonio Rosales: Whiteboard changed: Discussion: *Bug submission on charm failure. *Define a process around how charm maintainers respond to test failures and subsequent bugs. Can a user run a manual test and submit the test back to the bug report to update testing status to green. *Enable Autocharm tester to be more resilient against provider failures, and Jenkins usage. simulate provider failure, and be able to recover: $ juju ssh MACHINE sudo shutdown now * Define WIs to execute auto charm testing on Go. * Continuous Integration (also will help with gating on charm commits) * Juju Testing Blogging * Juju testing communication to Juju lists. * Work on integrating/fixing Charm runner (graph testing/ dependency/env set up testing). Add a Jenkins workflow to run a charm or a set of charms in the following LXC environments: -raring container on raring host -raring container on precise host -precise container on raring host -precise container on precise host Two modes of testing: -Unit (does the charm start, and report ready) -Workload (test the charms relations, and pushing data) Reference Links: *Charm Test Spec [html] https://juju.ubuntu.com/docs/charm-tests.html * Charm Test Spec [source] http://bazaar.launchpad.net/~juju/juju/docs/view/head:/source/charm-tests.rst * CharmTester Charm http://jujucharms.com/~mark-mims/oneiric/charmtester * Charm Runner: https://launchpad.net/charmrunner * Jenkins Charm Testing: https://jenkins.qa.ubuntu.com/view/Charms/ [USER STORIES] William is a juju user who wishes to know a charms current stability Saul is patching a charm and wants to in sure his changes are work with current tests Laura is a charm maintainer and wants to add tests to in sure her charm is stable Kara is a charm maintainer and needs to know when her charm is broken Lee is a charmer who, while reviewing charm submissions, needs to know if these changes break backwards compatibility with currently deployed services Gaius is a charm maintainer from an upstream project and needs an easy way to learn how to write tests for his charm [ASSUMPTIONS] - Charm tester/charm tester control will work with gojuju for at least graph testing [RISKS] - Relying soley on graph testing may result in inaccurate test results due to lack of embedded tests - Making tests too complicated may result in low adoption rate of embedded testing [IN SCOPE] [OUT OF SCOPE] [USER ACCEPTANCE] [RELEASE NOTE/BLOG] [Notes] UDS 1305 notes: http://pad.ubuntu.com/ep/pad/view/uds-1305-servercloud-s -juju-charm-testing/latest === UDS 1303 Notes === Pad: http://pad.ubuntu.com/uds-1303-servercloud-r-juju-charm-testing Question: Is there a way in meta-data to explicitly state provider support. -Example: Ceph: Does cloud provider have block support -More broadly stated does the cloud provider have the capabilities the charm needs Idea: -In charm testing status be able to show that a charm failure can be a result of the provider not providing the needed capabilities, ie the Ceph charm fails on a provider because it does not support object store. -Make interface usage more verbose in the charm description. -Need a rule/spec on how a interface should be implemented -Need to investigate possible enforment of interfaces -**Have the testing framework iterate through the operational deployment requirments. Interfaces doc link broken: -Example: http://jujucharms.com/interfaces/ceph-client Interface doc link broken: https://juju.ubuntu.com/Interfaces/ceph-client -- broken Meta-language testing (http://paste.ubuntu.com/5588570/): Lanugage suggestions: http://lettuce.it/ http://cukes.info/ Charm-Runner integration: - https://launchpad.net/juju-deployer Wrap Go/Py juju client status: - https://launchpad.net/python-jujuclient --- [notes from cloudsprint 2013-05] Topics to cover Current Testing Current todos Experiences from IS Ideas Review charm policy to include: Test must pass tests Charm must have tests We want embedded tests! All tests live in the charm Functional Tests /test (in charm) Integration /test.d (in charm) How to make it low entry for charmers to add tests charm create tests (charm-tools make a stub _simple_ test) leverage libraries, and possibly a deployment (dare I say declarative) testing lang. Sidnei mocks all the juju calls (U1 testing) have a library that stubs this for you. pull this into charm-helper library leverage Go-watch leverage charm testing with charm upgrade - - Story Points: (added to Blueprint work item) - Integration Testing includes framework that charm authors can write tests against (embedded in the charm). - Jenkins testing on new merge proposal, on
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Mark Mims: Whiteboard changed: Discussion: *Bug submission on charm failure. *Define a process around how charm maintainers respond to test failures and subsequent bugs. Can a user run a manual test and submit the test back to the bug report to update testing status to green. *Enable Autocharm tester to be more resilient against provider failures, and Jenkins usage. simulate provider failure, and be able to recover: $ juju ssh MACHINE sudo shutdown now * Define WIs to execute auto charm testing on Go. * Continuous Integration (also will help with gating on charm commits) * Juju Testing Blogging * Juju testing communication to Juju lists. * Work on integrating/fixing Charm runner (graph testing/ dependency/env set up testing). Add a Jenkins workflow to run a charm or a set of charms in the following LXC environments: -raring container on raring host -raring container on precise host -precise container on raring host -precise container on precise host Two modes of testing: -Unit (does the charm start, and report ready) -Workload (test the charms relations, and pushing data) Reference Links: *Charm Test Spec [html] https://juju.ubuntu.com/docs/charm-tests.html * Charm Test Spec [source] http://bazaar.launchpad.net/~juju/juju/docs/view/head:/source/charm-tests.rst * CharmTester Charm http://jujucharms.com/~mark-mims/oneiric/charmtester * Charm Runner: https://launchpad.net/charmrunner * Jenkins Charm Testing: https://jenkins.qa.ubuntu.com/view/Charms/ [USER STORIES] William is a juju user who wishes to know a charms current stability Saul is patching a charm and wants to in sure his changes are work with current tests Laura is a charm maintainer and wants to add tests to in sure her charm is stable Kara is a charm maintainer and needs to know when her charm is broken Lee is a charmer who, while reviewing charm submissions, needs to know if these changes break backwards compatibility with currently deployed services Gaius is a charm maintainer from an upstream project and needs an easy way to learn how to write tests for his charm [ASSUMPTIONS] - Charm tester/charm tester control will work with gojuju for at least graph testing [RISKS] - Relying soley on graph testing may result in inaccurate test results due to lack of embedded tests - Making tests too complicated may result in low adoption rate of embedded testing [IN SCOPE] [OUT OF SCOPE] [USER ACCEPTANCE] [RELEASE NOTE/BLOG] (Needs spec and WI definition) -[a.rosales; 12-DEC-2012] === UDS 1303 Notes === Pad: http://pad.ubuntu.com/uds-1303-servercloud-r-juju-charm-testing Question: Is there a way in meta-data to explicitly state provider support. -Example: Ceph: Does cloud provider have block support -More broadly stated does the cloud provider have the capabilities the charm needs Idea: -In charm testing status be able to show that a charm failure can be a result of the provider not providing the needed capabilities, ie the Ceph charm fails on a provider because it does not support object store. -Make interface usage more verbose in the charm description. -Need a rule/spec on how a interface should be implemented -Need to investigate possible enforment of interfaces -**Have the testing framework iterate through the operational deployment requirments. Interfaces doc link broken: -Example: http://jujucharms.com/interfaces/ceph-client Interface doc link broken: https://juju.ubuntu.com/Interfaces/ceph-client -- broken Meta-language testing (http://paste.ubuntu.com/5588570/): Lanugage suggestions: http://lettuce.it/ http://cukes.info/ Charm-Runner integration: - https://launchpad.net/juju-deployer Wrap Go/Py juju client status: - https://launchpad.net/python-jujuclient + + + --- + + [notes from cloudsprint 2013-05] + Topics to cover + Current Testing + Current todos + Experiences from IS + Ideas + + Review charm policy to include: + Test must pass tests + Charm must have tests + + We want embedded tests! + All tests live in the charm + Functional Tests + /test (in charm) + Integration + /test.d (in charm) + How to make it low entry for charmers to add tests + charm create tests (charm-tools make a stub _simple_ test) + leverage libraries, and possibly a deployment (dare I say declarative) testing lang. + Sidnei mocks all the juju calls (U1 testing) + have a library that stubs this for you. + pull this into charm-helper library + leverage Go-watch + leverage charm testing with charm upgrade + + Story Points: (added to Blueprint work item) + Integration Testing includes framework that charm authors can write tests against (embedded in the charm). + Jenkins testing on new merge proposal, on success it is a candidate for review + Develop Juju test
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Antonio Rosales: Work items set to: Work items: Integration Testing includes framework that charm authors can write tests against (embedded in the charm): TODO Jenkins testing on new merge proposal, on success it is a candidate for review: TODO Develop Juju test plugin (ie juju test): TODO Modify charm tools to capture a stub integration test: TODO Include charm-helper library for all charmers: TODO -- Juju Charm Testing https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-juju-charm-testing -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Marco Ceppi: Whiteboard changed: Discussion: *Bug submission on charm failure. *Define a process around how charm maintainers respond to test failures and subsequent bugs. Can a user run a manual test and submit the test back to the bug report to update testing status to green. *Enable Autocharm tester to be more resilient against provider failures, and Jenkins usage. simulate provider failure, and be able to recover: $ juju ssh MACHINE sudo shutdown now * Define WIs to execute auto charm testing on Go. * Continuous Integration (also will help with gating on charm commits) * Juju Testing Blogging * Juju testing communication to Juju lists. * Work on integrating/fixing Charm runner (graph testing/ dependency/env set up testing). Add a Jenkins workflow to run a charm or a set of charms in the following LXC environments: -raring container on raring host -raring container on precise host -precise container on raring host -precise container on precise host Two modes of testing: -Unit (does the charm start, and report ready) -Workload (test the charms relations, and pushing data) Reference Links: *Charm Test Spec [html] https://juju.ubuntu.com/docs/charm-tests.html * Charm Test Spec [source] http://bazaar.launchpad.net/~juju/juju/docs/view/head:/source/charm-tests.rst * CharmTester Charm http://jujucharms.com/~mark-mims/oneiric/charmtester * Charm Runner: https://launchpad.net/charmrunner * Jenkins Charm Testing: https://jenkins.qa.ubuntu.com/view/Charms/ [USER STORIES] + + William is a juju user who wishes to know a charms current stability + + Saul is patching a charm and wants to in sure his changes are work with + current tests + + Laura is a charm maintainer and wants to add tests to in sure her charm + is stable + + Kara is a charm maintainer and needs to know when her charm is broken + + Lee is a charmer who, while reviewing charm submissions, needs to know + if these changes break backwards compatibility with currently deployed + services + + Gaius is a charm maintainer from an upstream project and needs an easy + way to learn how to write tests for his charm + [ASSUMPTIONS] + + - Charm tester/charm tester control will work with gojuju for at least + graph testing + [RISKS] + + - Relying soley on graph testing may result in inaccurate test results due to lack of embedded tests + - Making tests too complicated may result in low adoption rate of embedded testing + [IN SCOPE] + [OUT OF SCOPE] + [USER ACCEPTANCE] + [RELEASE NOTE/BLOG] (Needs spec and WI definition) -[a.rosales; 12-DEC-2012] === UDS 1303 Notes === Pad: http://pad.ubuntu.com/uds-1303-servercloud-r-juju-charm-testing Question: Is there a way in meta-data to explicitly state provider support. -Example: Ceph: Does cloud provider have block support -More broadly stated does the cloud provider have the capabilities the charm needs Idea: -In charm testing status be able to show that a charm failure can be a result of the provider not providing the needed capabilities, ie the Ceph charm fails on a provider because it does not support object store. -Make interface usage more verbose in the charm description. -Need a rule/spec on how a interface should be implemented -Need to investigate possible enforment of interfaces -**Have the testing framework iterate through the operational deployment requirments. Interfaces doc link broken: -Example: http://jujucharms.com/interfaces/ceph-client Interface doc link broken: https://juju.ubuntu.com/Interfaces/ceph-client -- broken Meta-language testing (http://paste.ubuntu.com/5588570/): Lanugage suggestions: http://lettuce.it/ http://cukes.info/ Charm-Runner integration: - https://launchpad.net/juju-deployer Wrap Go/Py juju client status: - https://launchpad.net/python-jujuclient -- Juju Charm Testing https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-juju-charm-testing -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Antonio Rosales: Whiteboard changed: Discussion: *Bug submission on charm failure. *Define a process around how charm maintainers respond to test failures and subsequent bugs. Can a user run a manual test and submit the test back to the bug report to update testing status to green. *Enable Autocharm tester to be more resilient against provider failures, and Jenkins usage. simulate provider failure, and be able to recover: $ juju ssh MACHINE sudo shutdown now * Define WIs to execute auto charm testing on Go. * Continuous Integration (also will help with gating on charm commits) * Juju Testing Blogging * Juju testing communication to Juju lists. * Work on integrating/fixing Charm runner (graph testing/ dependency/env set up testing). Add a Jenkins workflow to run a charm or a set of charms in the following LXC environments: -raring container on raring host -raring container on precise host -precise container on raring host -precise container on precise host + + Two modes of testing: + -Unit (does the charm start, and report ready) + -Workload (test the charms relations, and pushing data) Reference Links: *Charm Test Spec [html] https://juju.ubuntu.com/docs/charm-tests.html * Charm Test Spec [source] http://bazaar.launchpad.net/~juju/juju/docs/view/head:/source/charm-tests.rst * CharmTester Charm http://jujucharms.com/~mark-mims/oneiric/charmtester * Charm Runner: https://launchpad.net/charmrunner * Jenkins Charm Testing: https://jenkins.qa.ubuntu.com/view/Charms/ [USER STORIES] [ASSUMPTIONS] [RISKS] [IN SCOPE] [OUT OF SCOPE] [USER ACCEPTANCE] [RELEASE NOTE/BLOG] (Needs spec and WI definition) -[a.rosales; 12-DEC-2012] === UDS 1303 Notes === Pad: http://pad.ubuntu.com/uds-1303-servercloud-r-juju-charm-testing Question: Is there a way in meta-data to explicitly state provider support. -Example: Ceph: Does cloud provider have block support -More broadly stated does the cloud provider have the capabilities the charm needs Idea: -In charm testing status be able to show that a charm failure can be a result of the provider not providing the needed capabilities, ie the Ceph charm fails on a provider because it does not support object store. -Make interface usage more verbose in the charm description. -Need a rule/spec on how a interface should be implemented -Need to investigate possible enforment of interfaces -**Have the testing framework iterate through the operational deployment requirments. Interfaces doc link broken: -Example: http://jujucharms.com/interfaces/ceph-client Interface doc link broken: https://juju.ubuntu.com/Interfaces/ceph-client -- broken Meta-language testing (http://paste.ubuntu.com/5588570/): Lanugage suggestions: http://lettuce.it/ http://cukes.info/ Charm-Runner integration: - - https://launchpad.net/juju-deployer + - https://launchpad.net/juju-deployer Wrap Go/Py juju client status: - - https://launchpad.net/python-jujuclient + - https://launchpad.net/python-jujuclient -- Juju Charm Testing https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-juju-charm-testing -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Antonio Rosales: Whiteboard changed: Discussion: *Bug submission on charm failure. *Define a process around how charm maintainers respond to test failures and subsequent bugs. Can a user run a manual test and submit the test back to the bug report to update testing status to green. *Enable Autocharm tester to be more resilient against provider failures, and Jenkins usage. simulate provider failure, and be able to recover: $ juju ssh MACHINE sudo shutdown now * Define WIs to execute auto charm testing on Go. * Continuous Integration (also will help with gating on charm commits) * Juju Testing Blogging * Juju testing communication to Juju lists. * Work on integrating/fixing Charm runner (graph testing/ dependency/env set up testing). Add a Jenkins workflow to run a charm or a set of charms in the following LXC environments: -raring container on raring host -raring container on precise host -precise container on raring host -precise container on precise host Reference Links: *Charm Test Spec [html] https://juju.ubuntu.com/docs/charm-tests.html * Charm Test Spec [source] http://bazaar.launchpad.net/~juju/juju/docs/view/head:/source/charm-tests.rst * CharmTester Charm http://jujucharms.com/~mark-mims/oneiric/charmtester * Charm Runner: https://launchpad.net/charmrunner * Jenkins Charm Testing: https://jenkins.qa.ubuntu.com/view/Charms/ [USER STORIES] [ASSUMPTIONS] [RISKS] [IN SCOPE] [OUT OF SCOPE] [USER ACCEPTANCE] [RELEASE NOTE/BLOG] (Needs spec and WI definition) -[a.rosales; 12-DEC-2012] === UDS 1303 Notes === Pad: http://pad.ubuntu.com/uds-1303-servercloud-r-juju-charm-testing Question: Is there a way in meta-data to explicitly state provider support. -Example: Ceph: Does cloud provider have block support -More broadly stated does the cloud provider have the capabilities the charm needs Idea: -In charm testing status be able to show that a charm failure can be a result of the provider not providing the needed capabilities, ie the Ceph charm fails on a provider because it does not support object store. -Make interface usage more verbose in the charm description. -Need a rule/spec on how a interface should be implemented -Need to investigate possible enforment of interfaces -**Have the testing framework iterate through the operational deployment requirments. Interfaces doc link broken: -Example: http://jujucharms.com/interfaces/ceph-client Interface doc link broken: https://juju.ubuntu.com/Interfaces/ceph-client -- broken Meta-language testing (http://paste.ubuntu.com/5588570/): Lanugage suggestions: http://lettuce.it/ http://cukes.info/ + + Charm-Runner integration: + - https://launchpad.net/juju-deployer + + Wrap Go/Py juju client status: + - https://launchpad.net/python-jujuclient -- Juju Charm Testing https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-juju-charm-testing -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing
Blueprint changed by Antonio Rosales: Name: servercloud-r-juju-charm-testing = servercloud-s-juju-charm-testing -- Juju Charm Testing https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-juju-charm-testing -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
