Yes, I think it's quite complementary. Looking forward to seeing the code land on the acceptance-test repo.
2014-05-13 4:25 GMT-07:00 Stephen Connolly <[email protected]> : > Oh I forgot my point... I suspect that some of the acceptance tests would > be easier to write if you use the scalability framework's DSL to set up a > lot of the acceptance test... at least they would be less fragile as you > have a known scenario with inter-job relationships. I have found that > setting up complex job scenarios via Selenium can be somewhat fragile and > you end up with brittle tests that fail at random. > > I do not think all acceptance tests should use the scalability framework's > Plan model to set up the test environment (other than the bare Plan with > plugins installed) as there is value in testing the path... but the more > complex ones where the setup path is already covered... they can get value > > > On 13 May 2014 12:21, Stephen Connolly <[email protected]>wrote: > >> Well I have a scalability test framework that makes it very easy to test >> jenkins at scale. I also think that the DSL for scaling jenkins is probably >> very handy for acceptace tests. For example, here is my *current* API for >> load testing the SSH Slaves connector: >> >> Plan p = new Plan(); >> Ec2MachineFactory factory = new Ec2MachineFactory(new >> Ec2Config()); >> try (Execution e = new Execution(p) >> .with(MasterNode.factory()) >> .with(SlaveNode.factory()) >> .machines(factory) >> .machines(new MultitenantMachinesFactory(factory, 15), >> PlannedSlave.class)) { >> PlannedMaster master = p.createMaster() >> .withPlugins("metrics", "random-job-builder") >> .withView(new >> PlannedCloudBeesDashboardView().withName("Dashboard")) >> .withPluginsDisabled("translation") >> .withExecutors(0); >> NioSshSlaveLauncher launcher = new NioSshSlaveLauncher(); >> for (int i = 0; i < 60; i++) { >> >> master.withSlave(p.createSlave().withLauncher(launcher).withExecutors(2)); >> } >> for (int f = 0; f < 10; f++) { >> PlannedFolder folder = p.createFolder(master); >> for (int g = 0; g < 3; g++) { >> PlannedFolder subfolder = p.createFolder(folder); >> for (int h = 0; h < 10; h++) { >> p.createMockJob(subfolder); >> } >> } >> } >> e.start(); >> MasterNode executionMaster = e.resolve(master); >> // give the user a browser window to observe the load >> >> Desktop.getDesktop().browse(URI.create(executionMaster.getJenkinsUrl()+"view/Dashboard/")); >> // wait for the user to request tear-down >> } >> >> I have found this *very* powerful for testing load. The above scenario >> let me determine, for example, that on an m3.large you can have about 60 >> concurrent mock load builds using the JNLP launcher, or about 10 concurrent >> mock load builds using the Trilead SSH launcher... above those levels by >> more than 5 concurrent builds and Jenkins blows up. With the CloudBees >> NioSSH launcher you can have 15 concurrent builds without slowing of build >> times... and you can have 120 concurrent builds without Jenkins blowing up. >> >> What I need to do is detangle some of the CloudBees specific bits: e.g. >> in the above example there are two classes that will not be made public: >> PlannedCloudBeesDashboardView and NioSshSlaveLauncher because those classes >> will decorate the provisioned Jenkins master with a CloudBees Jenkins >> Enterprise license ;-) >> >> I also want to get a hashing function for a Plan so that the JUT server >> can be made compatible in the face of requests for specific plans. >> >> >> On 13 May 2014 09:59, Ulli Hafner <[email protected]> wrote: >> >>> One of our students is currently trying to integrate Spock and Geb: this >>> also requires API changes (at least we need to introduce additional >>> interfaces for all those concrete selenium base classes). This part still >>> is work in progress. >>> >>> I also thought the purpose of this project is to collect all acceptance >>> tests for Jenkins. So if we need to change the framework we just go ahead >>> and commit the required changes. Up to now it was not clear that we need to >>> publish a „stable“ version at all. >>> Are there any other plans for additional test suites that are based on >>> the acceptance test harness project (besides your Cloudbees test suite)? >>> >>> Am 13.05.2014 um 02:26 schrieb Kohsuke Kawaguchi < >>> [email protected]>: >>> >>> > On 05/12/2014 04:09 PM, Stephen Connolly wrote: >>> >> You know I have *very* strong feelings on a lot of the lower >>> layers... but you >>> >> also know that I haven't had the time to get the replacement layer >>> published >>> >> yet... It's coming... there is the possibility that it may break some >>> stuff... >>> >> but I am reasonably confident that it shouldn't >>> > >>> > Good point, so there's at least that group of changes that we should >>> expect before we consider it solidified. >>> > >>> >> On 12 May 2014 23:28, Kohsuke Kawaguchi <[email protected] <mailto: >>> [email protected]>> >>> >> wrote: >>> >> >>> >> >>> >> So I jumped the gun a little bit and cut a release of >>> >> acceptance-test-harness, as it was blocking me on developing >>> other sets of >>> >> tests internally. >>> >> >>> >> But this triggered a comment from Oliver [1] in terms of what we >>> want to see >>> >> in the code before we consider the API stable enough. >>> >> >>> >> Specifically, >>> >> >>> >> I hoped there will be a discussion before making the API >>> public. I >>> >> wanted to get rid off all the public Controls exposed from >>> page objects >>> >> because we will not be able to preserve the API as Jenkins UI >>> evolves. >>> >> Changes like af98505 will surely follow. >>> >> >>> >> >>> >> So first of all, I wanted to set the expectation right that 1.0 >>> and 1.1 I >>> >> released shouldn't be considered stable. >>> >> >>> >> >>> >> With that said, personally I didn't anticipate that we hold >>> ourselves to the >>> >> level of backward compatibility that we do for Jenkins, and >>> public final >>> >> Control fields are nice, namely they are very concise to write. >>> >> >>> >> But I'm willing to follow what we want as a whole, so if you have >>> any strong >>> >> feelings and thoughts on this, this is the opportunity to speak >>> up. >>> >> >>> >> >>> >> [1] >>> >> >>> https://github.com/jenkinsci/__acceptance-test-harness/__commit/__8a4bacb386ee0b5a34c5dd499bd74f__50b35c726d#commitcomment-__6297001 >>> >> < >>> https://github.com/jenkinsci/acceptance-test-harness/commit/8a4bacb386ee0b5a34c5dd499bd74f50b35c726d#commitcomment-6297001 >>> > >>> >> -- >>> >> Kohsuke Kawaguchi http://kohsuke.org/ >>> >> >>> >> -- >>> >> You received this message because you are subscribed to the >>> Google Groups >>> >> "Jenkins Developers" group. >>> >> To unsubscribe from this group and stop receiving emails from it, >>> send an >>> >> email to jenkinsci-dev+unsubscribe@__googlegroups.com >>> >> <mailto:jenkinsci-dev%[email protected]>. >>> >> For more options, visit https://groups.google.com/d/__optout >>> >> <https://groups.google.com/d/optout>. >>> >> >>> >> >>> >> -- >>> >> You received this message because you are subscribed to the Google >>> Groups >>> >> "Jenkins Developers" group. >>> >> To unsubscribe from this group and stop receiving emails from it, >>> send an email >>> >> to [email protected] >>> >> <mailto:[email protected]>. >>> >> For more options, visit https://groups.google.com/d/optout. >>> >> >>> > >>> > >>> > -- >>> > Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ >>> > Try Jenkins Enterprise, our professional version of Jenkins >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Jenkins Developers" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> > For more options, visit https://groups.google.com/d/optout. >>> >>> >> > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- Kohsuke Kawaguchi -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
