Re: Solidifying acceptance-test-harness API
On Thu, Jun 5, 2014 at 5:16 PM, Ulli Hafner ullrich.haf...@gmail.com wrote: it feels quite unusual to make the Assert class the topmost class of our inheritance hierarchy Agreed, extending a class (or implementing an interface) to get static methods (resp. constants) is poor style when a static wildcard import would suffice. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Solidifying acceptance-test-harness API
Thanks. I'd encourage Chris to write up some stuff in docs/ to introduce this feature. And can we get extends Assert back in CapybaraPortingLayerImpl? Maybe it's just me, but personally I found it convenient that it was extending Assert, which makes my IDE auto-completion a lot smoother. On 06/04/2014 03:15 PM, Ulli Hafner wrote: The Spock and Geb changes are now merged. In order to get those frameworks running with the current classes, Christian needed to introduce several interfaces. The corresponding classes have been renamed to *Impl. Here is an example of the new capabilities: https://github.com/jenkinsci/acceptance-test-harness/blob/master/src/test/groovy/groovy/plugins/JacocoPluginTest.groovy Design of page objects is also quite simple: https://github.com/jenkinsci/acceptance-test-harness/tree/master/src/main/groovy/org/jenkinsci/test/acceptance/plugins/jacoco Am 13.05.2014 um 10:59 schrieb Ulli Hafner ullrich.haf...@gmail.com: One of our students is currently trying to integrate Spock and Geb: this also requires API changes (at least we need to introduce additional interfaces for all those concrete selenium base classes). This part still is work in progress. I also thought the purpose of this project is to collect all acceptance tests for Jenkins. So if we need to change the framework we just go ahead and commit the required changes. Up to now it was not clear that we need to publish a stable version at all. Are there any other plans for additional test suites that are based on the acceptance test harness project (besides your Cloudbees test suite)? Am 13.05.2014 um 02:26 schrieb Kohsuke Kawaguchi kkawagu...@cloudbees.com: On 05/12/2014 04:09 PM, Stephen Connolly wrote: You know I have *very* strong feelings on a lot of the lower layers... but you also know that I haven't had the time to get the replacement layer published yet... It's coming... there is the possibility that it may break some stuff... but I am reasonably confident that it shouldn't Good point, so there's at least that group of changes that we should expect before we consider it solidified. On 12 May 2014 23:28, Kohsuke Kawaguchi k...@kohsuke.org mailto:k...@kohsuke.org wrote: So I jumped the gun a little bit and cut a release of acceptance-test-harness, as it was blocking me on developing other sets of tests internally. But this triggered a comment from Oliver [1] in terms of what we want to see in the code before we consider the API stable enough. Specifically, I hoped there will be a discussion before making the API public. I wanted to get rid off all the public Controls exposed from page objects because we will not be able to preserve the API as Jenkins UI evolves. Changes like af98505 will surely follow. So first of all, I wanted to set the expectation right that 1.0 and 1.1 I released shouldn't be considered stable. With that said, personally I didn't anticipate that we hold ourselves to the level of backward compatibility that we do for Jenkins, and public final Control fields are nice, namely they are very concise to write. But I'm willing to follow what we want as a whole, so if you have any strong feelings and thoughts on this, this is the opportunity to speak up. [1] https://github.com/jenkinsci/__acceptance-test-harness/__commit/__8a4bacb386ee0b5a34c5dd499bd74f__50b35c726d#commitcomment-__6297001 https://github.com/jenkinsci/acceptance-test-harness/commit/8a4bacb386ee0b5a34c5dd499bd74f50b35c726d#commitcomment-6297001 -- Kohsuke Kawaguchi http://kohsuke.org/ -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@__googlegroups.com mailto:jenkinsci-dev%2bunsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/__optout https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com mailto:jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Jenkins Enterprise, our professional version of Jenkins -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Jenkins Enterprise, our professional version of Jenkins -- You received this message because you are subscribed to the
Re: Solidifying acceptance-test-harness API
Am 05.06.2014 um 19:00 schrieb Kohsuke Kawaguchi kkawagu...@cloudbees.com: Thanks. I'd encourage Chris to write up some stuff in docs/ to introduce this feature. Yes, this will be part of his next steps… And can we get extends Assert back in CapybaraPortingLayerImpl? Maybe it's just me, but personally I found it convenient that it was extending Assert, which makes my IDE auto-completion a lot smoother. I see. This is one of the drawbacks of IntelliJ:-) I also noted that after my switch from Eclipse that you need to press a different shortcut (CTRL-ALT-Space) to get code completion on static assert* methods (in Eclipse these completions are unified)… If this is too cumbersome we can revert that part of the change (even though it feels quite unusual to make the Assert class the topmost class of our inheritance hierarchy). On 06/04/2014 03:15 PM, Ulli Hafner wrote: The Spock and Geb changes are now merged. In order to get those frameworks running with the current classes, Christian needed to introduce several interfaces. The corresponding classes have been renamed to *Impl. Here is an example of the new capabilities: https://github.com/jenkinsci/acceptance-test-harness/blob/master/src/test/groovy/groovy/plugins/JacocoPluginTest.groovy Design of page objects is also quite simple: https://github.com/jenkinsci/acceptance-test-harness/tree/master/src/main/groovy/org/jenkinsci/test/acceptance/plugins/jacoco Am 13.05.2014 um 10:59 schrieb Ulli Hafner ullrich.haf...@gmail.com: One of our students is currently trying to integrate Spock and Geb: this also requires API changes (at least we need to introduce additional interfaces for all those concrete selenium base classes). This part still is work in progress. I also thought the purpose of this project is to collect all acceptance tests for Jenkins. So if we need to change the framework we just go ahead and commit the required changes. Up to now it was not clear that we need to publish a stable version at all. Are there any other plans for additional test suites that are based on the acceptance test harness project (besides your Cloudbees test suite)? Am 13.05.2014 um 02:26 schrieb Kohsuke Kawaguchi kkawagu...@cloudbees.com: On 05/12/2014 04:09 PM, Stephen Connolly wrote: You know I have *very* strong feelings on a lot of the lower layers... but you also know that I haven't had the time to get the replacement layer published yet... It's coming... there is the possibility that it may break some stuff... but I am reasonably confident that it shouldn't Good point, so there's at least that group of changes that we should expect before we consider it solidified. On 12 May 2014 23:28, Kohsuke Kawaguchi k...@kohsuke.org mailto:k...@kohsuke.org wrote: So I jumped the gun a little bit and cut a release of acceptance-test-harness, as it was blocking me on developing other sets of tests internally. But this triggered a comment from Oliver [1] in terms of what we want to see in the code before we consider the API stable enough. Specifically, I hoped there will be a discussion before making the API public. I wanted to get rid off all the public Controls exposed from page objects because we will not be able to preserve the API as Jenkins UI evolves. Changes like af98505 will surely follow. So first of all, I wanted to set the expectation right that 1.0 and 1.1 I released shouldn't be considered stable. With that said, personally I didn't anticipate that we hold ourselves to the level of backward compatibility that we do for Jenkins, and public final Control fields are nice, namely they are very concise to write. But I'm willing to follow what we want as a whole, so if you have any strong feelings and thoughts on this, this is the opportunity to speak up. [1] https://github.com/jenkinsci/__acceptance-test-harness/__commit/__8a4bacb386ee0b5a34c5dd499bd74f__50b35c726d#commitcomment-__6297001 https://github.com/jenkinsci/acceptance-test-harness/commit/8a4bacb386ee0b5a34c5dd499bd74f50b35c726d#commitcomment-6297001 -- Kohsuke Kawaguchi http://kohsuke.org/ -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@__googlegroups.com mailto:jenkinsci-dev%2bunsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/__optout https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com mailto:jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit
Re: Solidifying acceptance-test-harness API
The Spock and Geb changes are now merged. In order to get those frameworks running with the current classes, Christian needed to introduce several interfaces. The corresponding classes have been renamed to *Impl. Here is an example of the new capabilities: https://github.com/jenkinsci/acceptance-test-harness/blob/master/src/test/groovy/groovy/plugins/JacocoPluginTest.groovy Design of page objects is also quite simple: https://github.com/jenkinsci/acceptance-test-harness/tree/master/src/main/groovy/org/jenkinsci/test/acceptance/plugins/jacoco Am 13.05.2014 um 10:59 schrieb Ulli Hafner ullrich.haf...@gmail.com: One of our students is currently trying to integrate Spock and Geb: this also requires API changes (at least we need to introduce additional interfaces for all those concrete selenium base classes). This part still is work in progress. I also thought the purpose of this project is to collect all acceptance tests for Jenkins. So if we need to change the framework we just go ahead and commit the required changes. Up to now it was not clear that we need to publish a „stable“ version at all. Are there any other plans for additional test suites that are based on the acceptance test harness project (besides your Cloudbees test suite)? Am 13.05.2014 um 02:26 schrieb Kohsuke Kawaguchi kkawagu...@cloudbees.com: On 05/12/2014 04:09 PM, Stephen Connolly wrote: You know I have *very* strong feelings on a lot of the lower layers... but you also know that I haven't had the time to get the replacement layer published yet... It's coming... there is the possibility that it may break some stuff... but I am reasonably confident that it shouldn't Good point, so there's at least that group of changes that we should expect before we consider it solidified. On 12 May 2014 23:28, Kohsuke Kawaguchi k...@kohsuke.org mailto:k...@kohsuke.org wrote: So I jumped the gun a little bit and cut a release of acceptance-test-harness, as it was blocking me on developing other sets of tests internally. But this triggered a comment from Oliver [1] in terms of what we want to see in the code before we consider the API stable enough. Specifically, I hoped there will be a discussion before making the API public. I wanted to get rid off all the public Controls exposed from page objects because we will not be able to preserve the API as Jenkins UI evolves. Changes like af98505 will surely follow. So first of all, I wanted to set the expectation right that 1.0 and 1.1 I released shouldn't be considered stable. With that said, personally I didn't anticipate that we hold ourselves to the level of backward compatibility that we do for Jenkins, and public final Control fields are nice, namely they are very concise to write. But I'm willing to follow what we want as a whole, so if you have any strong feelings and thoughts on this, this is the opportunity to speak up. [1] https://github.com/jenkinsci/__acceptance-test-harness/__commit/__8a4bacb386ee0b5a34c5dd499bd74f__50b35c726d#commitcomment-__6297001 https://github.com/jenkinsci/acceptance-test-harness/commit/8a4bacb386ee0b5a34c5dd499bd74f50b35c726d#commitcomment-6297001 -- Kohsuke Kawaguchi http://kohsuke.org/ -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@__googlegroups.com mailto:jenkinsci-dev%2bunsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/__optout https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com mailto:jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Jenkins Enterprise, our professional version of Jenkins -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Solidifying acceptance-test-harness API
One of our students is currently trying to integrate Spock and Geb: this also requires API changes (at least we need to introduce additional interfaces for all those concrete selenium base classes). This part still is work in progress. I also thought the purpose of this project is to collect all acceptance tests for Jenkins. So if we need to change the framework we just go ahead and commit the required changes. Up to now it was not clear that we need to publish a „stable“ version at all. Are there any other plans for additional test suites that are based on the acceptance test harness project (besides your Cloudbees test suite)? Am 13.05.2014 um 02:26 schrieb Kohsuke Kawaguchi kkawagu...@cloudbees.com: On 05/12/2014 04:09 PM, Stephen Connolly wrote: You know I have *very* strong feelings on a lot of the lower layers... but you also know that I haven't had the time to get the replacement layer published yet... It's coming... there is the possibility that it may break some stuff... but I am reasonably confident that it shouldn't Good point, so there's at least that group of changes that we should expect before we consider it solidified. On 12 May 2014 23:28, Kohsuke Kawaguchi k...@kohsuke.org mailto:k...@kohsuke.org wrote: So I jumped the gun a little bit and cut a release of acceptance-test-harness, as it was blocking me on developing other sets of tests internally. But this triggered a comment from Oliver [1] in terms of what we want to see in the code before we consider the API stable enough. Specifically, I hoped there will be a discussion before making the API public. I wanted to get rid off all the public Controls exposed from page objects because we will not be able to preserve the API as Jenkins UI evolves. Changes like af98505 will surely follow. So first of all, I wanted to set the expectation right that 1.0 and 1.1 I released shouldn't be considered stable. With that said, personally I didn't anticipate that we hold ourselves to the level of backward compatibility that we do for Jenkins, and public final Control fields are nice, namely they are very concise to write. But I'm willing to follow what we want as a whole, so if you have any strong feelings and thoughts on this, this is the opportunity to speak up. [1] https://github.com/jenkinsci/__acceptance-test-harness/__commit/__8a4bacb386ee0b5a34c5dd499bd74f__50b35c726d#commitcomment-__6297001 https://github.com/jenkinsci/acceptance-test-harness/commit/8a4bacb386ee0b5a34c5dd499bd74f50b35c726d#commitcomment-6297001 -- Kohsuke Kawaguchi http://kohsuke.org/ -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@__googlegroups.com mailto:jenkinsci-dev%2bunsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/__optout https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com mailto:jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Jenkins Enterprise, our professional version of Jenkins -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. signature.asc Description: Message signed with OpenPGP using GPGMail
RE: Solidifying acceptance-test-harness API
As long as the version number reflects API compatibility I don't see any problems with not staying backwards compatible and no direct need for a stable version. I will probably have some internal test suite that tests the internal tool chain that can't be published, as long as I can easily see what bumping up to a newer version would bring in terms of adaptation effort I see no harm in keeping it fast and loose ;) Robert Sandell Sony Mobile Communications Tel: +46 10 80 12721 sonymobile.com -Original Message- From: jenkinsci-dev@googlegroups.com [mailto:jenkinsci-dev@googlegroups.com] On Behalf Of Ulli Hafner Sent: den 13 maj 2014 11:00 To: jenkinsci-dev@googlegroups.com Subject: Re: Solidifying acceptance-test-harness API One of our students is currently trying to integrate Spock and Geb: this also requires API changes (at least we need to introduce additional interfaces for all those concrete selenium base classes). This part still is work in progress. I also thought the purpose of this project is to collect all acceptance tests for Jenkins. So if we need to change the framework we just go ahead and commit the required changes. Up to now it was not clear that we need to publish a stable version at all. Are there any other plans for additional test suites that are based on the acceptance test harness project (besides your Cloudbees test suite)? Am 13.05.2014 um 02:26 schrieb Kohsuke Kawaguchi kkawagu...@cloudbees.com: On 05/12/2014 04:09 PM, Stephen Connolly wrote: You know I have *very* strong feelings on a lot of the lower layers... but you also know that I haven't had the time to get the replacement layer published yet... It's coming... there is the possibility that it may break some stuff... but I am reasonably confident that it shouldn't Good point, so there's at least that group of changes that we should expect before we consider it solidified. On 12 May 2014 23:28, Kohsuke Kawaguchi k...@kohsuke.org mailto:k...@kohsuke.org wrote: So I jumped the gun a little bit and cut a release of acceptance-test-harness, as it was blocking me on developing other sets of tests internally. But this triggered a comment from Oliver [1] in terms of what we want to see in the code before we consider the API stable enough. Specifically, I hoped there will be a discussion before making the API public. I wanted to get rid off all the public Controls exposed from page objects because we will not be able to preserve the API as Jenkins UI evolves. Changes like af98505 will surely follow. So first of all, I wanted to set the expectation right that 1.0 and 1.1 I released shouldn't be considered stable. With that said, personally I didn't anticipate that we hold ourselves to the level of backward compatibility that we do for Jenkins, and public final Control fields are nice, namely they are very concise to write. But I'm willing to follow what we want as a whole, so if you have any strong feelings and thoughts on this, this is the opportunity to speak up. [1] https://github.com/jenkinsci/__acceptance-test-harness/__commit/__8a4bacb386ee0b5a34c5dd499bd74f__50b35c726d#commitcomment-__6297001 https://github.com/jenkinsci/acceptance-test-harness/commit/8a4bacb386ee0b5a34c5dd499bd74f50b35c726d#commitcomment-6297001 -- Kohsuke Kawaguchi http://kohsuke.org/ -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@__googlegroups.com mailto:jenkinsci-dev%2bunsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/__optout https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com mailto:jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Jenkins Enterprise, our professional version of Jenkins -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https
Re: Solidifying acceptance-test-harness API
Oh I forgot my point... I suspect that some of the acceptance tests would
be easier to write if you use the scalability framework's DSL to set up a
lot of the acceptance test... at least they would be less fragile as you
have a known scenario with inter-job relationships. I have found that
setting up complex job scenarios via Selenium can be somewhat fragile and
you end up with brittle tests that fail at random.
I do not think all acceptance tests should use the scalability framework's
Plan model to set up the test environment (other than the bare Plan with
plugins installed) as there is value in testing the path... but the more
complex ones where the setup path is already covered... they can get value
On 13 May 2014 12:21, Stephen Connolly stephen.alan.conno...@gmail.comwrote:
Well I have a scalability test framework that makes it very easy to test
jenkins at scale. I also think that the DSL for scaling jenkins is probably
very handy for acceptace tests. For example, here is my *current* API for
load testing the SSH Slaves connector:
Plan p = new Plan();
Ec2MachineFactory factory = new Ec2MachineFactory(new Ec2Config());
try (Execution e = new Execution(p)
.with(MasterNode.factory())
.with(SlaveNode.factory())
.machines(factory)
.machines(new MultitenantMachinesFactory(factory, 15),
PlannedSlave.class)) {
PlannedMaster master = p.createMaster()
.withPlugins(metrics, random-job-builder)
.withView(new
PlannedCloudBeesDashboardView().withName(Dashboard))
.withPluginsDisabled(translation)
.withExecutors(0);
NioSshSlaveLauncher launcher = new NioSshSlaveLauncher();
for (int i = 0; i 60; i++) {
master.withSlave(p.createSlave().withLauncher(launcher).withExecutors(2));
}
for (int f = 0; f 10; f++) {
PlannedFolder folder = p.createFolder(master);
for (int g = 0; g 3; g++) {
PlannedFolder subfolder = p.createFolder(folder);
for (int h = 0; h 10; h++) {
p.createMockJob(subfolder);
}
}
}
e.start();
MasterNode executionMaster = e.resolve(master);
// give the user a browser window to observe the load
Desktop.getDesktop().browse(URI.create(executionMaster.getJenkinsUrl()+view/Dashboard/));
// wait for the user to request tear-down
}
I have found this *very* powerful for testing load. The above scenario let
me determine, for example, that on an m3.large you can have about 60
concurrent mock load builds using the JNLP launcher, or about 10 concurrent
mock load builds using the Trilead SSH launcher... above those levels by
more than 5 concurrent builds and Jenkins blows up. With the CloudBees
NioSSH launcher you can have 15 concurrent builds without slowing of build
times... and you can have 120 concurrent builds without Jenkins blowing up.
What I need to do is detangle some of the CloudBees specific bits: e.g. in
the above example there are two classes that will not be made public:
PlannedCloudBeesDashboardView and NioSshSlaveLauncher because those classes
will decorate the provisioned Jenkins master with a CloudBees Jenkins
Enterprise license ;-)
I also want to get a hashing function for a Plan so that the JUT server
can be made compatible in the face of requests for specific plans.
On 13 May 2014 09:59, Ulli Hafner ullrich.haf...@gmail.com wrote:
One of our students is currently trying to integrate Spock and Geb: this
also requires API changes (at least we need to introduce additional
interfaces for all those concrete selenium base classes). This part still
is work in progress.
I also thought the purpose of this project is to collect all acceptance
tests for Jenkins. So if we need to change the framework we just go ahead
and commit the required changes. Up to now it was not clear that we need to
publish a „stable“ version at all.
Are there any other plans for additional test suites that are based on
the acceptance test harness project (besides your Cloudbees test suite)?
Am 13.05.2014 um 02:26 schrieb Kohsuke Kawaguchi
kkawagu...@cloudbees.com:
On 05/12/2014 04:09 PM, Stephen Connolly wrote:
You know I have *very* strong feelings on a lot of the lower layers...
but you
also know that I haven't had the time to get the replacement layer
published
yet... It's coming... there is the possibility that it may break some
stuff...
but I am reasonably confident that it shouldn't
Good point, so there's at least that group of changes that we should
expect before we consider it solidified.
On 12 May 2014 23:28, Kohsuke Kawaguchi k...@kohsuke.org mailto:
k...@kohsuke.org
wrote:
So I jumped the
Re: Solidifying acceptance-test-harness API
On Tuesday, May 13, 2014 12:49:54 PM UTC+2, Robert Sandell wrote: As long as the version number reflects API compatibility I don't see any problems with not staying backwards compatible and no direct need for a stable version. This is a good compromise. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Solidifying acceptance-test-harness API
Yes, I think it's quite complementary.
Looking forward to seeing the code land on the acceptance-test repo.
2014-05-13 4:25 GMT-07:00 Stephen Connolly stephen.alan.conno...@gmail.com
:
Oh I forgot my point... I suspect that some of the acceptance tests would
be easier to write if you use the scalability framework's DSL to set up a
lot of the acceptance test... at least they would be less fragile as you
have a known scenario with inter-job relationships. I have found that
setting up complex job scenarios via Selenium can be somewhat fragile and
you end up with brittle tests that fail at random.
I do not think all acceptance tests should use the scalability framework's
Plan model to set up the test environment (other than the bare Plan with
plugins installed) as there is value in testing the path... but the more
complex ones where the setup path is already covered... they can get value
On 13 May 2014 12:21, Stephen Connolly stephen.alan.conno...@gmail.comwrote:
Well I have a scalability test framework that makes it very easy to test
jenkins at scale. I also think that the DSL for scaling jenkins is probably
very handy for acceptace tests. For example, here is my *current* API for
load testing the SSH Slaves connector:
Plan p = new Plan();
Ec2MachineFactory factory = new Ec2MachineFactory(new
Ec2Config());
try (Execution e = new Execution(p)
.with(MasterNode.factory())
.with(SlaveNode.factory())
.machines(factory)
.machines(new MultitenantMachinesFactory(factory, 15),
PlannedSlave.class)) {
PlannedMaster master = p.createMaster()
.withPlugins(metrics, random-job-builder)
.withView(new
PlannedCloudBeesDashboardView().withName(Dashboard))
.withPluginsDisabled(translation)
.withExecutors(0);
NioSshSlaveLauncher launcher = new NioSshSlaveLauncher();
for (int i = 0; i 60; i++) {
master.withSlave(p.createSlave().withLauncher(launcher).withExecutors(2));
}
for (int f = 0; f 10; f++) {
PlannedFolder folder = p.createFolder(master);
for (int g = 0; g 3; g++) {
PlannedFolder subfolder = p.createFolder(folder);
for (int h = 0; h 10; h++) {
p.createMockJob(subfolder);
}
}
}
e.start();
MasterNode executionMaster = e.resolve(master);
// give the user a browser window to observe the load
Desktop.getDesktop().browse(URI.create(executionMaster.getJenkinsUrl()+view/Dashboard/));
// wait for the user to request tear-down
}
I have found this *very* powerful for testing load. The above scenario
let me determine, for example, that on an m3.large you can have about 60
concurrent mock load builds using the JNLP launcher, or about 10 concurrent
mock load builds using the Trilead SSH launcher... above those levels by
more than 5 concurrent builds and Jenkins blows up. With the CloudBees
NioSSH launcher you can have 15 concurrent builds without slowing of build
times... and you can have 120 concurrent builds without Jenkins blowing up.
What I need to do is detangle some of the CloudBees specific bits: e.g.
in the above example there are two classes that will not be made public:
PlannedCloudBeesDashboardView and NioSshSlaveLauncher because those classes
will decorate the provisioned Jenkins master with a CloudBees Jenkins
Enterprise license ;-)
I also want to get a hashing function for a Plan so that the JUT server
can be made compatible in the face of requests for specific plans.
On 13 May 2014 09:59, Ulli Hafner ullrich.haf...@gmail.com wrote:
One of our students is currently trying to integrate Spock and Geb: this
also requires API changes (at least we need to introduce additional
interfaces for all those concrete selenium base classes). This part still
is work in progress.
I also thought the purpose of this project is to collect all acceptance
tests for Jenkins. So if we need to change the framework we just go ahead
and commit the required changes. Up to now it was not clear that we need to
publish a „stable“ version at all.
Are there any other plans for additional test suites that are based on
the acceptance test harness project (besides your Cloudbees test suite)?
Am 13.05.2014 um 02:26 schrieb Kohsuke Kawaguchi
kkawagu...@cloudbees.com:
On 05/12/2014 04:09 PM, Stephen Connolly wrote:
You know I have *very* strong feelings on a lot of the lower
layers... but you
also know that I haven't had the time to get the replacement layer
published
yet... It's coming... there is the possibility that it may break some
stuff...
but I am reasonably confident that it shouldn't
Good point, so there's at least
Solidifying acceptance-test-harness API
So I jumped the gun a little bit and cut a release of acceptance-test-harness, as it was blocking me on developing other sets of tests internally. But this triggered a comment from Oliver [1] in terms of what we want to see in the code before we consider the API stable enough. Specifically, I hoped there will be a discussion before making the API public. I wanted to get rid off all the public Controls exposed from page objects because we will not be able to preserve the API as Jenkins UI evolves. Changes like af98505 will surely follow. So first of all, I wanted to set the expectation right that 1.0 and 1.1 I released shouldn't be considered stable. With that said, personally I didn't anticipate that we hold ourselves to the level of backward compatibility that we do for Jenkins, and public final Control fields are nice, namely they are very concise to write. But I'm willing to follow what we want as a whole, so if you have any strong feelings and thoughts on this, this is the opportunity to speak up. [1] https://github.com/jenkinsci/acceptance-test-harness/commit/8a4bacb386ee0b5a34c5dd499bd74f50b35c726d#commitcomment-6297001 -- Kohsuke Kawaguchi http://kohsuke.org/ -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Solidifying acceptance-test-harness API
You know I have *very* strong feelings on a lot of the lower layers... but you also know that I haven't had the time to get the replacement layer published yet... It's coming... there is the possibility that it may break some stuff... but I am reasonably confident that it shouldn't On 12 May 2014 23:28, Kohsuke Kawaguchi k...@kohsuke.org wrote: So I jumped the gun a little bit and cut a release of acceptance-test-harness, as it was blocking me on developing other sets of tests internally. But this triggered a comment from Oliver [1] in terms of what we want to see in the code before we consider the API stable enough. Specifically, I hoped there will be a discussion before making the API public. I wanted to get rid off all the public Controls exposed from page objects because we will not be able to preserve the API as Jenkins UI evolves. Changes like af98505 will surely follow. So first of all, I wanted to set the expectation right that 1.0 and 1.1 I released shouldn't be considered stable. With that said, personally I didn't anticipate that we hold ourselves to the level of backward compatibility that we do for Jenkins, and public final Control fields are nice, namely they are very concise to write. But I'm willing to follow what we want as a whole, so if you have any strong feelings and thoughts on this, this is the opportunity to speak up. [1] https://github.com/jenkinsci/acceptance-test-harness/commit/ 8a4bacb386ee0b5a34c5dd499bd74f50b35c726d#commitcomment-6297001 -- Kohsuke Kawaguchi http://kohsuke.org/ -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Solidifying acceptance-test-harness API
On 05/12/2014 04:09 PM, Stephen Connolly wrote: You know I have *very* strong feelings on a lot of the lower layers... but you also know that I haven't had the time to get the replacement layer published yet... It's coming... there is the possibility that it may break some stuff... but I am reasonably confident that it shouldn't Good point, so there's at least that group of changes that we should expect before we consider it solidified. On 12 May 2014 23:28, Kohsuke Kawaguchi k...@kohsuke.org mailto:k...@kohsuke.org wrote: So I jumped the gun a little bit and cut a release of acceptance-test-harness, as it was blocking me on developing other sets of tests internally. But this triggered a comment from Oliver [1] in terms of what we want to see in the code before we consider the API stable enough. Specifically, I hoped there will be a discussion before making the API public. I wanted to get rid off all the public Controls exposed from page objects because we will not be able to preserve the API as Jenkins UI evolves. Changes like af98505 will surely follow. So first of all, I wanted to set the expectation right that 1.0 and 1.1 I released shouldn't be considered stable. With that said, personally I didn't anticipate that we hold ourselves to the level of backward compatibility that we do for Jenkins, and public final Control fields are nice, namely they are very concise to write. But I'm willing to follow what we want as a whole, so if you have any strong feelings and thoughts on this, this is the opportunity to speak up. [1] https://github.com/jenkinsci/__acceptance-test-harness/__commit/__8a4bacb386ee0b5a34c5dd499bd74f__50b35c726d#commitcomment-__6297001 https://github.com/jenkinsci/acceptance-test-harness/commit/8a4bacb386ee0b5a34c5dd499bd74f50b35c726d#commitcomment-6297001 -- Kohsuke Kawaguchi http://kohsuke.org/ -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@__googlegroups.com mailto:jenkinsci-dev%2bunsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/__optout https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com mailto:jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Jenkins Enterprise, our professional version of Jenkins -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
