On Wednesday, 4 June 2014, oliver gondža <[email protected]> wrote: > Stephen, > > You suggest strictly separating UI tests and behavior tests, but using > backdoor manipulation/verification via xml configuration is no longer > end-to-end testing. Not to mention that plugin config is not exposed to > user as XML.
But the xml file is what the user has when they upgrade. I am suggesting breaking the tests into two. Splits the bits that generate the config from the hits that test the config works. You still have end to end tested, but in two steps and you now have tests that don't need to change with the UI. I have seen the pattern of UI driven tests going end to end fail over and over as the system under test evolves. Further it becomes a pain to re validate all the many many test cases as the page objects are updated to ensure that you have not introduced false positives. This is what eventually leads to EITHER all the tests being thrown out and rewritten from scratch OR changes in the UI being rejected because they break too many tests... Not to mind that the number one question people have is: "If I upgrade my Jenkins will all my existing jobs still work the same" Contrast with the current pure UI driven tests that answer a different question: "If I download the new version of Jenkins and install it on a new server, can I configure new jobs that behave the same as the jobs I had on my old Jenkins" I argue that this second question, while important, is nowhere even within an order of magnitude close to the importance of the first question. > Plus, if we move test fixtures to xml and the plugin persistence evolves > (as in this case), we are testing that plugin is able to load obsolete > config and not that it can be configured to do the same using its new > version. Yes which is why you need to add new tests for the new configuration schema... Plus if the configuration changes you will have the UI test fail as they will fail to generate the xml configured as before, so you get an alert of this case. > Also note that with all the test we currently have, only a small part of > Jenkins UI is covered. The goal of covering UI in small portion of tests > and then focusing on the behavior does not sound achievable to me. In > effect we will have less UI coverage. Well the point is the current tests are only testing the UI in respect of the WHEN-THEN parts of the tests. You are fooling yourself if you think the exercising of the UI in the GIVEN part of the test is testing the UI. I am recommending that the GIVEN part of the tests not drive by UI. Thus we will not have a lower actual testing of the UI... We will have the same amount of UI coverage and we will have removed the false sense of security people currently are giving themselves with the GIVEN exercising of the UI > > -- > oliver > -- Sent from my phone -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
