On Wednesday, 4 June 2014, Ulli Hafner <[email protected]> wrote:
> Well, I think it would help if especially a team that develops a CI server > would use the TDD approach to introduce new changes (eat your own > dogfood!). So unrelated to the discussion if system tests (or acceptance > tests) are valuable I would never try to commit new changes (especially > large changes as you are describing) before all tests are GREEN. This is a > basic principle that we should use as Jenkins dev team. So if a tests > breaks due to a change, we should fix the bug or test (in the LDAP case the > test needed to get an update to the new LDAP UI configuration, this is > already fixed now - thanks to the work of Michael…) > > The use case you are describing (i.e. introduce additional integration > tests that work on injected xml files) is a totally different one and makes > sense too. I don’t see that we need to decide between two options, both are > valid. But since we are mostly part time developers we should focus only on > things we can actually handle in our valuable spare time. So I would > suggest to improve the existing UI tests first until we think they are in a > good shape before we continue with the next area of integration tests. > I.e., one step after the other… > Well I disagree on that point. I feel my time can deliver more value to the community if I integrate the scalability test framework I have developed into the acceptance test harness to enable this richer set of tests... Hence why I originally stated I would work on getting the scalability framework integrated first *and then* fix the tests of LDAP If others want to continue on UI driven tests that is their decision on where to put their time. In 1-2 weeks time I hope to have the scalability framework integrated and people will then be able to decide what to do with their existing tests... My view is that most of them should be reworked to drive via the scalability framework, but if others disagree that us their business and we will find out over time if my experiences of heavy UI driven tests end up poisoning the community as I fear or if my fears are unfounded. > I have also seen projects dumping their UI tests due to the amount of work > required to keep them up to date with the development (mostly due to > lacking framework functionality). But I also worked in projects that got > these test evolving with the underlying systems without any problems. This > is a matter of design of the UI and the test cases and I think we will > manage this in Jenkins and keep our test suite up to date without much > work). > > Am 04.06.2014 um 23:28 schrieb Stephen Connolly < > [email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>>: > > > > On Wednesday, 4 June 2014, oliver gondža <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > >> Stephen, >> >> You suggest strictly separating UI tests and behavior tests, but using >> backdoor manipulation/verification via xml configuration is no longer >> end-to-end testing. Not to mention that plugin config is not exposed to >> user as XML. > > > But the xml file is what the user has when they upgrade. > > I am suggesting breaking the tests into two. Splits the bits that generate > the config from the hits that test the config works. You still have end to > end tested, but in two steps and you now have tests that don't need to > change with the UI. > > I have seen the pattern of UI driven tests going end to end fail over and > over as the system under test evolves. > > Further it becomes a pain to re validate all the many many test cases as > the page objects are updated to ensure that you have not introduced false > positives. This is what eventually leads to EITHER all the tests being > thrown out and rewritten from scratch OR changes in the UI being rejected > because they break too many tests... > > Not to mind that the number one question people have is: > > "If I upgrade my Jenkins will all my existing jobs still work the same" > > Contrast with the current pure UI driven tests that answer a different > question: > > "If I download the new version of Jenkins and install it on a new server, > can I configure new jobs that behave the same as the jobs I had on my old > Jenkins" > > I argue that this second question, while important, is nowhere even within > an order of magnitude close to the importance of the first question. > > >> Plus, if we move test fixtures to xml and the plugin persistence evolves >> (as in this case), we are testing that plugin is able to load obsolete >> config and not that it can be configured to do the same using its new >> version. > > > Yes which is why you need to add new tests for the new configuration > schema... Plus if the configuration changes you will have the UI test fail > as they will fail to generate the xml configured as before, so you get an > alert of this case. > > >> Also note that with all the test we currently have, only a small part of >> Jenkins UI is covered. The goal of covering UI in small portion of tests >> and then focusing on the behavior does not sound achievable to me. In >> effect we will have less UI coverage. > > > Well the point is the current tests are only testing the UI in respect of > the WHEN-THEN parts of the tests. You are fooling yourself if you think the > exercising of the UI in the GIVEN part of the test is testing the UI. > > I am recommending that the GIVEN part of the tests not drive by UI. Thus > we will not have a lower actual testing of the UI... We will have the same > amount of UI coverage and we will have removed the false sense of security > people currently are giving themselves with the GIVEN exercising of the UI > > >> >> -- >> oliver >> > > > -- > Sent from my phone > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <javascript:_e(%7B%7D,'cvml','jenkinsci-dev%[email protected]');> > . > For more options, visit https://groups.google.com/d/optout. > > > -- Sent from my phone -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
