On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <vinc...@massol.net> wrote: > > Hi devs, > > I’m still not sure but FTM I was thinking of having 2 pipeline jobs: > > 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the > maximum number of configurations, in order to flesh out configs that don’t > start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat > 9.x docker image uses java9+). The job would not send a mail on failure but > it would update a report page (could even update a page on xwiki.org directly > or if too complex update some page on maven.xwiki.org somewhere). This job > would run not very often but say once per week. Note that one config takes > 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
That's interesting indeed. +1 > > 2) Job 2: Execute all functional tests on a subset of supported configs. For > example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we > already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will > take a long time to execute. We’ll start with 3-4 configs and will go to > about 10 configs when we add more. The tests will take roughly 2 hours to > execute per config I think. So a total of 20 hours when we have 10 configs. > If we run those once per week it should be fine. Can't those configs be executed in parallel in different agents ? > > Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as > part of the platform JenkinsFile. > > WDYT? > > Thanks > -Vincent > -- Thomas Mortagne
