Ok I’ve created a 3rd job for unsupported configs: https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_unsupported/
So we now have: * Job 1: run all tests on latest versions of supported configs * Job 2: run smoke tests (ie a few tests only) on all supported configs * Job 3: run smoke tests on unsupported configs (MySQL 8.x, Tomcat 9.x, etc) Definitions are all here: https://github.com/xwiki/xwiki-jenkins-pipeline/blob/06b679bf511261dbd0125a352a5f9074746aff83/src/org/xwiki/jenkins/DockerTests.groovy Thanks -Vincent > On 18 Nov 2018, at 16:57, Vincent Massol <vinc...@massol.net> wrote: > > > >> On 16 Nov 2018, at 22:03, Vincent Massol <vinc...@massol.net> wrote: >> >> Hi devs, >> >> So we now have 2 dockers jobs: >> - one that executes daily and runs the latest version of the supported >> configs. It runs all docker tests (right now that’s MenuIT but I’ll start >> converting some other tests and we need to convert them all). See >> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/ >> - another one that executes weekly and that runs all supported configs but >> only on smoke tests (right now that MenuIT only). See >> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/ > > And to see which configs are tested and passing or not: > https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/lastCompletedBuild/testReport/org.xwiki.menu.test.ui/MenuIT/ > > Thanks > -Vincent > >> All are passing FTM :) >> >> Thanks >> -Vincent >> >>> On 5 Nov 2018, at 17:22, Vincent Massol <vinc...@massol.net> wrote: >>> >>> Hi devs, >>> >>> I’m still not sure but FTM I was thinking of having 2 pipeline jobs: >>> >>> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the >>> maximum number of configurations, in order to flesh out configs that don’t >>> start properly. For example XWiki on Tomcat 9.x would fail (since the >>> Tomcat 9.x docker image uses java9+). The job would not send a mail on >>> failure but it would update a report page (could even update a page on >>> xwiki.org directly or if too complex update some page on maven.xwiki.org >>> somewhere). This job would run not very often but say once per week. Note >>> that one config takes 3-4 minutes to run, so 50 configs would take 3 hours >>> which is acceptable. >>> >>> 2) Job 2: Execute all functional tests on a subset of supported configs. >>> For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome >>> if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job >>> will take a long time to execute. We’ll start with 3-4 configs and will go >>> to about 10 configs when we add more. The tests will take roughly 2 hours >>> to execute per config I think. So a total of 20 hours when we have 10 >>> configs. If we run those once per week it should be fine. >>> >>> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add >>> as part of the platform JenkinsFile. >>> >>> WDYT? >>> >>> Thanks >>> -Vincent >>> >> >
