Hi devs, I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable. 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine. Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile. WDYT? Thanks -Vincent
