Re: [xwiki-devs] [Proposal] Jobs for the multiple configuration testing
Self reminder: Just realized that I now need to modify the clover pipeline so that we now run docker-based tests too! This means the clover job can only run on agent4. This also means that I’ll need to setup FF on agent4 for non-docker-based functional tests. Thanks -Vincent > On 18 Nov 2018, at 17:49, Vincent Massol wrote: > > Ok I’ve created a 3rd job for unsupported configs: > https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_unsupported/ > > So we now have: > * Job 1: run all tests on latest versions of supported configs > * Job 2: run smoke tests (ie a few tests only) on all supported configs > * Job 3: run smoke tests on unsupported configs (MySQL 8.x, Tomcat 9.x, etc) > > Definitions are all here: > https://github.com/xwiki/xwiki-jenkins-pipeline/blob/06b679bf511261dbd0125a352a5f9074746aff83/src/org/xwiki/jenkins/DockerTests.groovy > > Thanks > -Vincent > > >> On 18 Nov 2018, at 16:57, Vincent Massol wrote: >> >> >> >>> On 16 Nov 2018, at 22:03, Vincent Massol wrote: >>> >>> Hi devs, >>> >>> So we now have 2 dockers jobs: >>> - one that executes daily and runs the latest version of the supported >>> configs. It runs all docker tests (right now that’s MenuIT but I’ll start >>> converting some other tests and we need to convert them all). See >>> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/ >>> - another one that executes weekly and that runs all supported configs but >>> only on smoke tests (right now that MenuIT only). See >>> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/ >> >> And to see which configs are tested and passing or not: >> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/lastCompletedBuild/testReport/org.xwiki.menu.test.ui/MenuIT/ >> >> Thanks >> -Vincent >> >>> All are passing FTM :) >>> >>> Thanks >>> -Vincent >>> On 5 Nov 2018, at 17:22, Vincent Massol wrote: Hi devs, I’m still not sure but FTM I was thinking of having 2 pipeline jobs: 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable. 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine. Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile. WDYT? Thanks -Vincent >>> >> >
Re: [xwiki-devs] [Proposal] Jobs for the multiple configuration testing
Ok I’ve created a 3rd job for unsupported configs: https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_unsupported/ So we now have: * Job 1: run all tests on latest versions of supported configs * Job 2: run smoke tests (ie a few tests only) on all supported configs * Job 3: run smoke tests on unsupported configs (MySQL 8.x, Tomcat 9.x, etc) Definitions are all here: https://github.com/xwiki/xwiki-jenkins-pipeline/blob/06b679bf511261dbd0125a352a5f9074746aff83/src/org/xwiki/jenkins/DockerTests.groovy Thanks -Vincent > On 18 Nov 2018, at 16:57, Vincent Massol wrote: > > > >> On 16 Nov 2018, at 22:03, Vincent Massol wrote: >> >> Hi devs, >> >> So we now have 2 dockers jobs: >> - one that executes daily and runs the latest version of the supported >> configs. It runs all docker tests (right now that’s MenuIT but I’ll start >> converting some other tests and we need to convert them all). See >> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/ >> - another one that executes weekly and that runs all supported configs but >> only on smoke tests (right now that MenuIT only). See >> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/ > > And to see which configs are tested and passing or not: > https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/lastCompletedBuild/testReport/org.xwiki.menu.test.ui/MenuIT/ > > Thanks > -Vincent > >> All are passing FTM :) >> >> Thanks >> -Vincent >> >>> On 5 Nov 2018, at 17:22, Vincent Massol wrote: >>> >>> Hi devs, >>> >>> I’m still not sure but FTM I was thinking of having 2 pipeline jobs: >>> >>> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the >>> maximum number of configurations, in order to flesh out configs that don’t >>> start properly. For example XWiki on Tomcat 9.x would fail (since the >>> Tomcat 9.x docker image uses java9+). The job would not send a mail on >>> failure but it would update a report page (could even update a page on >>> xwiki.org directly or if too complex update some page on maven.xwiki.org >>> somewhere). This job would run not very often but say once per week. Note >>> that one config takes 3-4 minutes to run, so 50 configs would take 3 hours >>> which is acceptable. >>> >>> 2) Job 2: Execute all functional tests on a subset of supported configs. >>> For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome >>> if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job >>> will take a long time to execute. We’ll start with 3-4 configs and will go >>> to about 10 configs when we add more. The tests will take roughly 2 hours >>> to execute per config I think. So a total of 20 hours when we have 10 >>> configs. If we run those once per week it should be fine. >>> >>> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add >>> as part of the platform JenkinsFile. >>> >>> WDYT? >>> >>> Thanks >>> -Vincent >>> >> >
Re: [xwiki-devs] [Proposal] Jobs for the multiple configuration testing
> On 16 Nov 2018, at 22:03, Vincent Massol wrote: > > Hi devs, > > So we now have 2 dockers jobs: > - one that executes daily and runs the latest version of the supported > configs. It runs all docker tests (right now that’s MenuIT but I’ll start > converting some other tests and we need to convert them all). See > https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/ > - another one that executes weekly and that runs all supported configs but > only on smoke tests (right now that MenuIT only). See > https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/ And to see which configs are tested and passing or not: https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/lastCompletedBuild/testReport/org.xwiki.menu.test.ui/MenuIT/ Thanks -Vincent > All are passing FTM :) > > Thanks > -Vincent > >> On 5 Nov 2018, at 17:22, Vincent Massol wrote: >> >> Hi devs, >> >> I’m still not sure but FTM I was thinking of having 2 pipeline jobs: >> >> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the >> maximum number of configurations, in order to flesh out configs that don’t >> start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat >> 9.x docker image uses java9+). The job would not send a mail on failure but >> it would update a report page (could even update a page on xwiki.org >> directly or if too complex update some page on maven.xwiki.org somewhere). >> This job would run not very often but say once per week. Note that one >> config takes 3-4 minutes to run, so 50 configs would take 3 hours which is >> acceptable. >> >> 2) Job 2: Execute all functional tests on a subset of supported configs. For >> example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we >> already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will >> take a long time to execute. We’ll start with 3-4 configs and will go to >> about 10 configs when we add more. The tests will take roughly 2 hours to >> execute per config I think. So a total of 20 hours when we have 10 configs. >> If we run those once per week it should be fine. >> >> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as >> part of the platform JenkinsFile. >> >> WDYT? >> >> Thanks >> -Vincent >> >
Re: [xwiki-devs] [Proposal] Jobs for the multiple configuration testing
Hi devs, So we now have 2 dockers jobs: - one that executes daily and runs the latest version of the supported configs. It runs all docker tests (right now that’s MenuIT but I’ll start converting some other tests and we need to convert them all). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/ - another one that executes weekly and that runs all supported configs but only on smoke tests (right now that MenuIT only). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/ All are passing FTM :) Thanks -Vincent > On 5 Nov 2018, at 17:22, Vincent Massol wrote: > > Hi devs, > > I’m still not sure but FTM I was thinking of having 2 pipeline jobs: > > 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the > maximum number of configurations, in order to flesh out configs that don’t > start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat > 9.x docker image uses java9+). The job would not send a mail on failure but > it would update a report page (could even update a page on xwiki.org directly > or if too complex update some page on maven.xwiki.org somewhere). This job > would run not very often but say once per week. Note that one config takes > 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable. > > 2) Job 2: Execute all functional tests on a subset of supported configs. For > example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we > already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will > take a long time to execute. We’ll start with 3-4 configs and will go to > about 10 configs when we add more. The tests will take roughly 2 hours to > execute per config I think. So a total of 20 hours when we have 10 configs. > If we run those once per week it should be fine. > > Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as > part of the platform JenkinsFile. > > WDYT? > > Thanks > -Vincent >
Re: [xwiki-devs] [Proposal] Jobs for the multiple configuration testing
+1 for both but we need to push for more powerful CI agents. ATM the build is faster on my laptop. Thanks, Marius On Mon, Nov 5, 2018 at 6:22 PM Vincent Massol wrote: > Hi devs, > > I’m still not sure but FTM I was thinking of having 2 pipeline jobs: > > 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on > the maximum number of configurations, in order to flesh out configs that > don’t start properly. For example XWiki on Tomcat 9.x would fail (since the > Tomcat 9.x docker image uses java9+). The job would not send a mail on > failure but it would update a report page (could even update a page on > xwiki.org directly or if too complex update some page on maven.xwiki.org > somewhere). This job would run not very often but say once per week. Note > that one config takes 3-4 minutes to run, so 50 configs would take 3 hours > which is acceptable. > > 2) Job 2: Execute all functional tests on a subset of supported configs. > For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome > if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job > will take a long time to execute. We’ll start with 3-4 configs and will go > to about 10 configs when we add more. The tests will take roughly 2 hours > to execute per config I think. So a total of 20 hours when we have 10 > configs. If we run those once per week it should be fine. > > Note: Once we have job 1 & 2, we won't need to have the smoke tests I add > as part of the platform JenkinsFile. > > WDYT? > > Thanks > -Vincent > >
Re: [xwiki-devs] [Proposal] Jobs for the multiple configuration testing
Hello, I'm +1 to try having both jobs. Thanks, Adel On Wed, Nov 7, 2018 at 9:20 AM Vincent Massol wrote: > > Hi Thomas, > > > On 7 Nov 2018, at 08:44, Thomas Mortagne wrote: > > > > On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol wrote: > >> > >> Hi devs, > >> > >> I’m still not sure but FTM I was thinking of having 2 pipeline jobs: > >> > >> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on > >> the maximum number of configurations, in order to flesh out configs that > >> don’t start properly. For example XWiki on Tomcat 9.x would fail (since > >> the Tomcat 9.x docker image uses java9+). The job would not send a mail on > >> failure but it would update a report page (could even update a page on > >> xwiki.org directly or if too complex update some page on maven.xwiki.org > >> somewhere). This job would run not very often but say once per week. Note > >> that one config takes 3-4 minutes to run, so 50 configs would take 3 hours > >> which is acceptable. > > > > That's interesting indeed. +1 > > > >> > >> 2) Job 2: Execute all functional tests on a subset of supported configs. > >> For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome > >> if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This > >> job will take a long time to execute. We’ll start with 3-4 configs and > >> will go to about 10 configs when we add more. The tests will take roughly > >> 2 hours to execute per config I think. So a total of 20 hours when we have > >> 10 configs. If we run those once per week it should be fine. > > > > Can't those configs be executed in parallel in different agents ? > > Yes this is fully parallelizable. However we have only one agent with docker > ATM. Once we migrate more agents then we can have them in parallel. > > Thanks > -Vincent > > > > >> > >> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add > >> as part of the platform JenkinsFile. > >> > >> WDYT? > >> > >> Thanks > >> -Vincent > >> > > > > > > -- > > Thomas Mortagne >
Re: [xwiki-devs] [Proposal] Jobs for the multiple configuration testing
On Wed, Nov 7, 2018 at 9:20 AM Vincent Massol wrote: > > Hi Thomas, > > > On 7 Nov 2018, at 08:44, Thomas Mortagne wrote: > > > > On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol wrote: > >> > >> Hi devs, > >> > >> I’m still not sure but FTM I was thinking of having 2 pipeline jobs: > >> > >> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on > >> the maximum number of configurations, in order to flesh out configs that > >> don’t start properly. For example XWiki on Tomcat 9.x would fail (since > >> the Tomcat 9.x docker image uses java9+). The job would not send a mail on > >> failure but it would update a report page (could even update a page on > >> xwiki.org directly or if too complex update some page on maven.xwiki.org > >> somewhere). This job would run not very often but say once per week. Note > >> that one config takes 3-4 minutes to run, so 50 configs would take 3 hours > >> which is acceptable. > > > > That's interesting indeed. +1 > > > >> > >> 2) Job 2: Execute all functional tests on a subset of supported configs. > >> For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome > >> if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This > >> job will take a long time to execute. We’ll start with 3-4 configs and > >> will go to about 10 configs when we add more. The tests will take roughly > >> 2 hours to execute per config I think. So a total of 20 hours when we have > >> 10 configs. If we run those once per week it should be fine. > > > > Can't those configs be executed in parallel in different agents ? > > Yes this is fully parallelizable. However we have only one agent with docker > ATM. Once we migrate more agents then we can have them in parallel. Indeed forgot about that. Anyway +1 for both > > Thanks > -Vincent > > > > >> > >> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add > >> as part of the platform JenkinsFile. > >> > >> WDYT? > >> > >> Thanks > >> -Vincent > >> > > > > > > -- > > Thomas Mortagne > -- Thomas Mortagne
Re: [xwiki-devs] [Proposal] Jobs for the multiple configuration testing
Hi Thomas, > On 7 Nov 2018, at 08:44, Thomas Mortagne wrote: > > On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol wrote: >> >> Hi devs, >> >> I’m still not sure but FTM I was thinking of having 2 pipeline jobs: >> >> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the >> maximum number of configurations, in order to flesh out configs that don’t >> start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat >> 9.x docker image uses java9+). The job would not send a mail on failure but >> it would update a report page (could even update a page on xwiki.org >> directly or if too complex update some page on maven.xwiki.org somewhere). >> This job would run not very often but say once per week. Note that one >> config takes 3-4 minutes to run, so 50 configs would take 3 hours which is >> acceptable. > > That's interesting indeed. +1 > >> >> 2) Job 2: Execute all functional tests on a subset of supported configs. For >> example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we >> already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will >> take a long time to execute. We’ll start with 3-4 configs and will go to >> about 10 configs when we add more. The tests will take roughly 2 hours to >> execute per config I think. So a total of 20 hours when we have 10 configs. >> If we run those once per week it should be fine. > > Can't those configs be executed in parallel in different agents ? Yes this is fully parallelizable. However we have only one agent with docker ATM. Once we migrate more agents then we can have them in parallel. Thanks -Vincent > >> >> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as >> part of the platform JenkinsFile. >> >> WDYT? >> >> Thanks >> -Vincent >> > > > -- > Thomas Mortagne
Re: [xwiki-devs] [Proposal] Jobs for the multiple configuration testing
On 07/11/2018 08:44, Thomas Mortagne wrote: On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol wrote: Hi devs, I’m still not sure but FTM I was thinking of having 2 pipeline jobs: 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable. That's interesting indeed. +1 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine. Can't those configs be executed in parallel in different agents ? Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile. WDYT? +1 for both jobs Thanks -Vincent -- Simon Urli Software Engineer at XWiki SAS simon.u...@xwiki.com More about us at http://www.xwiki.com
Re: [xwiki-devs] [Proposal] Jobs for the multiple configuration testing
On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol wrote: > > Hi devs, > > I’m still not sure but FTM I was thinking of having 2 pipeline jobs: > > 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the > maximum number of configurations, in order to flesh out configs that don’t > start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat > 9.x docker image uses java9+). The job would not send a mail on failure but > it would update a report page (could even update a page on xwiki.org directly > or if too complex update some page on maven.xwiki.org somewhere). This job > would run not very often but say once per week. Note that one config takes > 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable. That's interesting indeed. +1 > > 2) Job 2: Execute all functional tests on a subset of supported configs. For > example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we > already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will > take a long time to execute. We’ll start with 3-4 configs and will go to > about 10 configs when we add more. The tests will take roughly 2 hours to > execute per config I think. So a total of 20 hours when we have 10 configs. > If we run those once per week it should be fine. Can't those configs be executed in parallel in different agents ? > > Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as > part of the platform JenkinsFile. > > WDYT? > > Thanks > -Vincent > -- Thomas Mortagne
[xwiki-devs] [Proposal] Jobs for the multiple configuration testing
Hi devs, I’m still not sure but FTM I was thinking of having 2 pipeline jobs: 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable. 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine. Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile. WDYT? Thanks -Vincent
