[
https://jira.nuxeo.com/browse/NXBT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Carsique updated NXBT-348:
---------------------------------
Description:
h2. Technical dept on Test
h3. Non Regression tests
- rewrite Selenium tests in WebDriver
=> make them more modular and easier to debug
- improve Nuxeo HTML layout to make tests easier
=> make tests more robust
h3. Funkload tests
Currently only Ben can really fix a broken test.
We should probably improve training and documentation so that not only Ben can
fix them when needed.
h3. Multi-DB / Multi-OS
MultiDB and Multi-OS tests matrix have been added recently.
There is still a lot of work to be done so that they all run :
- fix unit tests on MSSQL and MySQL (h2, PGSQL et Oracle are ok)
- Windows : there are still failed unit tests under windows
- OpenJDK 7 : there are still failed unit tests under windows
Non regression testing in Matrix should also be fixed but it's probably
pointless until all unit tests are fixed.
h2. CI infrastructure speedup
The goal is to make CI feedback quicker.
We can not expect CI feedback to take a only few minutes :
- build is big
- we have a lot of tests in different configuration
But there are some actions we can do to improve the feedback time.
h3. fix Nexus / Jenkins concurrency issue
Nexus and Jenkins master share the same server and the same NAS.
- Jenkins Master uses NAS storage to store the workspace (build and test
results)
- Nexus uses NAS to store the repo
Monitoring shows that the server has a load average about 2 and that Jenkins or
Nexus are faster when the other service is off.
This means we should work on the global optimization of this Nexus/Jenkins
setup : that will globalling improve speed of all CI :
- for fetching / pushing artifacts
- for storing results
Based on Ben's remarks, the main lead for improvement seems to be the NFS share
:
- separate NASs for Jenkins / Nexus ?
- use local storage instead of NAS ?
- NFS / NAS tunning ?
- ...
h3. GWT Build issue
GWT build is really very slow and represent a big part of the global Nuxeo
build time event if we only have a few (3 or 4) GWT modules.
=> This needs to be fixed, both for CI and for developpers
Identified solutions include :
- change profile to limit the number of target browser (meaning nb of
permitation) during the standard Build
( we will still need to do the full build for releases)
- disable GWT optimizations during standard builds
- be sure to only trigger the GWT build when needed
See
http://stackoverflow.com/questions/1011863/how-do-i-speed-up-the-gwt-compiler
h3. Slave build performances
Currently the build on the Jenkins slaves is about the same time than on a dev
laptop.
=> we can make better !
It's true that a big part of the Jenkins slave are poor hardware (the dwarf).
Before taking any decision, we should probably do some profiling to see exactly
what are the real impacting factors :
- test build on a bigger CPU (like Octo)
- use RAMFS to speedup I/O
- tweak compile options ?
- test on big RackSpace VMs
=> based on that we will be able to define what kind of Build slaves we need
(then we will have to see what we can host internaly and what must be external)
h2. CI Config
h3. Permanent checks
Permanent checks currently don't wait for nuxeo-features to be built but won't
raise relevant issues until nuxeo-features is built.
In cases where there are a lot of commits, nuxeo-features build can take
several hours.
=> need to change that
So basically, we could have permanent checks be closer to the IT builds :
- rebuild everything from zero
- do build / no UT / distrib on a fast slave
- dispatch non-reg tests on different hosts
h3. Addons
Current distribution build depends on some addons: the one that are considered
standard and used in distribs / MP packages.
=> Create a job that build all "standard" addons so that distribution does not
have to wait for 20 jobs
h3. JBoss distrib
We need to update JBoss distribution to align on tomcat/MP build.
h2. Tests to add
There are still some part of the platform that are poorly / not enough tested
h3. Studio
XXX Let Anahide list what she has in mind, but basically :
- define a standard Studio project
- run build against different targets
- check result in a running Nuxeo on several target versions
h3.Admin Center / Marketplace
We already have a webdriver test that tests the basis.
We need to extend it so that we validate for example :
- add CMF / uninstall DM
- Install DAM / uninstall DAM + install CMF
- ...
XXX more here
was:
h2. Technical dept on Test
h3. Non Regression tests
- rewrite Selenium tests in WebDriver
=> make them more modular and easier to debug
- improve Nuxeo HTML layout to make tests easier
=> make tests more robust
h3. Funkload tests
Currently only Ben can really fix a broken test.
We should probably improve training and documentation so that not only Ben can
fix them when needed.
h3. Multi-DB / Multi-OS
MultiDB and Multi-OS tests matrix have been added recently.
There is still a lot of work to be done so that they all run :
- fix unit tests on MSSQL and MySQL (h2, PGSQL et Oracle are ok)
- Windows : there are still failed unit tests under windows
- OpenJDK 7 : there are still failed unit tests under windows
Non regression testing in Matrix should also be fixed but it's probably
pointless until all unit tests are fixed.
h2. CI infrastructure speedup
The goal is to make CI feedback quicker.
We can not expect CI feedback to take a only few minutes :
- build is big
- we have a lot of tests in different configuration
But there are some actions we can do to improve the feedback time.
h3. fix Nexus / Jenkins concurrency issue
Nexus and Jenkins master share the same server and the same NAS.
- Jenkins Master uses NAS storage to store the workspace (build and test
results)
- Nexus uses NAS to store the repo
Monitoring shows that the server has a load average about 2 and that Jenkins or
Nexus are faster when the other service is off.
This means we should work on the global optimization of this Nexus/Jenkins
setup : that will globalling improve speed of all CI :
- for fetching / pushing artifacts
- for storing results
Based on Ben's remarks, the main lead for improvement seems to be the NFS share
:
- separate NASs for Jenkins / Nexus ?
- use local storage instead of NAS ?
- NFS / NAS tunning ?
- ...
h3. GWT Build issue
GWT build is really very slow and represent a big part of the global Nuxeo
build time event if we only have a few (3 or 4) GWT modules.
=> This needs to be fixed, both for CI and for developpers
Identified solutions include :
- change profile to limit the number of target browser (meaning nb of
permitation) during the standard Build
( we will still need to do the full build for releases)
- disable GWT optimizations during standard builds
- be sure to only trigger the GWT build when needed
See
http://stackoverflow.com/questions/1011863/how-do-i-speed-up-the-gwt-compiler
h3. Slave build performances
Currently the build on the Jenkins slaves is about the same time than on a dev
laptop.
=> we can make better !
It's true that a big part of the Jenkins slave are poor hardware (the dwarf).
Before taking any decision, we should probably do some profiling to see exactly
what are the real impacting factors :
- test build on a bigger CPU (like Octo)
- use RAMFS to speedup I/O
- tweak compile options ?
- test on big RackSpace VMs
=> based on that we will be able to define what kind of Build slaves we need
(then we will have to see what we can host internaly and what must be external)
h2. CI Config
h3. Permanent checks
Permanent checks currently wait for feature to be built.
In cases where there are a lot of commits, features build can take several
hours.
=> need to change that
So basically, we could have permanent checks be closer to the IT builds :
- rebuild everything from zero
- do build / no TU / distrib on a fast slave
- dispatch non-reg tests on different hosts
h3. Addons
Current distibution build depends on some addons : the one that are considered
standard and used in distribs / MP packages.
=> Create a job that build all "standard" addons so that distribution does not
have to wait for 20 jobs
h3. JBoss distrib
We need to update JBoss distribution to align on tomat/MP build.
h2. Tests to add
There are still some part of the platform that are poorly / not enough tested
h3. Studio
XXX Let Anahide list what she has in mind, but basically :
- define a standard Studio project
- run build against different targets
- check result in a running Nuxeo on several target versions
h3.Admin Center / Marketplace
We already have a webdriver test that tests the basis.
We need to extend it so that we validate for exemple :
- add CMF / desinstall DM
- Install DAM / DAM + install CMF
- ...
XXX more here
> Improve CI and Testing framework
> --------------------------------
>
> Key: NXBT-348
> URL: https://jira.nuxeo.com/browse/NXBT-348
> Project: Nuxeo ECM Build/Test Environment
> Issue Type: User story
> Reporter: Thierry Delprat
> Assignee: Julien Carsique
>
> h2. Technical dept on Test
> h3. Non Regression tests
> - rewrite Selenium tests in WebDriver
> => make them more modular and easier to debug
> - improve Nuxeo HTML layout to make tests easier
> => make tests more robust
> h3. Funkload tests
> Currently only Ben can really fix a broken test.
> We should probably improve training and documentation so that not only Ben
> can fix them when needed.
> h3. Multi-DB / Multi-OS
> MultiDB and Multi-OS tests matrix have been added recently.
> There is still a lot of work to be done so that they all run :
> - fix unit tests on MSSQL and MySQL (h2, PGSQL et Oracle are ok)
> - Windows : there are still failed unit tests under windows
> - OpenJDK 7 : there are still failed unit tests under windows
> Non regression testing in Matrix should also be fixed but it's probably
> pointless until all unit tests are fixed.
> h2. CI infrastructure speedup
> The goal is to make CI feedback quicker.
> We can not expect CI feedback to take a only few minutes :
> - build is big
> - we have a lot of tests in different configuration
> But there are some actions we can do to improve the feedback time.
> h3. fix Nexus / Jenkins concurrency issue
> Nexus and Jenkins master share the same server and the same NAS.
> - Jenkins Master uses NAS storage to store the workspace (build and test
> results)
> - Nexus uses NAS to store the repo
> Monitoring shows that the server has a load average about 2 and that Jenkins
> or Nexus are faster when the other service is off.
> This means we should work on the global optimization of this Nexus/Jenkins
> setup : that will globalling improve speed of all CI :
> - for fetching / pushing artifacts
> - for storing results
> Based on Ben's remarks, the main lead for improvement seems to be the NFS
> share :
> - separate NASs for Jenkins / Nexus ?
> - use local storage instead of NAS ?
> - NFS / NAS tunning ?
> - ...
> h3. GWT Build issue
> GWT build is really very slow and represent a big part of the global Nuxeo
> build time event if we only have a few (3 or 4) GWT modules.
> => This needs to be fixed, both for CI and for developpers
> Identified solutions include :
> - change profile to limit the number of target browser (meaning nb of
> permitation) during the standard Build
> ( we will still need to do the full build for releases)
> - disable GWT optimizations during standard builds
> - be sure to only trigger the GWT build when needed
> See
> http://stackoverflow.com/questions/1011863/how-do-i-speed-up-the-gwt-compiler
> h3. Slave build performances
> Currently the build on the Jenkins slaves is about the same time than on a
> dev laptop.
> => we can make better !
> It's true that a big part of the Jenkins slave are poor hardware (the dwarf).
> Before taking any decision, we should probably do some profiling to see
> exactly what are the real impacting factors :
> - test build on a bigger CPU (like Octo)
> - use RAMFS to speedup I/O
> - tweak compile options ?
> - test on big RackSpace VMs
> => based on that we will be able to define what kind of Build slaves we need
> (then we will have to see what we can host internaly and what must be
> external)
> h2. CI Config
> h3. Permanent checks
> Permanent checks currently don't wait for nuxeo-features to be built but
> won't raise relevant issues until nuxeo-features is built.
> In cases where there are a lot of commits, nuxeo-features build can take
> several hours.
> => need to change that
> So basically, we could have permanent checks be closer to the IT builds :
> - rebuild everything from zero
> - do build / no UT / distrib on a fast slave
> - dispatch non-reg tests on different hosts
> h3. Addons
> Current distribution build depends on some addons: the one that are
> considered standard and used in distribs / MP packages.
> => Create a job that build all "standard" addons so that distribution does
> not have to wait for 20 jobs
> h3. JBoss distrib
> We need to update JBoss distribution to align on tomcat/MP build.
> h2. Tests to add
> There are still some part of the platform that are poorly / not enough tested
> h3. Studio
> XXX Let Anahide list what she has in mind, but basically :
> - define a standard Studio project
> - run build against different targets
> - check result in a running Nuxeo on several target versions
> h3.Admin Center / Marketplace
> We already have a webdriver test that tests the basis.
> We need to extend it so that we validate for example :
> - add CMF / uninstall DM
> - Install DAM / uninstall DAM + install CMF
> - ...
> XXX more here
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
_______________________________________________
ECM-tickets mailing list
[email protected]
http://lists.nuxeo.com/mailman/listinfo/ecm-tickets
