Re: [HELP] Build times for Infra
Build https://builds.apache.org/job/TomEE/job/master-build-full/776/ finished after 4 hr 25 min, still not much improvement from the average build time with the MAVEN_OPTS-Xmx1536m https://builds.apache.org/job/TomEE/job/master-build-full/buildTimeTrend I reverted MAVEN OPTS to -Xmx1536m and removed the maven property help:system in the job https://ci-builds.apache.org/job/TomEE/job/master-build-full/ El vie, 14 oct 2022 a las 12:29, Cesar Hernandez () escribió: > Anyway long story short: yes, our build could be faster with better >> hardware. Just not very clear what that hardware should be. > > > We don't know what is running under the hood in > https://ci-builds.apache.org/ for on-demand workers, but I do see other > projects that have specific controller types: > https://jenkins-ccos.apache.org/ > From the numbers so far, it seems that anything bigger than AWS t2.xlarge > (4 vCpus, 16 GB ram) can speed by half the current timing executor in the > apache infra is taking > https://builds.apache.org/job/TomEE/job/master-build-full/buildTimeTrend > > Based on Jonathan's feedback, I reviewed, and the current > https://builds.apache.org/job/TomEE/job/master-build-full is using in > MAVEN_OPTS = -Xmx1536m > I wonder if the result shared in this thread have specific MAVEN_OPTS > values, mine are = -Xms2g -Xmx8g > > I updated https://builds.apache.org/job/TomEE/job/builder-info/ to show > the worker memory, and it seems we have 18 GB free out of 94 GB. [1], but > this is distributed amount of other projects running in that worker, so > there is no guarantee we will always have X or Y GB of RAM available. > I don't have access to check the runner's limits peer node in the entire > builds.apache.org. and > https://cwiki.apache.org/confluence/display/INFRA/Jenkins+node+labels > documentation seems to be outdated. > > > Today's build already finished > https://builds.apache.org/job/TomEE/job/master-build-full/774/ and took ~ > 4 hour. > I updated https://builds.apache.org/job/TomEE/job/master-build-full/ with > the following changes: > > a) Changed MAVEN_OPTS-Xmx1536m to MAVEN_OPTS=-Xmx4096m > b) Changed -U --show-version --fail-at-end clean install -Pall-adapters > -Dfile.encoding=UTF-8 to -U help:system --show-version --fail-at-end clean > install -Pall-adapters -Dfile.encoding=UTF-8 > The only change is I added help:system maven argument to maven to > double-check the OTPS are read. > c) Build https://builds.apache.org/job/TomEE/job/master-build-full/776/ > is currently running with a) and b) applied, I don't expect to see major > improvements even if we are now using 4GB and the load on the shared server > is unpredictable. > > After building > https://builds.apache.org/job/TomEE/job/master-build-full/776/ I plan to > revert to previous job configurations if need it. > > > [1] > > free -g > totalusedfree shared buff/cache > available > Mem: 94 10 18 0 64 > 82 > > > > > > > El vie, 14 oct 2022 a las 8:18, Jonathan Gallimore (< > jonathan.gallim...@gmail.com>) escribió: > >> Taking the commands very literally, the timings for 2 machines here are >> below. I'm curious though - are there any JAVA_OPTS or MAVEN_OPTS we >> should >> use here (e.g. max heap)? The slower machine here seems to be running at >> an >> absolute crawl 2/3rds of the way through the build here with the default >> settings. The result below is after a reboot and MAVEN_OPTS="-Xmx4g". >> >> Laptop (WSL2/Ubuntu on Windows 11, i7-12700H, 32GB Ram): >> >> [INFO] >> >> [INFO] BUILD FAILURE >> [INFO] >> >> [INFO] Total time: 01:36 h >> [INFO] Finished at: 2022-10-14T11:10:08+01:00 >> [INFO] >> >> >> Dell Optiplex 7040 "Server" machine (Ubuntu 20.04.5 LTS, i7-6700T, 16GB >> Ram): >> >> [INFO] BUILD FAILURE >> [INFO] >> >> [INFO] Total time: 01:57 h >> [INFO] Finished at: 2022-10-14T15:16:58+01:00 >> [INFO] >> >> >> On Fri, Oct 14, 2022 at 12:13 AM David Blevins >> wrote: >> >> > Alright, here are some build times for the runs I've done. Results are >> > quite varied. >> > >> > c1.xlarge : [INFO] Total time: 03:59 h >> > t2.medium : [INFO] Total time: 05:05 h >> > t2.xlarge : [INFO] Total time: 02:30 h >> > t3.medium : [INFO] Total time: 03:21 h >> > my.laptop : [INFO] Total time: 01:56 h >> > >> > Specs for the AWS instances are public. Here are the stats for my >> laptop: >> > >> > MacBook Pro (16-inch, 2019) >> > 2.4 GHz 8-Core Intel Core i9 >> > Memory 64 GB 2667 MHz DDR4 >> > >> > I did try some other AWS instance types, but anything with 2GB or less >> > failed due to out
Re: [HELP] Build times for Infra
> > Anyway long story short: yes, our build could be faster with better > hardware. Just not very clear what that hardware should be. We don't know what is running under the hood in https://ci-builds.apache.org/ for on-demand workers, but I do see other projects that have specific controller types: https://jenkins-ccos.apache.org/ >From the numbers so far, it seems that anything bigger than AWS t2.xlarge (4 vCpus, 16 GB ram) can speed by half the current timing executor in the apache infra is taking https://builds.apache.org/job/TomEE/job/master-build-full/buildTimeTrend Based on Jonathan's feedback, I reviewed, and the current https://builds.apache.org/job/TomEE/job/master-build-full is using in MAVEN_OPTS = -Xmx1536m I wonder if the result shared in this thread have specific MAVEN_OPTS values, mine are = -Xms2g -Xmx8g I updated https://builds.apache.org/job/TomEE/job/builder-info/ to show the worker memory, and it seems we have 18 GB free out of 94 GB. [1], but this is distributed amount of other projects running in that worker, so there is no guarantee we will always have X or Y GB of RAM available. I don't have access to check the runner's limits peer node in the entire builds.apache.org. and https://cwiki.apache.org/confluence/display/INFRA/Jenkins+node+labels documentation seems to be outdated. Today's build already finished https://builds.apache.org/job/TomEE/job/master-build-full/774/ and took ~ 4 hour. I updated https://builds.apache.org/job/TomEE/job/master-build-full/ with the following changes: a) Changed MAVEN_OPTS-Xmx1536m to MAVEN_OPTS=-Xmx4096m b) Changed -U --show-version --fail-at-end clean install -Pall-adapters -Dfile.encoding=UTF-8 to -U help:system --show-version --fail-at-end clean install -Pall-adapters -Dfile.encoding=UTF-8 The only change is I added help:system maven argument to maven to double-check the OTPS are read. c) Build https://builds.apache.org/job/TomEE/job/master-build-full/776/ is currently running with a) and b) applied, I don't expect to see major improvements even if we are now using 4GB and the load on the shared server is unpredictable. After building https://builds.apache.org/job/TomEE/job/master-build-full/776/ I plan to revert to previous job configurations if need it. [1] free -g totalusedfree shared buff/cache available Mem: 94 10 18 0 64 82 El vie, 14 oct 2022 a las 8:18, Jonathan Gallimore (< jonathan.gallim...@gmail.com>) escribió: > Taking the commands very literally, the timings for 2 machines here are > below. I'm curious though - are there any JAVA_OPTS or MAVEN_OPTS we should > use here (e.g. max heap)? The slower machine here seems to be running at an > absolute crawl 2/3rds of the way through the build here with the default > settings. The result below is after a reboot and MAVEN_OPTS="-Xmx4g". > > Laptop (WSL2/Ubuntu on Windows 11, i7-12700H, 32GB Ram): > > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:36 h > [INFO] Finished at: 2022-10-14T11:10:08+01:00 > [INFO] > > > Dell Optiplex 7040 "Server" machine (Ubuntu 20.04.5 LTS, i7-6700T, 16GB > Ram): > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:57 h > [INFO] Finished at: 2022-10-14T15:16:58+01:00 > [INFO] > > > On Fri, Oct 14, 2022 at 12:13 AM David Blevins > wrote: > > > Alright, here are some build times for the runs I've done. Results are > > quite varied. > > > > c1.xlarge : [INFO] Total time: 03:59 h > > t2.medium : [INFO] Total time: 05:05 h > > t2.xlarge : [INFO] Total time: 02:30 h > > t3.medium : [INFO] Total time: 03:21 h > > my.laptop : [INFO] Total time: 01:56 h > > > > Specs for the AWS instances are public. Here are the stats for my > laptop: > > > > MacBook Pro (16-inch, 2019) > > 2.4 GHz 8-Core Intel Core i9 > > Memory 64 GB 2667 MHz DDR4 > > > > I did try some other AWS instance types, but anything with 2GB or less > > failed due to out of memory issues. I'd be interested to try some other > > types, particularly the AMD-based types. > > > > Anyway long story short: yes, our build could be faster with better > > hardware. Just not very clear what that hardware should be. > > > > > > -David > > > > > > > On Oct 11, 2022, at 3:05 PM, David Blevins > > wrote: > > > > > > All, > > > > > > I'm collecting some stats on how long it takes to run our full build > > exactly as Jenkins does. The goal is to work with them to see if we can > > get some better hardware -- I assume that will require donations, etc. > > > > > > If you'd like to help in collecting data, here
Re: [HELP] Build times for Infra
Taking the commands very literally, the timings for 2 machines here are below. I'm curious though - are there any JAVA_OPTS or MAVEN_OPTS we should use here (e.g. max heap)? The slower machine here seems to be running at an absolute crawl 2/3rds of the way through the build here with the default settings. The result below is after a reboot and MAVEN_OPTS="-Xmx4g". Laptop (WSL2/Ubuntu on Windows 11, i7-12700H, 32GB Ram): [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:36 h [INFO] Finished at: 2022-10-14T11:10:08+01:00 [INFO] Dell Optiplex 7040 "Server" machine (Ubuntu 20.04.5 LTS, i7-6700T, 16GB Ram): [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:57 h [INFO] Finished at: 2022-10-14T15:16:58+01:00 [INFO] On Fri, Oct 14, 2022 at 12:13 AM David Blevins wrote: > Alright, here are some build times for the runs I've done. Results are > quite varied. > > c1.xlarge : [INFO] Total time: 03:59 h > t2.medium : [INFO] Total time: 05:05 h > t2.xlarge : [INFO] Total time: 02:30 h > t3.medium : [INFO] Total time: 03:21 h > my.laptop : [INFO] Total time: 01:56 h > > Specs for the AWS instances are public. Here are the stats for my laptop: > > MacBook Pro (16-inch, 2019) > 2.4 GHz 8-Core Intel Core i9 > Memory 64 GB 2667 MHz DDR4 > > I did try some other AWS instance types, but anything with 2GB or less > failed due to out of memory issues. I'd be interested to try some other > types, particularly the AMD-based types. > > Anyway long story short: yes, our build could be faster with better > hardware. Just not very clear what that hardware should be. > > > -David > > > > On Oct 11, 2022, at 3:05 PM, David Blevins > wrote: > > > > All, > > > > I'm collecting some stats on how long it takes to run our full build > exactly as Jenkins does. The goal is to work with them to see if we can > get some better hardware -- I assume that will require donations, etc. > > > > If you'd like to help in collecting data, here's the script I'm running: > > > > - curl > https://gist.githubusercontent.com/dblevins/b39cc3300bcdd89b426ca33b87b5452b/raw/7c68d4df71e9246c8bf2d0a741f8b145ca5d0820/buildtime.sh > | bash > > > > Send the time reported in the build.log along with your system > information (os, number of cores, if you disk is an SSD, etc) > > > > > > -- > > David Blevins > > http://twitter.com/dblevins > > http://www.tomitribe.com > > > >
Re: [HELP] Build times for Infra
Total time: 01:52 h Custom VM: 16 Cores, 16 GB Ram, SSD SAN Storage Running on Intel(R) Xeon(R) Gold 6312U CPU @ 2.40GHz Gruß Richard Am Donnerstag, dem 13.10.2022 um 16:13 -0700 schrieb David Blevins: > Alright, here are some build times for the runs I've done. Results > are quite varied. > > c1.xlarge : [INFO] Total time: 03:59 h > t2.medium : [INFO] Total time: 05:05 h > t2.xlarge : [INFO] Total time: 02:30 h > t3.medium : [INFO] Total time: 03:21 h > my.laptop : [INFO] Total time: 01:56 h > > Specs for the AWS instances are public. Here are the stats for my > laptop: > > MacBook Pro (16-inch, 2019) > 2.4 GHz 8-Core Intel Core i9 > Memory 64 GB 2667 MHz DDR4 > > I did try some other AWS instance types, but anything with 2GB or > less failed due to out of memory issues. I'd be interested to try > some other types, particularly the AMD-based types. > > Anyway long story short: yes, our build could be faster with better > hardware. Just not very clear what that hardware should be. > > > -David > > > > On Oct 11, 2022, at 3:05 PM, David Blevins > > wrote: > > > > All, > > > > I'm collecting some stats on how long it takes to run our full > > build exactly as Jenkins does. The goal is to work with them to > > see if we can get some better hardware -- I assume that will > > require donations, etc. > > > > If you'd like to help in collecting data, here's the script I'm > > running: > > > > - curl > > https://gist.githubusercontent.com/dblevins/b39cc3300bcdd89b426ca33b87b5452b/raw/7c68d4df71e9246c8bf2d0a741f8b145ca5d0820/buildtime.sh > > | bash > > > > Send the time reported in the build.log along with your system > > information (os, number of cores, if you disk is an SSD, etc) > > > > > > -- > > David Blevins > > http://twitter.com/dblevins > > http://www.tomitribe.com > >
