Re: [HELP] Build times for Infra
>Great work!! Thank you, good catch on your PR comment, I just updated it and added the missing license headers. >So, the idea of building in AWS is finally coming to fruition? Yes, that's right. The Infra team has done an awesome job moving the initiative forward. El jue, 25 may 2023 a las 3:10, Richard Zowalla () escribió: > Great work!! > > > > Am Mittwoch, dem 24.05.2023 um 16:58 -0700 schrieb David Blevins: > > That’s amazing. So the idea of building in AWS is finally coming to > > fruition? > > > > > > -David > > > > > On May 24, 2023, at 4:13 PM, Cesar Hernandez > > > wrote: > > > > > > Good news, I opened a PR in > > > https://issues.apache.org/jira/browse/TOMEE-4196 > > > to include Jenkins pipelines for our main branch. > > > Notice that the type of machine now used allows us now to have a > > > full build > > > of 2 hours instead of the ~4.5 we currently have. > > > Once approved the PR, I can swap the existing maven jobs with the > > > pipelines > > > ones for main branch and then we can rinse an repeat for the rest > > > of > > > branches. > > > > > > > > > > > > > > > El lun, 24 oct 2022 a las 6:03, Richard Zowalla () > > > escribió: > > > > > > > Awesome news! Looking forward to see faster turn around times. > > > > > > > > Gruß > > > > Richard > > > > > > > > Am Donnerstag, dem 20.10.2022 um 18:57 -0700 schrieb David > > > > Blevins: > > > > > A few of us got together with some fine folks in Apache Infra > > > > > and > > > > > discussed getting builds going in AWS using an instance type > > > > > that > > > > > would give us the under 2-hour build time. Turns out this one > > > > > was > > > > > also on Infra's wishlist and just needed someone to volunteer > > > > > to step > > > > > up and own the task. Cesar volunteered to lead an effort. :) > > > > > > > > > > Big thank you to Cesar!! If anyone wants to help, join this > > > > > list and > > > > > keep an eye out for related emails. > > > > > > > > > > - https://lists.apache.org/list.html?bui...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Atentamente: > > > César Hernández. > > > > -- Atentamente: César Hernández.
Re: [HELP] Build times for Infra
Great work!! Am Mittwoch, dem 24.05.2023 um 16:58 -0700 schrieb David Blevins: > That’s amazing. So the idea of building in AWS is finally coming to > fruition? > > > -David > > > On May 24, 2023, at 4:13 PM, Cesar Hernandez > > wrote: > > > > Good news, I opened a PR in > > https://issues.apache.org/jira/browse/TOMEE-4196 > > to include Jenkins pipelines for our main branch. > > Notice that the type of machine now used allows us now to have a > > full build > > of 2 hours instead of the ~4.5 we currently have. > > Once approved the PR, I can swap the existing maven jobs with the > > pipelines > > ones for main branch and then we can rinse an repeat for the rest > > of > > branches. > > > > > > > > > > El lun, 24 oct 2022 a las 6:03, Richard Zowalla () > > escribió: > > > > > Awesome news! Looking forward to see faster turn around times. > > > > > > Gruß > > > Richard > > > > > > Am Donnerstag, dem 20.10.2022 um 18:57 -0700 schrieb David > > > Blevins: > > > > A few of us got together with some fine folks in Apache Infra > > > > and > > > > discussed getting builds going in AWS using an instance type > > > > that > > > > would give us the under 2-hour build time. Turns out this one > > > > was > > > > also on Infra's wishlist and just needed someone to volunteer > > > > to step > > > > up and own the task. Cesar volunteered to lead an effort. :) > > > > > > > > Big thank you to Cesar!! If anyone wants to help, join this > > > > list and > > > > keep an eye out for related emails. > > > > > > > > - https://lists.apache.org/list.html?bui...@apache.org > > > > > > > > > > > > > > > > > > -- > > Atentamente: > > César Hernández. >
Re: [HELP] Build times for Infra
That’s amazing. So the idea of building in AWS is finally coming to fruition? -David > On May 24, 2023, at 4:13 PM, Cesar Hernandez wrote: > > Good news, I opened a PR in https://issues.apache.org/jira/browse/TOMEE-4196 > to include Jenkins pipelines for our main branch. > Notice that the type of machine now used allows us now to have a full build > of 2 hours instead of the ~4.5 we currently have. > Once approved the PR, I can swap the existing maven jobs with the pipelines > ones for main branch and then we can rinse an repeat for the rest of > branches. > > > > > El lun, 24 oct 2022 a las 6:03, Richard Zowalla () > escribió: > >> Awesome news! Looking forward to see faster turn around times. >> >> Gruß >> Richard >> >> Am Donnerstag, dem 20.10.2022 um 18:57 -0700 schrieb David Blevins: >>> A few of us got together with some fine folks in Apache Infra and >>> discussed getting builds going in AWS using an instance type that >>> would give us the under 2-hour build time. Turns out this one was >>> also on Infra's wishlist and just needed someone to volunteer to step >>> up and own the task. Cesar volunteered to lead an effort. :) >>> >>> Big thank you to Cesar!! If anyone wants to help, join this list and >>> keep an eye out for related emails. >>> >>> - https://lists.apache.org/list.html?bui...@apache.org >>> >>> >> >> > > -- > Atentamente: > César Hernández.
Re: [HELP] Build times for Infra
Good news, I opened a PR in https://issues.apache.org/jira/browse/TOMEE-4196 to include Jenkins pipelines for our main branch. Notice that the type of machine now used allows us now to have a full build of 2 hours instead of the ~4.5 we currently have. Once approved the PR, I can swap the existing maven jobs with the pipelines ones for main branch and then we can rinse an repeat for the rest of branches. El lun, 24 oct 2022 a las 6:03, Richard Zowalla () escribió: > Awesome news! Looking forward to see faster turn around times. > > Gruß > Richard > > Am Donnerstag, dem 20.10.2022 um 18:57 -0700 schrieb David Blevins: > > A few of us got together with some fine folks in Apache Infra and > > discussed getting builds going in AWS using an instance type that > > would give us the under 2-hour build time. Turns out this one was > > also on Infra's wishlist and just needed someone to volunteer to step > > up and own the task. Cesar volunteered to lead an effort. :) > > > > Big thank you to Cesar!! If anyone wants to help, join this list and > > keep an eye out for related emails. > > > > - https://lists.apache.org/list.html?bui...@apache.org > > > > > > -- Atentamente: César Hernández.
Re: [HELP] Build times for Infra
Awesome news! Looking forward to see faster turn around times. Gruß Richard Am Donnerstag, dem 20.10.2022 um 18:57 -0700 schrieb David Blevins: > A few of us got together with some fine folks in Apache Infra and > discussed getting builds going in AWS using an instance type that > would give us the under 2-hour build time. Turns out this one was > also on Infra's wishlist and just needed someone to volunteer to step > up and own the task. Cesar volunteered to lead an effort. :) > > Big thank you to Cesar!! If anyone wants to help, join this list and > keep an eye out for related emails. > > - https://lists.apache.org/list.html?bui...@apache.org > >
Re: [HELP] Build times for Infra
A few of us got together with some fine folks in Apache Infra and discussed getting builds going in AWS using an instance type that would give us the under 2-hour build time. Turns out this one was also on Infra's wishlist and just needed someone to volunteer to step up and own the task. Cesar volunteered to lead an effort. :) Big thank you to Cesar!! If anyone wants to help, join this list and keep an eye out for related emails. - https://lists.apache.org/list.html?bui...@apache.org -- David Blevins http://twitter.com/dblevins http://www.tomitribe.com > On Oct 16, 2022, at 11:50 AM, David Blevins wrote: > > Here's an updated set of data with some new EC2 instance types: > > c1.xlarge-gp2 : [INFO] Total time: 03:59 h > c1.xlarge-gp3 : [INFO] Total time: 04:00 h > c6i.2xlarge : [INFO] Total time: 01:40 h > c6i.8xlarge : [INFO] Total time: 01:34 h > c7g.2xlarge : [INFO] Total time: 02:02 h > c7g.4xlarge : [INFO] Total time: 01:55 h > c7g.8xlarge : [INFO] Total time: 01:51 h > t2.medium : [INFO] Total time: 05:05 h > t2.xlarge : [INFO] Total time: 02:30 h > t3.medium : [INFO] Total time: 03:21 h > > These all ran with MAVEN_OPTS="-Xmx4096m". I did a compare of the gp2 and > gp3 using the c1.xlarge, they look pretty identical. All of the c6i and c7g > instances also ran with gp3 storage. > > FYI the c6i types are "3rd Generation Intel Xeon Scalable processors with an > all-core turbo frequency of 3.5 GHz" > The c7g types are "AWS Graviton3" processors > > Looks like for our build the c6i.2xlarge is just fine. It's $0.34 per hour. > With 2 builds a day and 2 hours a build, that's $248.2 US/year. Let's assume > we build every PR and call it $1k or $2k / year, it's still reasonable. > > I'll chat with infra to see what kind of options we have for getting or > donating what is needed. > > > -David > > >> On Oct 13, 2022, at 4:13 PM, David Blevins wrote: >> >> Alright, here are some build times for the runs I've done. Results are >> quite varied. >> >> c1.xlarge : [INFO] Total time: 03:59 h >> t2.medium : [INFO] Total time: 05:05 h >> t2.xlarge : [INFO] Total time: 02:30 h >> t3.medium : [INFO] Total time: 03:21 h >> my.laptop : [INFO] Total time: 01:56 h >> >> Specs for the AWS instances are public. Here are the stats for my laptop: >> >> MacBook Pro (16-inch, 2019) >> 2.4 GHz 8-Core Intel Core i9 >> Memory 64 GB 2667 MHz DDR4 >> >> I did try some other AWS instance types, but anything with 2GB or less >> failed due to out of memory issues. I'd be interested to try some other >> types, particularly the AMD-based types. >> >> Anyway long story short: yes, our build could be faster with better >> hardware. Just not very clear what that hardware should be. >> >> >> -David >> >> >>> On Oct 11, 2022, at 3:05 PM, David Blevins wrote: >>> >>> All, >>> >>> I'm collecting some stats on how long it takes to run our full build >>> exactly as Jenkins does. The goal is to work with them to see if we can >>> get some better hardware -- I assume that will require donations, etc. >>> >>> If you'd like to help in collecting data, here's the script I'm running: >>> >>> - curl >>> https://gist.githubusercontent.com/dblevins/b39cc3300bcdd89b426ca33b87b5452b/raw/7c68d4df71e9246c8bf2d0a741f8b145ca5d0820/buildtime.sh >>> | bash >>> >>> Send the time reported in the build.log along with your system information >>> (os, number of cores, if you disk is an SSD, etc) >>> >>> >>> -- >>> David Blevins >>> http://twitter.com/dblevins >>> http://www.tomitribe.com >>> >> > smime.p7s Description: S/MIME cryptographic signature
Re: [HELP] Build times for Infra
Here's an updated set of data with some new EC2 instance types: c1.xlarge-gp2 : [INFO] Total time: 03:59 h c1.xlarge-gp3 : [INFO] Total time: 04:00 h c6i.2xlarge : [INFO] Total time: 01:40 h c6i.8xlarge : [INFO] Total time: 01:34 h c7g.2xlarge : [INFO] Total time: 02:02 h c7g.4xlarge : [INFO] Total time: 01:55 h c7g.8xlarge : [INFO] Total time: 01:51 h t2.medium : [INFO] Total time: 05:05 h t2.xlarge : [INFO] Total time: 02:30 h t3.medium : [INFO] Total time: 03:21 h These all ran with MAVEN_OPTS="-Xmx4096m". I did a compare of the gp2 and gp3 using the c1.xlarge, they look pretty identical. All of the c6i and c7g instances also ran with gp3 storage. FYI the c6i types are "3rd Generation Intel Xeon Scalable processors with an all-core turbo frequency of 3.5 GHz" The c7g types are "AWS Graviton3" processors Looks like for our build the c6i.2xlarge is just fine. It's $0.34 per hour. With 2 builds a day and 2 hours a build, that's $248.2 US/year. Let's assume we build every PR and call it $1k or $2k / year, it's still reasonable. I'll chat with infra to see what kind of options we have for getting or donating what is needed. -David > On Oct 13, 2022, at 4:13 PM, David Blevins wrote: > > Alright, here are some build times for the runs I've done. Results are quite > varied. > > c1.xlarge : [INFO] Total time: 03:59 h > t2.medium : [INFO] Total time: 05:05 h > t2.xlarge : [INFO] Total time: 02:30 h > t3.medium : [INFO] Total time: 03:21 h > my.laptop : [INFO] Total time: 01:56 h > > Specs for the AWS instances are public. Here are the stats for my laptop: > > MacBook Pro (16-inch, 2019) > 2.4 GHz 8-Core Intel Core i9 > Memory 64 GB 2667 MHz DDR4 > > I did try some other AWS instance types, but anything with 2GB or less failed > due to out of memory issues. I'd be interested to try some other types, > particularly the AMD-based types. > > Anyway long story short: yes, our build could be faster with better hardware. > Just not very clear what that hardware should be. > > > -David > > >> On Oct 11, 2022, at 3:05 PM, David Blevins wrote: >> >> All, >> >> I'm collecting some stats on how long it takes to run our full build exactly >> as Jenkins does. The goal is to work with them to see if we can get some >> better hardware -- I assume that will require donations, etc. >> >> If you'd like to help in collecting data, here's the script I'm running: >> >> - curl >> https://gist.githubusercontent.com/dblevins/b39cc3300bcdd89b426ca33b87b5452b/raw/7c68d4df71e9246c8bf2d0a741f8b145ca5d0820/buildtime.sh >> | bash >> >> Send the time reported in the build.log along with your system information >> (os, number of cores, if you disk is an SSD, etc) >> >> >> -- >> David Blevins >> http://twitter.com/dblevins >> http://www.tomitribe.com >> > smime.p7s Description: S/MIME cryptographic signature
Re: [HELP] Build times for Infra
Build https://builds.apache.org/job/TomEE/job/master-build-full/776/ finished after 4 hr 25 min, still not much improvement from the average build time with the MAVEN_OPTS-Xmx1536m https://builds.apache.org/job/TomEE/job/master-build-full/buildTimeTrend I reverted MAVEN OPTS to -Xmx1536m and removed the maven property help:system in the job https://ci-builds.apache.org/job/TomEE/job/master-build-full/ El vie, 14 oct 2022 a las 12:29, Cesar Hernandez () escribió: > Anyway long story short: yes, our build could be faster with better >> hardware. Just not very clear what that hardware should be. > > > We don't know what is running under the hood in > https://ci-builds.apache.org/ for on-demand workers, but I do see other > projects that have specific controller types: > https://jenkins-ccos.apache.org/ > From the numbers so far, it seems that anything bigger than AWS t2.xlarge > (4 vCpus, 16 GB ram) can speed by half the current timing executor in the > apache infra is taking > https://builds.apache.org/job/TomEE/job/master-build-full/buildTimeTrend > > Based on Jonathan's feedback, I reviewed, and the current > https://builds.apache.org/job/TomEE/job/master-build-full is using in > MAVEN_OPTS = -Xmx1536m > I wonder if the result shared in this thread have specific MAVEN_OPTS > values, mine are = -Xms2g -Xmx8g > > I updated https://builds.apache.org/job/TomEE/job/builder-info/ to show > the worker memory, and it seems we have 18 GB free out of 94 GB. [1], but > this is distributed amount of other projects running in that worker, so > there is no guarantee we will always have X or Y GB of RAM available. > I don't have access to check the runner's limits peer node in the entire > builds.apache.org. and > https://cwiki.apache.org/confluence/display/INFRA/Jenkins+node+labels > documentation seems to be outdated. > > > Today's build already finished > https://builds.apache.org/job/TomEE/job/master-build-full/774/ and took ~ > 4 hour. > I updated https://builds.apache.org/job/TomEE/job/master-build-full/ with > the following changes: > > a) Changed MAVEN_OPTS-Xmx1536m to MAVEN_OPTS=-Xmx4096m > b) Changed -U --show-version --fail-at-end clean install -Pall-adapters > -Dfile.encoding=UTF-8 to -U help:system --show-version --fail-at-end clean > install -Pall-adapters -Dfile.encoding=UTF-8 > The only change is I added help:system maven argument to maven to > double-check the OTPS are read. > c) Build https://builds.apache.org/job/TomEE/job/master-build-full/776/ > is currently running with a) and b) applied, I don't expect to see major > improvements even if we are now using 4GB and the load on the shared server > is unpredictable. > > After building > https://builds.apache.org/job/TomEE/job/master-build-full/776/ I plan to > revert to previous job configurations if need it. > > > [1] > > free -g > totalusedfree shared buff/cache > available > Mem: 94 10 18 0 64 > 82 > > > > > > > El vie, 14 oct 2022 a las 8:18, Jonathan Gallimore (< > jonathan.gallim...@gmail.com>) escribió: > >> Taking the commands very literally, the timings for 2 machines here are >> below. I'm curious though - are there any JAVA_OPTS or MAVEN_OPTS we >> should >> use here (e.g. max heap)? The slower machine here seems to be running at >> an >> absolute crawl 2/3rds of the way through the build here with the default >> settings. The result below is after a reboot and MAVEN_OPTS="-Xmx4g". >> >> Laptop (WSL2/Ubuntu on Windows 11, i7-12700H, 32GB Ram): >> >> [INFO] >> >> [INFO] BUILD FAILURE >> [INFO] >> >> [INFO] Total time: 01:36 h >> [INFO] Finished at: 2022-10-14T11:10:08+01:00 >> [INFO] >> >> >> Dell Optiplex 7040 "Server" machine (Ubuntu 20.04.5 LTS, i7-6700T, 16GB >> Ram): >> >> [INFO] BUILD FAILURE >> [INFO] >> >> [INFO] Total time: 01:57 h >> [INFO] Finished at: 2022-10-14T15:16:58+01:00 >> [INFO] >> >> >> On Fri, Oct 14, 2022 at 12:13 AM David Blevins >> wrote: >> >> > Alright, here are some build times for the runs I've done. Results are >> > quite varied. >> > >> > c1.xlarge : [INFO] Total time: 03:59 h >> > t2.medium : [INFO] Total time: 05:05 h >> > t2.xlarge : [INFO] Total time: 02:30 h >> > t3.medium : [INFO] Total time: 03:21 h >> > my.laptop : [INFO] Total time: 01:56 h >> > >> > Specs for the AWS instances are public. Here are the stats for my >> laptop: >> > >> > MacBook Pro (16-inch, 2019) >> > 2.4 GHz 8-Core Intel Core i9 >> > Memory 64 GB 2667 MHz DDR4 >> > >> > I did try some other AWS instance types, but anything with 2GB or less >> > failed due to
Re: [HELP] Build times for Infra
> > Anyway long story short: yes, our build could be faster with better > hardware. Just not very clear what that hardware should be. We don't know what is running under the hood in https://ci-builds.apache.org/ for on-demand workers, but I do see other projects that have specific controller types: https://jenkins-ccos.apache.org/ >From the numbers so far, it seems that anything bigger than AWS t2.xlarge (4 vCpus, 16 GB ram) can speed by half the current timing executor in the apache infra is taking https://builds.apache.org/job/TomEE/job/master-build-full/buildTimeTrend Based on Jonathan's feedback, I reviewed, and the current https://builds.apache.org/job/TomEE/job/master-build-full is using in MAVEN_OPTS = -Xmx1536m I wonder if the result shared in this thread have specific MAVEN_OPTS values, mine are = -Xms2g -Xmx8g I updated https://builds.apache.org/job/TomEE/job/builder-info/ to show the worker memory, and it seems we have 18 GB free out of 94 GB. [1], but this is distributed amount of other projects running in that worker, so there is no guarantee we will always have X or Y GB of RAM available. I don't have access to check the runner's limits peer node in the entire builds.apache.org. and https://cwiki.apache.org/confluence/display/INFRA/Jenkins+node+labels documentation seems to be outdated. Today's build already finished https://builds.apache.org/job/TomEE/job/master-build-full/774/ and took ~ 4 hour. I updated https://builds.apache.org/job/TomEE/job/master-build-full/ with the following changes: a) Changed MAVEN_OPTS-Xmx1536m to MAVEN_OPTS=-Xmx4096m b) Changed -U --show-version --fail-at-end clean install -Pall-adapters -Dfile.encoding=UTF-8 to -U help:system --show-version --fail-at-end clean install -Pall-adapters -Dfile.encoding=UTF-8 The only change is I added help:system maven argument to maven to double-check the OTPS are read. c) Build https://builds.apache.org/job/TomEE/job/master-build-full/776/ is currently running with a) and b) applied, I don't expect to see major improvements even if we are now using 4GB and the load on the shared server is unpredictable. After building https://builds.apache.org/job/TomEE/job/master-build-full/776/ I plan to revert to previous job configurations if need it. [1] free -g totalusedfree shared buff/cache available Mem: 94 10 18 0 64 82 El vie, 14 oct 2022 a las 8:18, Jonathan Gallimore (< jonathan.gallim...@gmail.com>) escribió: > Taking the commands very literally, the timings for 2 machines here are > below. I'm curious though - are there any JAVA_OPTS or MAVEN_OPTS we should > use here (e.g. max heap)? The slower machine here seems to be running at an > absolute crawl 2/3rds of the way through the build here with the default > settings. The result below is after a reboot and MAVEN_OPTS="-Xmx4g". > > Laptop (WSL2/Ubuntu on Windows 11, i7-12700H, 32GB Ram): > > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:36 h > [INFO] Finished at: 2022-10-14T11:10:08+01:00 > [INFO] > > > Dell Optiplex 7040 "Server" machine (Ubuntu 20.04.5 LTS, i7-6700T, 16GB > Ram): > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:57 h > [INFO] Finished at: 2022-10-14T15:16:58+01:00 > [INFO] > > > On Fri, Oct 14, 2022 at 12:13 AM David Blevins > wrote: > > > Alright, here are some build times for the runs I've done. Results are > > quite varied. > > > > c1.xlarge : [INFO] Total time: 03:59 h > > t2.medium : [INFO] Total time: 05:05 h > > t2.xlarge : [INFO] Total time: 02:30 h > > t3.medium : [INFO] Total time: 03:21 h > > my.laptop : [INFO] Total time: 01:56 h > > > > Specs for the AWS instances are public. Here are the stats for my > laptop: > > > > MacBook Pro (16-inch, 2019) > > 2.4 GHz 8-Core Intel Core i9 > > Memory 64 GB 2667 MHz DDR4 > > > > I did try some other AWS instance types, but anything with 2GB or less > > failed due to out of memory issues. I'd be interested to try some other > > types, particularly the AMD-based types. > > > > Anyway long story short: yes, our build could be faster with better > > hardware. Just not very clear what that hardware should be. > > > > > > -David > > > > > > > On Oct 11, 2022, at 3:05 PM, David Blevins > > wrote: > > > > > > All, > > > > > > I'm collecting some stats on how long it takes to run our full build > > exactly as Jenkins does. The goal is to work with them to see if we can > > get some better hardware -- I assume that will require donations, etc. > > > > > > If you'd like to help in collecting data,
Re: [HELP] Build times for Infra
Taking the commands very literally, the timings for 2 machines here are below. I'm curious though - are there any JAVA_OPTS or MAVEN_OPTS we should use here (e.g. max heap)? The slower machine here seems to be running at an absolute crawl 2/3rds of the way through the build here with the default settings. The result below is after a reboot and MAVEN_OPTS="-Xmx4g". Laptop (WSL2/Ubuntu on Windows 11, i7-12700H, 32GB Ram): [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:36 h [INFO] Finished at: 2022-10-14T11:10:08+01:00 [INFO] Dell Optiplex 7040 "Server" machine (Ubuntu 20.04.5 LTS, i7-6700T, 16GB Ram): [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:57 h [INFO] Finished at: 2022-10-14T15:16:58+01:00 [INFO] On Fri, Oct 14, 2022 at 12:13 AM David Blevins wrote: > Alright, here are some build times for the runs I've done. Results are > quite varied. > > c1.xlarge : [INFO] Total time: 03:59 h > t2.medium : [INFO] Total time: 05:05 h > t2.xlarge : [INFO] Total time: 02:30 h > t3.medium : [INFO] Total time: 03:21 h > my.laptop : [INFO] Total time: 01:56 h > > Specs for the AWS instances are public. Here are the stats for my laptop: > > MacBook Pro (16-inch, 2019) > 2.4 GHz 8-Core Intel Core i9 > Memory 64 GB 2667 MHz DDR4 > > I did try some other AWS instance types, but anything with 2GB or less > failed due to out of memory issues. I'd be interested to try some other > types, particularly the AMD-based types. > > Anyway long story short: yes, our build could be faster with better > hardware. Just not very clear what that hardware should be. > > > -David > > > > On Oct 11, 2022, at 3:05 PM, David Blevins > wrote: > > > > All, > > > > I'm collecting some stats on how long it takes to run our full build > exactly as Jenkins does. The goal is to work with them to see if we can > get some better hardware -- I assume that will require donations, etc. > > > > If you'd like to help in collecting data, here's the script I'm running: > > > > - curl > https://gist.githubusercontent.com/dblevins/b39cc3300bcdd89b426ca33b87b5452b/raw/7c68d4df71e9246c8bf2d0a741f8b145ca5d0820/buildtime.sh > | bash > > > > Send the time reported in the build.log along with your system > information (os, number of cores, if you disk is an SSD, etc) > > > > > > -- > > David Blevins > > http://twitter.com/dblevins > > http://www.tomitribe.com > > > >
Re: [HELP] Build times for Infra
Total time: 01:52 h Custom VM: 16 Cores, 16 GB Ram, SSD SAN Storage Running on Intel(R) Xeon(R) Gold 6312U CPU @ 2.40GHz Gruß Richard Am Donnerstag, dem 13.10.2022 um 16:13 -0700 schrieb David Blevins: > Alright, here are some build times for the runs I've done. Results > are quite varied. > > c1.xlarge : [INFO] Total time: 03:59 h > t2.medium : [INFO] Total time: 05:05 h > t2.xlarge : [INFO] Total time: 02:30 h > t3.medium : [INFO] Total time: 03:21 h > my.laptop : [INFO] Total time: 01:56 h > > Specs for the AWS instances are public. Here are the stats for my > laptop: > > MacBook Pro (16-inch, 2019) > 2.4 GHz 8-Core Intel Core i9 > Memory 64 GB 2667 MHz DDR4 > > I did try some other AWS instance types, but anything with 2GB or > less failed due to out of memory issues. I'd be interested to try > some other types, particularly the AMD-based types. > > Anyway long story short: yes, our build could be faster with better > hardware. Just not very clear what that hardware should be. > > > -David > > > > On Oct 11, 2022, at 3:05 PM, David Blevins > > wrote: > > > > All, > > > > I'm collecting some stats on how long it takes to run our full > > build exactly as Jenkins does. The goal is to work with them to > > see if we can get some better hardware -- I assume that will > > require donations, etc. > > > > If you'd like to help in collecting data, here's the script I'm > > running: > > > > - curl > > https://gist.githubusercontent.com/dblevins/b39cc3300bcdd89b426ca33b87b5452b/raw/7c68d4df71e9246c8bf2d0a741f8b145ca5d0820/buildtime.sh > > | bash > > > > Send the time reported in the build.log along with your system > > information (os, number of cores, if you disk is an SSD, etc) > > > > > > -- > > David Blevins > > http://twitter.com/dblevins > > http://www.tomitribe.com > >
Re: [HELP] Build times for Infra
Hello David, Is the full build: * CPU bound * Memory bound * IOPS bound ? Which can be found using top/iostat metrics during its run. If the later, then I guess that a good test would be to run the build on an EBS volume of GP3 type (minimum 3000 IOPS, up to 16000 IOPS) and see if that improves significantly build time ; but of course the more IOPS, the higher price is. Another approach to keep price/performance ratio as low as possible could be to use cheap AWS instances with https://plugins.jenkins.io/ec2-fleet/ to let Jenkins create AWS instances based on load. Disclaimer: I haven't experienced the later approach myself, but I work in a place with a homemade build system, designed to allocate build jobs on a kind of grid of large VMs, with quota management & automatic cleanups to keep costs within planned bounds. Alex Le mer. 12 oct. 2022 à 00:05, David Blevins a écrit : > > All, > > I'm collecting some stats on how long it takes to run our full build exactly > as Jenkins does. The goal is to work with them to see if we can get some > better hardware -- I assume that will require donations, etc. > > If you'd like to help in collecting data, here's the script I'm running: > > - curl > https://gist.githubusercontent.com/dblevins/b39cc3300bcdd89b426ca33b87b5452b/raw/7c68d4df71e9246c8bf2d0a741f8b145ca5d0820/buildtime.sh > | bash > > Send the time reported in the build.log along with your system information > (os, number of cores, if you disk is an SSD, etc) > > > -- > David Blevins > http://twitter.com/dblevins > http://www.tomitribe.com >
Re: [HELP] Build times for Infra
Alright, here are some build times for the runs I've done. Results are quite varied. c1.xlarge : [INFO] Total time: 03:59 h t2.medium : [INFO] Total time: 05:05 h t2.xlarge : [INFO] Total time: 02:30 h t3.medium : [INFO] Total time: 03:21 h my.laptop : [INFO] Total time: 01:56 h Specs for the AWS instances are public. Here are the stats for my laptop: MacBook Pro (16-inch, 2019) 2.4 GHz 8-Core Intel Core i9 Memory 64 GB 2667 MHz DDR4 I did try some other AWS instance types, but anything with 2GB or less failed due to out of memory issues. I'd be interested to try some other types, particularly the AMD-based types. Anyway long story short: yes, our build could be faster with better hardware. Just not very clear what that hardware should be. -David > On Oct 11, 2022, at 3:05 PM, David Blevins wrote: > > All, > > I'm collecting some stats on how long it takes to run our full build exactly > as Jenkins does. The goal is to work with them to see if we can get some > better hardware -- I assume that will require donations, etc. > > If you'd like to help in collecting data, here's the script I'm running: > > - curl > https://gist.githubusercontent.com/dblevins/b39cc3300bcdd89b426ca33b87b5452b/raw/7c68d4df71e9246c8bf2d0a741f8b145ca5d0820/buildtime.sh > | bash > > Send the time reported in the build.log along with your system information > (os, number of cores, if you disk is an SSD, etc) > > > -- > David Blevins > http://twitter.com/dblevins > http://www.tomitribe.com > smime.p7s Description: S/MIME cryptographic signature
Re: [HELP] Build times for Infra
FYI I updated the script yesterday so it uses --fail-never instead of --fail-at-end. It seems that even though our jobs have --fail-at-end, the actual behavior we're getting is --fail-never. With --fail-at-end the build skips most the modules and ends early, hence the 37 minute build time. Posting my results shortly, but with --fail-never I'm seeing build times in the 2hr to 3hr range. -David > On Oct 13, 2022, at 8:18 AM, Cesar Hernandez wrote: > > > Hi, > > Machine and environment details: > OS: mac os x", version: "12.6", arch: "x86_64 > Cores: 3.5 GHz Dual-Core Intel Core i7 > Ram: 16 GB > Disk Type: SSD > Maven home: 3.8.5 > Java version: 11.0.13, vendor: Eclipse Adoptium > Default locale: en_GT, platform encoding: UTF-8 > > Also, this is the time reported in the build.log [1] and the full reactor > summary [2]. > > > [1] > > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 37:46 min > [INFO] Finished at: 2022-10-12T22:04:27-06:00 > [INFO] > > > > [2] > [INFO] Reactor Summary: > [INFO] > [INFO] Apache OpenEJB 9.0.0-M9-SNAPSHOT ... SUCCESS [ 1.626 > s] > [INFO] TomEE :: Deps 9.0.0-M9-SNAPSHOT SUCCESS [ 0.058 > s] > [INFO] TomEE :: Deps :: Taglibs Shade 9.0.0-M9-SNAPSHOT ... SUCCESS [ 43.925 > s] > [INFO] TomEE :: Deps :: CXF Shade 9.0.0-M9-SNAPSHOT ... SUCCESS [01:02 > min] > [INFO] TomEE :: Deps :: CXF Rest Client Shade 9.0.0-M9-SNAPSHOT SUCCESS [ > 1.235 s] > [INFO] TomEE :: Deps :: SXC Shade 9.0.0-M9-SNAPSHOT ... SUCCESS [ 0.332 > s] > [INFO] TomEE :: Deps :: Apache ActiveMQ Client shade 9.0.0-M9-SNAPSHOT > SUCCESS [ 0.907 s] > [INFO] TomEE :: Deps :: Apache ActiveMQ Broker shade 9.0.0-M9-SNAPSHOT > SUCCESS [ 0.813 s] > [INFO] TomEE :: Deps :: Apache ActiveMQ KahaDB Store shade 9.0.0-M9-SNAPSHOT > SUCCESS [ 0.481 s] > [INFO] TomEE :: Deps :: Apache ActiveMQ RA shade 9.0.0-M9-SNAPSHOT SUCCESS [ > 0.332 s] > [INFO] TomEE :: Deps :: Apache Commons DBCP2 shade 9.0.0-M9-SNAPSHOT SUCCESS > [ 0.317 s] > [INFO] TomEE :: Deps :: Apache Commons FileUpload shade 9.0.0-M9-SNAPSHOT > SUCCESS [ 0.212 s] > [INFO] TomEE :: BOMs 9.0.0-M9-SNAPSHOT SUCCESS [ 0.034 > s] > [INFO] TomEE :: BOMs :: JAXB Runtime 9.0.0-M9-SNAPSHOT SUCCESS [ 0.130 > s] > [INFO] TomEE :: Container 9.0.0-M9-SNAPSHOT ... SUCCESS [ 0.029 > s] > [INFO] TomEE :: Container :: Loader 9.0.0-M9-SNAPSHOT . SUCCESS [ 22.797 > s] > [INFO] TomEE :: Container :: MBean Annotation API 9.0.0-M9-SNAPSHOT SUCCESS [ > 0.244 s] > [INFO] TomEE :: Container :: JPA Integration 9.0.0-M9-SNAPSHOT SUCCESS [ > 0.780 s] > [INFO] TomEE :: Container :: API 9.0.0-M9-SNAPSHOT SUCCESS [ 0.357 > s] > [INFO] TomEE :: Container :: Java Agent 9.0.0-M9-SNAPSHOT . SUCCESS [ 0.372 > s] > [INFO] TomEE :: Container :: Java EE 9.0.0-M9-SNAPSHOT SUCCESS [ 20.428 > s] > [INFO] TomEE :: Container :: Java EE Accessors 9.0.0-M9-SNAPSHOT SUCCESS [ > 5.729 s] > [INFO] TomEE :: iTests 9.0.0-M9-SNAPSHOT .. SUCCESS [ 0.021 > s] > [INFO] TomEE :: iTests :: Beans 9.0.0-M9-SNAPSHOT . SUCCESS [ 3.606 > s] > [INFO] TomEE :: iTests :: Interceptor Beans 9.0.0-M9-SNAPSHOT SUCCESS [ > 0.349 s] > [INFO] TomEE :: iTests :: Servlets 9.0.0-M9-SNAPSHOT .. SUCCESS [ 0.375 > s] > [INFO] TomEE :: iTests :: Client 9.0.0-M9-SNAPSHOT SUCCESS [ 3.860 > s] > [INFO] TomEE :: iTests :: App 9.0.0-M9-SNAPSHOT ... SUCCESS [ 0.810 > s] > [INFO] TomEE :: iTests :: Failover EJB 9.0.0-M9-SNAPSHOT .. SUCCESS [ 0.200 > s] > [INFO] TomEE :: Container :: Core 9.0.0-M9-SNAPSHOT ... SUCCESS [20:37 > min] > [INFO] TomEE :: TomEE 9.0.0-M9-SNAPSHOT ... SUCCESS [ 0.417 > s] > [INFO] TomEE :: TomEE :: Bootstrap 9.0.0-M9-SNAPSHOT .. SUCCESS [ 1.427 > s] > [INFO] TomEE :: TomEE :: Config 9.0.0-M9-SNAPSHOT . SUCCESS [ 0.614 > s] > [INFO] TomEE :: Server 9.0.0-M9-SNAPSHOT .. SUCCESS [ 0.020 > s] > [INFO] TomEE :: Server :: Client 9.0.0-M9-SNAPSHOT SUCCESS [ 10.463 > s] > [INFO] TomEE :: TomEE :: Common 9.0.0-M9-SNAPSHOT . SUCCESS [ 1.664 > s] > [INFO] TomEE :: TomEE :: Common 9.0.0-M9-SNAPSHOT . SUCCESS [ 1.664 > s] > [INFO] TomEE :: TomEE :: Juli 9.0.0-M9-SNAPSHOT ... SUCCESS [ 38.528 > s] > [INFO] TomEE :: TomEE :: Loader 9.0.0-M9-SNAPSHOT . SUCCESS [ 1.776 > s] > [INFO] TomEE :: TomEE :: JDBC 9.0.0-M9-SNAPSHOT ... SUCCESS [ 9.989 > s] > [INFO] TomEE :: Server :: Core 9.0.0-M9-SNAPSHOT .. SUCCESS [ 5.085 > s] > [INFO] TomEE :: Server :: Multicast Discovery 9.0.0-M9-SNAPSHOT SUCCESS [ > 39.901 s] > [INFO] TomEE :: Server :: EJBd 9.0.0-M9-SNAPSHOT .. FAILURE
Re: [HELP] Build times for Infra
Hi, So I've been doing some tests. Parallelism does not give us much savings unless tests are awiting a lot (metrics for example) and a couple more modules. Otherwise, the CPU of the Jenkins slave is too overloaded to get any benefit. As David mentioned yesterday on slack, they are 12+ years old servers, so not a surprise. So I went into another direction and created 4 new Jenkins job triggered after the sanity checks such as we will run in 5 different machines - quick build and basics tests - examples - itests - arquillian tests - tck Of course every night, we run everything together in the full build. But the 5 other should give a good view in less than 2 hours. -- Jean-Louis Monteiro http://twitter.com/jlouismonteiro http://www.tomitribe.com On Thu, Oct 13, 2022 at 5:20 PM Cesar Hernandez wrote: > > Hi, > > Machine and environment details: > OS: mac os x", version: "12.6", arch: "x86_64 > Cores: 3.5 GHz Dual-Core Intel Core i7 > Ram: 16 GB > Disk Type: SSD > Maven home: 3.8.5 > Java version: 11.0.13, vendor: Eclipse Adoptium > Default locale: en_GT, platform encoding: UTF-8 > > Also, this is the time reported in the build.log [1] and the full reactor > summary [2]. > > > [1] > > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 37:46 min > [INFO] Finished at: 2022-10-12T22:04:27-06:00 > [INFO] > > > > [2] > [INFO] Reactor Summary: > [INFO] > [INFO] Apache OpenEJB 9.0.0-M9-SNAPSHOT ... SUCCESS [ > 1.626 s] > [INFO] TomEE :: Deps 9.0.0-M9-SNAPSHOT SUCCESS [ > 0.058 s] > [INFO] TomEE :: Deps :: Taglibs Shade 9.0.0-M9-SNAPSHOT ... SUCCESS [ > 43.925 s] > [INFO] TomEE :: Deps :: CXF Shade 9.0.0-M9-SNAPSHOT ... SUCCESS [01:02 > min] > [INFO] TomEE :: Deps :: CXF Rest Client Shade 9.0.0-M9-SNAPSHOT SUCCESS [ > 1.235 s] > [INFO] TomEE :: Deps :: SXC Shade 9.0.0-M9-SNAPSHOT ... SUCCESS [ > 0.332 s] > [INFO] TomEE :: Deps :: Apache ActiveMQ Client shade 9.0.0-M9-SNAPSHOT > SUCCESS [ 0.907 s] > [INFO] TomEE :: Deps :: Apache ActiveMQ Broker shade 9.0.0-M9-SNAPSHOT > SUCCESS [ 0.813 s] > [INFO] TomEE :: Deps :: Apache ActiveMQ KahaDB Store shade > 9.0.0-M9-SNAPSHOT SUCCESS [ 0.481 s] > [INFO] TomEE :: Deps :: Apache ActiveMQ RA shade 9.0.0-M9-SNAPSHOT SUCCESS > [ 0.332 s] > [INFO] TomEE :: Deps :: Apache Commons DBCP2 shade 9.0.0-M9-SNAPSHOT > SUCCESS [ 0.317 s] > [INFO] TomEE :: Deps :: Apache Commons FileUpload shade 9.0.0-M9-SNAPSHOT > SUCCESS [ 0.212 s] > [INFO] TomEE :: BOMs 9.0.0-M9-SNAPSHOT SUCCESS [ > 0.034 s] > [INFO] TomEE :: BOMs :: JAXB Runtime 9.0.0-M9-SNAPSHOT SUCCESS [ > 0.130 s] > [INFO] TomEE :: Container 9.0.0-M9-SNAPSHOT ... SUCCESS [ > 0.029 s] > [INFO] TomEE :: Container :: Loader 9.0.0-M9-SNAPSHOT . SUCCESS [ > 22.797 s] > [INFO] TomEE :: Container :: MBean Annotation API 9.0.0-M9-SNAPSHOT > SUCCESS [ 0.244 s] > [INFO] TomEE :: Container :: JPA Integration 9.0.0-M9-SNAPSHOT SUCCESS [ > 0.780 s] > [INFO] TomEE :: Container :: API 9.0.0-M9-SNAPSHOT SUCCESS [ > 0.357 s] > [INFO] TomEE :: Container :: Java Agent 9.0.0-M9-SNAPSHOT . SUCCESS [ > 0.372 s] > [INFO] TomEE :: Container :: Java EE 9.0.0-M9-SNAPSHOT SUCCESS [ > 20.428 s] > [INFO] TomEE :: Container :: Java EE Accessors 9.0.0-M9-SNAPSHOT SUCCESS [ > 5.729 s] > [INFO] TomEE :: iTests 9.0.0-M9-SNAPSHOT .. SUCCESS [ > 0.021 s] > [INFO] TomEE :: iTests :: Beans 9.0.0-M9-SNAPSHOT . SUCCESS [ > 3.606 s] > [INFO] TomEE :: iTests :: Interceptor Beans 9.0.0-M9-SNAPSHOT SUCCESS [ > 0.349 s] > [INFO] TomEE :: iTests :: Servlets 9.0.0-M9-SNAPSHOT .. SUCCESS [ > 0.375 s] > [INFO] TomEE :: iTests :: Client 9.0.0-M9-SNAPSHOT SUCCESS [ > 3.860 s] > [INFO] TomEE :: iTests :: App 9.0.0-M9-SNAPSHOT ... SUCCESS [ > 0.810 s] > [INFO] TomEE :: iTests :: Failover EJB 9.0.0-M9-SNAPSHOT .. SUCCESS [ > 0.200 s] > [INFO] TomEE :: Container :: Core 9.0.0-M9-SNAPSHOT ... SUCCESS [20:37 > min] > [INFO] TomEE :: TomEE 9.0.0-M9-SNAPSHOT ... SUCCESS [ > 0.417 s] > [INFO] TomEE :: TomEE :: Bootstrap 9.0.0-M9-SNAPSHOT .. SUCCESS [ > 1.427 s] > [INFO] TomEE :: TomEE :: Config 9.0.0-M9-SNAPSHOT . SUCCESS [ > 0.614 s] > [INFO] TomEE :: Server 9.0.0-M9-SNAPSHOT .. SUCCESS [ > 0.020 s] > [INFO] TomEE :: Server :: Client 9.0.0-M9-SNAPSHOT SUCCESS [ > 10.463 s] > [INFO] TomEE :: TomEE :: Common 9.0.0-M9-SNAPSHOT . SUCCESS [ > 1.664 s] > [INFO] TomEE :: TomEE :: Common 9.0.0-M9-SNAPSHOT . SUCCESS [ > 1.664 s] > [INFO] TomEE :: TomEE :: Juli 9.0.0-M9-SNAPSHOT ... SUCCESS [ > 38.528 s] > [INFO] TomEE :: TomEE :: Loader 9.0.0-M9-SNAPSHOT . SUCCESS [ > 1.776 s] > [INFO] TomEE :: TomEE
Re: [HELP] Build times for Infra
Hi, I am starting this test on a virtual machine in our infra now. @all For building main you need to either use a JDK 11 or JDK 17. This information is also relevant as well as the Maven version used to for building. In addition, it might be required to switch the git repo url in the sh script from the ssh-based to the http based version ( https://github.com/apache/tomee.git), so you can run on a dedicated machine in which you do not need to setup private/public key auth. Gruß Richard Am Dienstag, dem 11.10.2022 um 15:05 -0700 schrieb David Blevins: > All, > > I'm collecting some stats on how long it takes to run our full build > exactly as Jenkins does. The goal is to work with them to see if we > can get some better hardware -- I assume that will require donations, > etc. > > If you'd like to help in collecting data, here's the script I'm > running: > > - curl > https://gist.githubusercontent.com/dblevins/b39cc3300bcdd89b426ca33b87b5452b/raw/7c68d4df71e9246c8bf2d0a741f8b145ca5d0820/buildtime.sh > | bash > > Send the time reported in the build.log along with your system > information (os, number of cores, if you disk is an SSD, etc) > > smime.p7s Description: S/MIME cryptographic signature
[HELP] Build times for Infra
All, I'm collecting some stats on how long it takes to run our full build exactly as Jenkins does. The goal is to work with them to see if we can get some better hardware -- I assume that will require donations, etc. If you'd like to help in collecting data, here's the script I'm running: - curl https://gist.githubusercontent.com/dblevins/b39cc3300bcdd89b426ca33b87b5452b/raw/7c68d4df71e9246c8bf2d0a741f8b145ca5d0820/buildtime.sh | bash Send the time reported in the build.log along with your system information (os, number of cores, if you disk is an SSD, etc) -- David Blevins http://twitter.com/dblevins http://www.tomitribe.com smime.p7s Description: S/MIME cryptographic signature
