I was looking at the slides for Peter Palaga, for one of his talks about Maven/Maven Daemon and it gave me some ideas about trying further improvements to the build.
I logged them on the following ticket: https://issues.apache.org/jira/browse/CAMEL-17894 I am not sure when/if I'll have time to look further into this, but I'm sharing the ticket in case the community has additional ideas/data to share. Kind regards On Wed, Feb 16, 2022 at 6:33 PM Otavio Rodolfo Piske <angusyo...@gmail.com> wrote: > Great work, thanks for contributing! > > For now I'd keep it as it is, so we can experiment with the parallel > runner on GH. We have a few more things happening on the Jenkins pipeline > and it would be better to keep it as stable as possible for now. > > Kind regards > > On Wed, Feb 16, 2022 at 5:43 PM Nicolas Filotto <nfilo...@talend.com> > wrote: > >> Hi, >> >> I just wanted to let you know that with Otavio, we finally could make the >> Github's runner uses the maven daemon (with 2 threads) as you may have >> seen. It seems to help a little bit since the validation of a PR now takes >> between 45-48 minutes while it used to take more than an hour. I know that >> it is not revolutionary, but I'm afraid that with 2 cores on the runner, it >> will be hard to do much better. >> >> Would it make sense to do the same on the Jenkins' runner? >> >> Regards, >> Nicolas >> >> ________________________________ >> From: Nicolas Filotto <nfilo...@talend.com> >> Sent: Thursday, February 10, 2022 14:10 >> To: dev@camel.apache.org <dev@camel.apache.org> >> Subject: Re: Parallel Build in Camel >> >> !-------------------------------------------------------------------| >> This Message Is From an External Sender >> This message came from outside your organization. >> Exercise caution when opening attachments or clicking any >> links. >> |-------------------------------------------------------------------! >> >> Hi Guillaume, >> >> That's awsome, no I didn't know that option, thanks for sharing, it will >> definitively help. >> >> Actually even the maven daemon as it is now (wondeful job by the way) >> could help. >> Indeed, on my machine even with 2 threads, the command "mvnd clean >> install -Pfastinstall -Psourcecheck -T 2" takes 32 minutes while the >> equivalent with maven takes 48 minutes. >> >> I know that we cannot predict the results on the build pipeline based on >> this but at least it sounds like it could be interesting to try it, don't >> you agree? >> >> Regards, >> Nicolas >> >> ________________________________ >> From: Guillaume Nodet <gno...@apache.org> >> Sent: Thursday, February 10, 2022 11:24 >> To: dev@camel.apache.org <dev@camel.apache.org> >> Subject: Re: Parallel Build in Camel >> >> !-------------------------------------------------------------------| >> This Message Is From an External Sender >> This message came from outside your organization. >> Exercise caution when opening attachments or clicking any >> links. >> |-------------------------------------------------------------------! >> >> Fwiw, i did a lot of work on the build a while ago, but my main goal was >> to >> optimize subsequent `mvnd -DskipTests` run. >> Ideall, in such cases, no jars should be changed at all, but that's not >> the >> case completely, especially for maven plugins (try running `ls -ltr >> **/*.jar` after a build). There are pending changes in maven to fix that >> and I plan to push some changes in camel (mostly plugins updates) once >> those are released. >> >> In case you don't know the option, you can use the `mvnd -Dmvnd.buildTime` >> option to gather statistics on the mojo execution time. >> >> In addition, I've been working a few weeks ago on the >> maven-build-cache-extension [1], which is a powerful extension that >> provides a full per-module build cache. The cache can be configured >> locally and remotely (where the output of a module will be downloaded from >> the remote cache instead of being built locally). It currently requires >> some changes in maven-core that will only be released in maven 3.9.0. >> However, I can try to set up an experimental mvnd branch that would embed >> this extension if that's of any interest. >> >> Guillaume >> >> [1] >> https://urldefense.com/v3/__https://github.com/apache/maven-build-cache-extension__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehHmiEG7g$ >> >> Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <angusyo...@gmail.com >> > >> a écrit : >> >> > Thanks for looking into this. >> > >> > On a related note, I think one area where we could investigate potential >> > improvements to the compilation is by looking at the performance of our >> > maven plugins and code generators. >> > >> > For example the camel-endpointdsl module takes a long time to generate >> > sources and build (that one takes almost 2 minutes to run on GH). The >> > camel-componentdsl one takes about 1 minute. I suspect that if we could >> > trace why they are so slow, there may be a route for optimizing the >> build >> > time (and if we are lucky, that could happen across the whole build). >> > >> > A secondary branch of investigation could be looking at whether we can >> > adjust our build so that we can have it running with "process-classes" >> > target, to avoid the costly process of compressing and installing the >> jars. >> > >> > Kind regards >> > >> > On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nfilo...@talend.com> >> > wrote: >> > >> > > Hi again, >> > > >> > > Just to let you know that it appears that there are only 2 cores on >> the >> > > target server which is obviously not enough to have an impact on the >> > build >> > > time. >> > > >> > > Regards, >> > > Nicolas >> > > ________________________________ >> > > From: Nicolas Filotto <nfilo...@talend.com> >> > > Sent: Wednesday, February 9, 2022 11:33 >> > > To: dev@camel.apache.org <dev@camel.apache.org> >> > > Subject: Re: Parallel Build in Camel >> > > >> > > !-------------------------------------------------------------------| >> > > This Message Is From an External Sender >> > > This message came from outside your organization. >> > > Exercise caution when opening attachments or clicking any >> > > links. >> > > |-------------------------------------------------------------------! >> > > >> > > Hi, >> > > >> > > Thank you very much for your feedbacks. Here is the related PR >> > > >> > >> https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$ >> > > . >> > > >> > > Regards, >> > > Nicolas >> > > >> > > ________________________________ >> > > From: Otavio Rodolfo Piske <angusyo...@gmail.com> >> > > Sent: Wednesday, February 9, 2022 10:26 >> > > To: dev@camel.apache.org <dev@camel.apache.org> >> > > Subject: Re: Parallel Build in Camel >> > > >> > > !-------------------------------------------------------------------| >> > > This Message Is From an External Sender >> > > This message came from outside your organization. >> > > Exercise caution when opening attachments or clicking any >> > > links. >> > > |-------------------------------------------------------------------! >> > > >> > > Hi, >> > > >> > > I think it would be OK to give it a try. Please, feel free to open a >> PR >> > > with this suggestion. >> > > >> > > Kind regards >> > > >> > > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nfilo...@talend.com> >> > > wrote: >> > > >> > > > Hi, >> > > > >> > > > If the parallel build is something that can be used in the build >> > > pipeline, >> > > > why not simply starting by adding the -T option of maven to the >> > commands >> > > > launched by the build ( >> > > > >> > > >> > >> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$ >> > > ) >> > > > as short/middle term solution before eventually considering the >> maven >> > > > daemon? What do you think of it? >> > > > >> > > > Regards, >> > > > Nicolas >> > > > >> > > > ________________________________ >> > > > From: Otavio Rodolfo Piske <angusyo...@gmail.com> >> > > > Sent: Wednesday, February 9, 2022 10:03 >> > > > To: dev@camel.apache.org <dev@camel.apache.org> >> > > > Subject: Re: Parallel Build in Camel >> > > > >> > > > >> !-------------------------------------------------------------------| >> > > > This Message Is From an External Sender >> > > > This message came from outside your organization. >> > > > Exercise caution when opening attachments or clicking any >> > > > links. >> > > > >> |-------------------------------------------------------------------! >> > > > >> > > > Hi, >> > > > >> > > > Yeah, that helps a bit, although it's still quite a lot. One day I'd >> > like >> > > > to take a closer look at some of the plugins we have on the build to >> > see >> > > if >> > > > there's some easy speed up we could do. >> > > > >> > > > As for the Github, I think we depend on 2 things: >> > > > 1. I believe we need a Github action that contains and runs maven >> > daemon >> > > > 2. I believe this action needs to be approved somehow by the ASF. I >> am >> > > not >> > > > sure about this, though ... maybe others in the community know more >> > about >> > > > it. >> > > > >> > > > Kind regards >> > > > >> > > > >> > > > >> > > > As a recipient of an email from Talend, your contact personal data >> will >> > > be >> > > > on our systems. Please see our privacy notice (updated August 2020) >> at >> > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> >> > > > >> > > > >> > > > >> > > >> > > -- >> > > Otavio R. Piske >> > > >> > > >> > >> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$ >> > > >> > > As a recipient of an email from Talend, your contact personal data >> will >> > be >> > > on our systems. Please see our privacy notice (updated August 2020) at >> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> >> > > >> > > >> > > >> > > As a recipient of an email from Talend, your contact personal data >> will >> > be >> > > on our systems. Please see our privacy notice (updated August 2020) at >> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> >> > > >> > > >> > > >> > >> > -- >> > Otavio R. Piske >> > >> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehQC12Kv4$ >> > >> >> >> -- >> ------------------------ >> Guillaume Nodet >> >> As a recipient of an email from Talend, your contact personal data will >> be on our systems. Please see our privacy notice (updated August 2020) at >> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/> >> >> >> >> As a recipient of an email from Talend, your contact personal data will >> be on our systems. Please see our privacy notice. < >> https://www.talend.com/privacy/> >> >> >> > > -- > Otavio R. Piske > http://orpiske.net > -- Otavio R. Piske http://orpiske.net
