Le sam. 14 sept. 2019 à 19:11, Maximilian Novikov <maximilian.novi...@db.com> a écrit :
> Classification: For internal use only > > It's moving to off-topic. > > Monorepo != single build > Agree but then how your figures relates to your repo? You exposed a single build (or equivalent with downstream jobs) state. Monorepo != monolithic application > Ack > We moved from poly-repo to monorepo 3 years ago and see value in this. > Let's say that this approach exists. It has own benefits/drawbacks > comparing to poly-repo. > Ack I am just trying to catch you on your particular case and original figures which don't represent a monorepo but a monoproject build for now. Goal is to ensure we fix it right for you too and not aside. Incremental build is good but monorepo is another story where a quick win is inter projects build (project vs mojo level to make it clear). > -----Original Message----- > From: Romain Manni-Bucau [mailto:rmannibu...@gmail.com] > Sent: Saturday, September 14, 2019 7:42 PM > To: Maven Developers List <dev@maven.apache.org> > Subject: Re: [VOTE] Maven incremental build for BIG-sized projects with > local and remote caching > > Hope I didnt miss it but how monorepo=single build? > > It is working well to not have a common parent too and is unlinked to > monorepo which uses local relative paths in general (at least in the > references you quoted which are also not about java ;)). > > Unrelated to making maven better at incremental builds but both tracks can > help you to get a very fast build feedback. > > Le sam. 14 sept. 2019 à 17:35, Robert Scholte <rfscho...@apache.org> a > écrit : > > > https://issues.apache.org/jira/browse/MPLUGIN-350 is the issue to > > start with. > > > > Please read all the comments, because my original thought won't work. > > > > thanks, > > Robert > > > > On Sat, 14 Sep 2019 17:10:13 +0200, Alexander Ashitkin > > <ashitkin.a...@gmail.com> wrote: > > > > > We checked and price of 550$ per user makes us think twice of what's > > the > > > best way to proceed here :-) > > > Regarding plugin api - yes, changes are desirable to make maven > > > model cache-friendly. Both in plugin invocation model and > > > Mojo#execute input/output apis. But it is possible to work with > > > current model with declarative approach. > > > > > > Thanks in advance > > > > > > On 2019/09/14 10:45:24, Tibor Digana <tibordig...@apache.org> wrote: > > >> But I do not understand why the Maven should be responsible for the > > >> project cahe control/management of "/target" directories. > > >> It is a responsibility of the build manager which is the Jenkins. > > >> The Jenkins has the ability to archive files and such property > > >> already exists in the Jenkins. > > >> > > >> So the Jenkins has a full knowledge about: > > >> > > >> 1. how long the workspace content retains intact 2. what commit > > >> hash is for the last build/job/branch 3. and what commit was > > >> successful > > >> > > >> If the target directories retain intact (or renewed from archive) > > >> in the workspace for very long time and the workspace was reused by > > >> the next build then I would say that the improvement should work as > > >> it is on CI level. > > >> > > >> Maybe what is necessary is only that improvement in Maven where we > > >> would obtain the list of modules or directories of changes in the > > >> current commit. > > >> Then the Maven can highly optimize its own build steps and build > > >> only those modules which have been changed and their dependent > > >> modules. > > >> So the interface between CI and Maven is needed in a kind of > > >> extension or the class MavenCli can be extended with some new > > >> entrypoint. > > >> > > >> But I do not hink that Maven has to take care of responsibilities > > >> of CI (project cache mgmt), that's not our task I would say and we > > >> as Maven would never know all about the miscellaneous CI specifics > > >> and therefore we would not cope with CI related troubles. > > >> > > >> Cheers > > >> Tibor17 > > >> > > >> > > >> > > >> On Sat, Sep 14, 2019 at 11:08 AM Robert Scholte > > >> <rfscho...@apache.org> > > >> wrote: > > >> > > >> > On Fri, 13 Sep 2019 23:37:15 +0200, Romain Manni-Bucau > > >> > <rmannibu...@gmail.com> wrote: > > >> > > > >> > > There are multiple possible incremental support: > > >> > > > > >> > > 1. Scm related: do a status and rebuild downstream reactor 2. > > >> > > Full and module build graph: seems it is the one you target, ie > > >> bypass > > >> > > modules without change. Note that it only works if upstream > > >> > > graph is taken into account. > > >> > > 3. Full build: each mojo has incremental support so the full > > >> > > build > > >> gets > > >> > > it. > > >> > > Issue is that it requires each mojo to know if it needs to be > > >> executed or > > >> > > give enough info to the mojo executor to do so (gradle requires > > >> > > all inputs/outputs to assume this state - which is still just > > >> > > an > > >> heuristic > > >> > > and > > >> > > not 100% reliable). > > >> > > > > >> > > In current state, 2. sounds like a good option since 3 can > > >> > > require > > > > >> a > > >> > > loot > > >> > > of work for external plugins (today's builds have a lot more of > > not > > >> maven > > >> > > provide plugins than core plugins). > > >> > > Now, we should be able to activate it or not so having a > > >> cacheLocation > > >> > > config in settings.xml can be good. > > >> > > > > >> > > Side notes: > > >> > > > > >> > > 1. having it on by default will break builds - reactor is > > >> deterministic > > >> > > and > > >> > > bypassing a module can break a build since it can init maven > > >> properties - > > >> > > for ex - for next modules > > >> > > 2. You cant find all in/out paths from the pom in general so > > >> > > your > > >> algo is > > >> > > not generic, a meta config can be needed in .mvn 3. We should > > >> > > let a mojo be able to disable that to replace default > > >> logic > > >> > > (surefire is a good example where it must be refined and it can > > >> > > save hours there ;)) 4. Let's try to impl it as a mvn extension > > >> > > first then if it works > > >> well on > > >> > > multiple big project get it to core? > > >> > > > >> > Did anyone Google for "maven extension build cache"? There are > > >> > already commercial solutions for it. > > >> > Even though I would like to see improvements in this area, the > > >> > old architecture of Maven makes it quite hard to move to that > situation. > > >> > First > > >> > of all it requires changes to the Plugin API (without breaking > > >> backwards > > >> > compatibility) to have support out of the box. > > >> > > > >> > Robert > > >> > > > >> > > > > >> > > Romain > > >> > > > > >> > > > > >> > > > > >> > > Le ven. 13 sept. 2019 à 23:18, Tibor Digana > > >> <tibordig...@apache.org> a > > >> > > écrit : > > >> > > > > >> > >> In theory, the incremental compiler would make it faster. > > >> > >> But this can be told only if you present a demo project with > > >> > >> has > > >> trivial > > >> > >> tests taking much less time to complete than the compiler. > > >> > >> > > >> > >> In reality the tests in huge projects take significantly > > >> > >> longer > > >> time > > >> > >> than > > >> > >> the compiler. > > >> > >> Some developers say "switch off all the tests" in the release > > >> phase but > > >> > >> that's wrong because then the quality goes down and > > >> > >> methodologies > > >> are > > >> > >> broken. > > >> > >> > > >> > >> I can see a big problem that we do not have an interface > > >> > >> between Surefire and Compiler plugin negotiating which tests > > >> > >> have been modified > > >> including > > >> > >> modules and classes in the entire structure. > > >> > >> > > >> > >> Having incremental compiler is easy, just use compiler:3.8.1 > > >> > >> or > > >> use the > > >> > >> Takari compiler. > > >> > >> But IMO the biggest benefit in performance would be after > > >> > >> having > > >> the > > >> > >> truly > > >> > >> incremental test executor. > > >> > >> > > >> > >> On Fri, Sep 13, 2019 at 10:46 PM Maximilian Novikov < > > >> > >> maximilian.novi...@db.com> wrote: > > >> > >> > > >> > >> > Hi All, > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > *We want to create upstream change to Maven* to support true > > >> > >> incremental > > >> > >> > build for big-sized projects. > > >> > >> > > > >> > >> > To raise a pull request we have to pass long chain of > > >> > >> > Deutsche > > >> Bank’s > > >> > >> > internal procedures. So, *before starting the process we > > >> > >> > would > > >> like to > > >> > >> > get your feedback regarding this feature*. > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > *Motivation:* > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > Our project is hosted in mono-repo and contains ~600 > > >> > >> > modules. All > > >> > >> modules > > >> > >> > has the same SNAPSHOT version. > > >> > >> > > > >> > >> > There are lot of test automation around this, everything is > > >> tested > > >> > >> before > > >> > >> > merge into release branch. > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > Current setup helps us to simplify build/release/dependency > > >> management > > >> > >> for > > >> > >> > 10+ teams those contribute into codebase. We can release > > >> everything in > > >> > >> > 1-click. > > >> > >> > > > >> > >> > The major drawback of such approach is build time: *full > > >> > >> > local > > >> build > > >> > >> took > > >> > >> > 45-60 min (*-T8)*, CI build ~25min(*-T16*)*. > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > To speed-up our build we needed 2 features: incremental > > >> > >> > build and > > >> > >> shared > > >> > >> > cache. > > >> > >> > > > >> > >> > Initially we started to think about migration to Gradle or > > >> Bazel. As > > >> > >> > migration costs for the mentioned tools were too high, we > > >> decided to > > >> > >> add > > >> > >> > similar functionality into Maven. > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > Current results we get: *1-2 mins for local build(*-T8*)* if > > >> build was > > >> > >> > cached by CI*, CI build ~5 mins (*-T16*).* > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > *Feature description:* > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > The idea is to calculate checksum for inputs and save > > >> > >> > outputs in > > >> > >> cache. > > >> > >> > > > >> > >> > [image: image2019-8-27_20-0-14.png] > > >> > >> > > > >> > >> > Each node checksum calculated with: > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > · Effective POM hash > > >> > >> > > > >> > >> > · Sources hash > > >> > >> > > > >> > >> > · Dependencies hash (dependencies within multi-module > > >> project) > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > Project sources inputs are searched inside project + all > > >> > >> > paths > > >> from > > >> > >> > plugins configuration: > > >> > >> > > > >> > >> > [image: image2019-8-30_10-28-56.png] > > >> > >> > > > >> > >> > How does it work in practice: > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > 1. CI: runs builds and stores outputs in shared cache > > >> > >> > > > >> > >> > 2. CI: reuse outputs for same inputs, so time is > decreasing > > >> > >> > > > >> > >> > 3. Locally: when I checkout branch and run ‘install’ for > > >> whole > > >> > >> > project, I get all actual snapshots from remote cache for > > >> > >> > this > > >> branch > > >> > >> > > > >> > >> > 4. Locally: if I change multiple modules in tree, only > > >> changed > > >> > >> > subtree is rebuilt > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > Impact on current Maven codebase is very localized > > >> > >> > (MojoExecutor, > > >> > >> where > > >> > >> we > > >> > >> > injected cache controller). > > >> > >> > > > >> > >> > Caching can be activated/deactivated by property, so current > > >> maven > > >> > >> flow > > >> > >> > will work as is. > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > And the big plus is that you don’t need to re-work your > > >> > >> > current > > >> > >> project. > > >> > >> > Caching should work out of box, just need to add config in > > >> > >> > .mvn > > >> > >> folder. > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > Please let us know what do you think. We are ready to invest > > >> > >> > in > > >> this > > >> > >> > feature and address any further feedback. > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > Kind regards, > > >> > >> > > > >> > >> > Max > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > --- > > >> > >> > This e-mail may contain confidential and/or privileged > > >> information. If > > >> > >> you > > >> > >> > are not the intended recipient (or have received this e-mail > > >> > >> > in > > >> error) > > >> > >> > please notify the sender immediately and delete this e-mail. > > >> > >> > Any unauthorized copying, disclosure or distribution of the > > material > > >> in > > >> > >> this > > >> > >> > e-mail is strictly forbidden. > > >> > >> > > > >> > >> > Please refer to https://www.db.com/disclosures for > > >> > >> > additional EU corporate and regulatory disclosures and to > > >> > >> > http://www.db.com/unitedkingdom/content/privacy.htm for > > >> information > > >> > >> about > > >> > >> > privacy. > > >> > >> > > > >> > > > >> > ----------------------------------------------------------------- > > >> > ---- To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For > > >> > additional commands, e-mail: dev-h...@maven.apache.org > > >> > > > >> > > > >> > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For > > > additional commands, e-mail: dev-h...@maven.apache.org > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For > > additional commands, e-mail: dev-h...@maven.apache.org > > > > > > > --- > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient (or have received this e-mail in error) > please notify the sender immediately and delete this e-mail. Any > unauthorized copying, disclosure or distribution of the material in this > e-mail is strictly forbidden. > > Please refer to https://www.db.com/disclosures for additional EU > corporate and regulatory disclosures and to > http://www.db.com/unitedkingdom/content/privacy.htm for information about > privacy. >
