Le sam. 14 sept. 2019 à 19:34, Maximilian Novikov <maximilian.novi...@db.com> a écrit :
> Classification: For internal use only > > Ok, looks like this is the only option for us: create extension and > override MojoExecutor. > > > The only challenge is an exhaustive test suite since your current impl > can easily fake a passing build (as gradle does today if you don't disable > the daemon and state cache on the CI). > I assume we are safe here. Our solution provides incremental build with > per project granularity. So there is no need in smart "test relationship > discovery" within a project. > Well, for you I trust you, but not as a generic core maven feature so if it stays like that it can't exit extension state IMHO. > -----Original Message----- > From: Romain Manni-Bucau [mailto:rmannibu...@gmail.com] > Sent: Saturday, September 14, 2019 11:48 AM > To: Maven Developers List <dev@maven.apache.org> > Subject: Re: [VOTE] Maven incremental build for BIG-sized projects with > local and remote caching > > Le sam. 14 sept. 2019 à 08:00, Alexander Ashitkin <ashitkin.a...@gmail.com> > a écrit : > > > Indeed we have a kind of the option 2 with variations. Current > > implementation is opt-in feature driven by configuration with some > > metadata of required cache behavior and hints. > > > > Maven extensions is the option, but we would love to have it in maven > > itself which is in interest of maven community i believe. Extension is > > a way we are trying to avoid and even not sure it could be implemented > > as extension as it requires changes in maven core. > > > > No real change required in maven core here since guice enables to override > any bean or even just to rewrite the pom to remove modules to just rebuild > the minimum set (keeping downstream project). > > The only challenge is an exhaustive test suite since your current impl can > easily fake a passing build (as gradle does today if you dont disable the > daemon and state cache on the CI). > > Side note: test relationship discovery is close to AOT in terms of impl > and very very slow so can be worse than doing the full suite in simple > projects and it still asks the IT question. > > So due to the numerous "?" of a core solution, extension is the way to go. > Now if a guice bean in core can help to write your extension, it can > surely be reviewed more easily IMHO. > > Hope it helps. > > > > Thanks in advance, Aleks > > > > On 2019/09/13 21:37:15, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > > > There are multiple possible incremental support: > > > > > > 1. Scm related: do a status and rebuild downstream reactor 2. Full > > > and module build graph: seems it is the one you target, ie bypass > > > modules without change. Note that it only works if upstream graph is > > taken > > > into account. > > > 3. Full build: each mojo has incremental support so the full build > > > gets > > it. > > > Issue is that it requires each mojo to know if it needs to be > > > executed or give enough info to the mojo executor to do so (gradle > > > requires all inputs/outputs to assume this state - which is still > > > just an heuristic > > and > > > not 100% reliable). > > > > > > In current state, 2. sounds like a good option since 3 can require > > > a > > loot > > > of work for external plugins (today's builds have a lot more of not > > > maven provide plugins than core plugins). > > > Now, we should be able to activate it or not so having a > > > cacheLocation config in settings.xml can be good. > > > > > > Side notes: > > > > > > 1. having it on by default will break builds - reactor is > > > deterministic > > and > > > bypassing a module can break a build since it can init maven > > > properties - for ex - for next modules 2. You cant find all in/out > > > paths from the pom in general so your algo is not generic, a meta > > > config can be needed in .mvn 3. We should let a mojo be able to > > > disable that to replace default logic (surefire is a good example > > > where it must be refined and it can save > > hours > > > there ;)) > > > 4. Let's try to impl it as a mvn extension first then if it works > > > well on multiple big project get it to core? > > > > > > Romain > > > > > > > > > > > > Le ven. 13 sept. 2019 à 23:18, Tibor Digana <tibordig...@apache.org> > > > a écrit : > > > > > > > In theory, the incremental compiler would make it faster. > > > > But this can be told only if you present a demo project with has > > trivial > > > > tests taking much less time to complete than the compiler. > > > > > > > > In reality the tests in huge projects take significantly longer > > > > time > > than > > > > the compiler. > > > > Some developers say "switch off all the tests" in the release > > > > phase but that's wrong because then the quality goes down and > > > > methodologies are broken. > > > > > > > > I can see a big problem that we do not have an interface between > > Surefire > > > > and Compiler plugin negotiating which tests have been modified > > including > > > > modules and classes in the entire structure. > > > > > > > > Having incremental compiler is easy, just use compiler:3.8.1 or > > > > use the Takari compiler. > > > > But IMO the biggest benefit in performance would be after having > > > > the > > truly > > > > incremental test executor. > > > > > > > > On Fri, Sep 13, 2019 at 10:46 PM Maximilian Novikov < > > > > maximilian.novi...@db.com> wrote: > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > *We want to create upstream change to Maven* to support true > > incremental > > > > > build for big-sized projects. > > > > > > > > > > To raise a pull request we have to pass long chain of Deutsche > > > > > Bank’s internal procedures. So, *before starting the process we > > > > > would like > > to > > > > > get your feedback regarding this feature*. > > > > > > > > > > > > > > > > > > > > *Motivation:* > > > > > > > > > > > > > > > > > > > > Our project is hosted in mono-repo and contains ~600 modules. > > > > > All > > modules > > > > > has the same SNAPSHOT version. > > > > > > > > > > There are lot of test automation around this, everything is > > > > > tested > > before > > > > > merge into release branch. > > > > > > > > > > > > > > > > > > > > Current setup helps us to simplify build/release/dependency > > management > > > > for > > > > > 10+ teams those contribute into codebase. We can release > > > > > 10+ everything > > in > > > > > 1-click. > > > > > > > > > > The major drawback of such approach is build time: *full local > > > > > build > > took > > > > > 45-60 min (*-T8)*, CI build ~25min(*-T16*)*. > > > > > > > > > > > > > > > > > > > > To speed-up our build we needed 2 features: incremental build > > > > > and > > shared > > > > > cache. > > > > > > > > > > Initially we started to think about migration to Gradle or > > > > > Bazel. As migration costs for the mentioned tools were too high, > > > > > we decided to > > add > > > > > similar functionality into Maven. > > > > > > > > > > > > > > > > > > > > Current results we get: *1-2 mins for local build(*-T8*)* if > > > > > build > > was > > > > > cached by CI*, CI build ~5 mins (*-T16*).* > > > > > > > > > > > > > > > > > > > > *Feature description:* > > > > > > > > > > > > > > > > > > > > The idea is to calculate checksum for inputs and save outputs in > > cache. > > > > > > > > > > [image: image2019-8-27_20-0-14.png] > > > > > > > > > > Each node checksum calculated with: > > > > > > > > > > > > > > > > > > > > · Effective POM hash > > > > > > > > > > · Sources hash > > > > > > > > > > · Dependencies hash (dependencies within multi-module > > project) > > > > > > > > > > > > > > > > > > > > Project sources inputs are searched inside project + all paths > > > > > from plugins configuration: > > > > > > > > > > [image: image2019-8-30_10-28-56.png] > > > > > > > > > > How does it work in practice: > > > > > > > > > > > > > > > > > > > > 1. CI: runs builds and stores outputs in shared cache > > > > > > > > > > 2. CI: reuse outputs for same inputs, so time is decreasing > > > > > > > > > > 3. Locally: when I checkout branch and run ‘install’ for > whole > > > > > project, I get all actual snapshots from remote cache for this > > > > > branch > > > > > > > > > > 4. Locally: if I change multiple modules in tree, only > changed > > > > > subtree is rebuilt > > > > > > > > > > > > > > > > > > > > Impact on current Maven codebase is very localized > > > > > (MojoExecutor, > > where > > > > we > > > > > injected cache controller). > > > > > > > > > > Caching can be activated/deactivated by property, so current > > > > > maven > > flow > > > > > will work as is. > > > > > > > > > > > > > > > > > > > > And the big plus is that you don’t need to re-work your current > > project. > > > > > Caching should work out of box, just need to add config in .mvn > > folder. > > > > > > > > > > > > > > > > > > > > Please let us know what do you think. We are ready to invest in > > > > > this feature and address any further feedback. > > > > > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > > > Max > > > > > > > > > > > > > > > > > > > > > > > > > --- > > > > > This e-mail may contain confidential and/or privileged information. > > If > > > > you > > > > > are not the intended recipient (or have received this e-mail in > > error) > > > > > please notify the sender immediately and delete this e-mail. Any > > > > > unauthorized copying, disclosure or distribution of the material > > > > > in > > this > > > > > e-mail is strictly forbidden. > > > > > > > > > > Please refer to https://www.db.com/disclosures for additional EU > > > > > corporate and regulatory disclosures and to > > > > > http://www.db.com/unitedkingdom/content/privacy.htm for > > > > > information > > > > about > > > > > privacy. > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For > > additional commands, e-mail: dev-h...@maven.apache.org > > > > > > > --- > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient (or have received this e-mail in error) > please notify the sender immediately and delete this e-mail. Any > unauthorized copying, disclosure or distribution of the material in this > e-mail is strictly forbidden. > > Please refer to https://www.db.com/disclosures for additional EU > corporate and regulatory disclosures and to > http://www.db.com/unitedkingdom/content/privacy.htm for information about > privacy. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org >
