The feature doesn't do per Test/Classes analysis. Granularity of incremental build is per project - if project is invalidated by changes in dependencies/source code/plugins it will be rebuilt fully. So, single module of multi-module build doesn't receive any increment, but for multi module projects with hundreds of modules just skipping not affected part of graph is a huge win.
Sincerely yours, Aleks On 2019/09/13 21:48:37, Tibor Digana <tibordig...@apache.org> wrote: > Disabling a Surefire/Failsafe in a particular module is easy but it won't > gain the performance so much if you do not analyse the relations between > classes and the test. > > If you analyse the relations then you can easily fetch the list of the > tests in -Dtests or in the included/excludedTests. So everything is in > Surefire but I guess that the analysis of the code would be so time > demanding that it would be all contraprodactive effort on our side. > > Regarding the cache, the repo is the cache. And if the CI build deletes the > repo, we would not know it today. > So these performance improvements must be optional feature only enabled by > the user and not by default. > > On Fri, Sep 13, 2019 at 11:37 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > > > There are multiple possible incremental support: > > > > 1. Scm related: do a status and rebuild downstream reactor > > 2. Full and module build graph: seems it is the one you target, ie bypass > > modules without change. Note that it only works if upstream graph is taken > > into account. > > 3. Full build: each mojo has incremental support so the full build gets it. > > Issue is that it requires each mojo to know if it needs to be executed or > > give enough info to the mojo executor to do so (gradle requires all > > inputs/outputs to assume this state - which is still just an heuristic and > > not 100% reliable). > > > > In current state, 2. sounds like a good option since 3 can require a loot > > of work for external plugins (today's builds have a lot more of not maven > > provide plugins than core plugins). > > Now, we should be able to activate it or not so having a cacheLocation > > config in settings.xml can be good. > > > > Side notes: > > > > 1. having it on by default will break builds - reactor is deterministic and > > bypassing a module can break a build since it can init maven properties - > > for ex - for next modules > > 2. You cant find all in/out paths from the pom in general so your algo is > > not generic, a meta config can be needed in .mvn > > 3. We should let a mojo be able to disable that to replace default logic > > (surefire is a good example where it must be refined and it can save hours > > there ;)) > > 4. Let's try to impl it as a mvn extension first then if it works well on > > multiple big project get it to core? > > > > Romain > > > > > > > > Le ven. 13 sept. 2019 à 23:18, Tibor Digana <tibordig...@apache.org> a > > écrit : > > > > > In theory, the incremental compiler would make it faster. > > > But this can be told only if you present a demo project with has trivial > > > tests taking much less time to complete than the compiler. > > > > > > In reality the tests in huge projects take significantly longer time than > > > the compiler. > > > Some developers say "switch off all the tests" in the release phase but > > > that's wrong because then the quality goes down and methodologies are > > > broken. > > > > > > I can see a big problem that we do not have an interface between Surefire > > > and Compiler plugin negotiating which tests have been modified including > > > modules and classes in the entire structure. > > > > > > Having incremental compiler is easy, just use compiler:3.8.1 or use the > > > Takari compiler. > > > But IMO the biggest benefit in performance would be after having the > > truly > > > incremental test executor. > > > > > > On Fri, Sep 13, 2019 at 10:46 PM Maximilian Novikov < > > > maximilian.novi...@db.com> wrote: > > > > > > > Hi All, > > > > > > > > > > > > > > > > *We want to create upstream change to Maven* to support true > > incremental > > > > build for big-sized projects. > > > > > > > > To raise a pull request we have to pass long chain of Deutsche Bank’s > > > > internal procedures. So, *before starting the process we would like to > > > > get your feedback regarding this feature*. > > > > > > > > > > > > > > > > *Motivation:* > > > > > > > > > > > > > > > > Our project is hosted in mono-repo and contains ~600 modules. All > > modules > > > > has the same SNAPSHOT version. > > > > > > > > There are lot of test automation around this, everything is tested > > before > > > > merge into release branch. > > > > > > > > > > > > > > > > Current setup helps us to simplify build/release/dependency management > > > for > > > > 10+ teams those contribute into codebase. We can release everything in > > > > 1-click. > > > > > > > > The major drawback of such approach is build time: *full local build > > took > > > > 45-60 min (*-T8)*, CI build ~25min(*-T16*)*. > > > > > > > > > > > > > > > > To speed-up our build we needed 2 features: incremental build and > > shared > > > > cache. > > > > > > > > Initially we started to think about migration to Gradle or Bazel. As > > > > migration costs for the mentioned tools were too high, we decided to > > add > > > > similar functionality into Maven. > > > > > > > > > > > > > > > > Current results we get: *1-2 mins for local build(*-T8*)* if build was > > > > cached by CI*, CI build ~5 mins (*-T16*).* > > > > > > > > > > > > > > > > *Feature description:* > > > > > > > > > > > > > > > > The idea is to calculate checksum for inputs and save outputs in cache. > > > > > > > > [image: image2019-8-27_20-0-14.png] > > > > > > > > Each node checksum calculated with: > > > > > > > > > > > > > > > > · Effective POM hash > > > > > > > > · Sources hash > > > > > > > > · Dependencies hash (dependencies within multi-module project) > > > > > > > > > > > > > > > > Project sources inputs are searched inside project + all paths from > > > > plugins configuration: > > > > > > > > [image: image2019-8-30_10-28-56.png] > > > > > > > > How does it work in practice: > > > > > > > > > > > > > > > > 1. CI: runs builds and stores outputs in shared cache > > > > > > > > 2. CI: reuse outputs for same inputs, so time is decreasing > > > > > > > > 3. Locally: when I checkout branch and run ‘install’ for whole > > > > project, I get all actual snapshots from remote cache for this branch > > > > > > > > 4. Locally: if I change multiple modules in tree, only changed > > > > subtree is rebuilt > > > > > > > > > > > > > > > > Impact on current Maven codebase is very localized (MojoExecutor, where > > > we > > > > injected cache controller). > > > > > > > > Caching can be activated/deactivated by property, so current maven flow > > > > will work as is. > > > > > > > > > > > > > > > > And the big plus is that you don’t need to re-work your current > > project. > > > > Caching should work out of box, just need to add config in .mvn folder. > > > > > > > > > > > > > > > > Please let us know what do you think. We are ready to invest in this > > > > feature and address any further feedback. > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > Max > > > > > > > > > > > > > > > > > > > > --- > > > > This e-mail may contain confidential and/or privileged information. If > > > you > > > > are not the intended recipient (or have received this e-mail in error) > > > > please notify the sender immediately and delete this e-mail. Any > > > > unauthorized copying, disclosure or distribution of the material in > > this > > > > e-mail is strictly forbidden. > > > > > > > > Please refer to https://www.db.com/disclosures for additional EU > > > > corporate and regulatory disclosures and to > > > > http://www.db.com/unitedkingdom/content/privacy.htm for information > > > about > > > > privacy. > > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
