Let us evaluate this approach. But if we go extension way, it will be not so big motivation to make it part of maven. and i'm not sure what are long term strategy for maven, but without incremental builld it becomes less and less attractive in our multi-branched world
Thank you On 2019/09/14 08:48:00, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Le sam. 14 sept. 2019 à 08:00, Alexander Ashitkin <ashitkin.a...@gmail.com> > a écrit : > > > Indeed we have a kind of the option 2 with variations. Current > > implementation is opt-in feature driven by configuration with some metadata > > of required cache behavior and hints. > > > > Maven extensions is the option, but we would love to have it in maven > > itself which is in interest of maven community i believe. Extension is a > > way we are trying to avoid and even not sure it could be implemented as > > extension as it requires changes in maven core. > > > > No real change required in maven core here since guice enables to override > any bean or even just to rewrite the pom to remove modules to just rebuild > the minimum set (keeping downstream project). > > The only challenge is an exhaustive test suite since your current impl can > easily fake a passing build (as gradle does today if you dont disable the > daemon and state cache on the CI). > > Side note: test relationship discovery is close to AOT in terms of impl and > very very slow so can be worse than doing the full suite in simple projects > and it still asks the IT question. > > So due to the numerous "?" of a core solution, extension is the way to go. > Now if a guice bean in core can help to write your extension, it can surely > be reviewed more easily IMHO. > > Hope it helps. > > > > Thanks in advance, Aleks > > > > On 2019/09/13 21:37:15, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > There are multiple possible incremental support: > > > > > > 1. Scm related: do a status and rebuild downstream reactor > > > 2. Full and module build graph: seems it is the one you target, ie bypass > > > modules without change. Note that it only works if upstream graph is > > taken > > > into account. > > > 3. Full build: each mojo has incremental support so the full build gets > > it. > > > Issue is that it requires each mojo to know if it needs to be executed or > > > give enough info to the mojo executor to do so (gradle requires all > > > inputs/outputs to assume this state - which is still just an heuristic > > and > > > not 100% reliable). > > > > > > In current state, 2. sounds like a good option since 3 can require a > > loot > > > of work for external plugins (today's builds have a lot more of not maven > > > provide plugins than core plugins). > > > Now, we should be able to activate it or not so having a cacheLocation > > > config in settings.xml can be good. > > > > > > Side notes: > > > > > > 1. having it on by default will break builds - reactor is deterministic > > and > > > bypassing a module can break a build since it can init maven properties - > > > for ex - for next modules > > > 2. You cant find all in/out paths from the pom in general so your algo is > > > not generic, a meta config can be needed in .mvn > > > 3. We should let a mojo be able to disable that to replace default logic > > > (surefire is a good example where it must be refined and it can save > > hours > > > there ;)) > > > 4. Let's try to impl it as a mvn extension first then if it works well on > > > multiple big project get it to core? > > > > > > Romain > > > > > > > > > > > > Le ven. 13 sept. 2019 à 23:18, Tibor Digana <tibordig...@apache.org> a > > > écrit : > > > > > > > In theory, the incremental compiler would make it faster. > > > > But this can be told only if you present a demo project with has > > trivial > > > > tests taking much less time to complete than the compiler. > > > > > > > > In reality the tests in huge projects take significantly longer time > > than > > > > the compiler. > > > > Some developers say "switch off all the tests" in the release phase but > > > > that's wrong because then the quality goes down and methodologies are > > > > broken. > > > > > > > > I can see a big problem that we do not have an interface between > > Surefire > > > > and Compiler plugin negotiating which tests have been modified > > including > > > > modules and classes in the entire structure. > > > > > > > > Having incremental compiler is easy, just use compiler:3.8.1 or use the > > > > Takari compiler. > > > > But IMO the biggest benefit in performance would be after having the > > truly > > > > incremental test executor. > > > > > > > > On Fri, Sep 13, 2019 at 10:46 PM Maximilian Novikov < > > > > maximilian.novi...@db.com> wrote: > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > *We want to create upstream change to Maven* to support true > > incremental > > > > > build for big-sized projects. > > > > > > > > > > To raise a pull request we have to pass long chain of Deutsche Bank’s > > > > > internal procedures. So, *before starting the process we would like > > to > > > > > get your feedback regarding this feature*. > > > > > > > > > > > > > > > > > > > > *Motivation:* > > > > > > > > > > > > > > > > > > > > Our project is hosted in mono-repo and contains ~600 modules. All > > modules > > > > > has the same SNAPSHOT version. > > > > > > > > > > There are lot of test automation around this, everything is tested > > before > > > > > merge into release branch. > > > > > > > > > > > > > > > > > > > > Current setup helps us to simplify build/release/dependency > > management > > > > for > > > > > 10+ teams those contribute into codebase. We can release everything > > in > > > > > 1-click. > > > > > > > > > > The major drawback of such approach is build time: *full local build > > took > > > > > 45-60 min (*-T8)*, CI build ~25min(*-T16*)*. > > > > > > > > > > > > > > > > > > > > To speed-up our build we needed 2 features: incremental build and > > shared > > > > > cache. > > > > > > > > > > Initially we started to think about migration to Gradle or Bazel. As > > > > > migration costs for the mentioned tools were too high, we decided to > > add > > > > > similar functionality into Maven. > > > > > > > > > > > > > > > > > > > > Current results we get: *1-2 mins for local build(*-T8*)* if build > > was > > > > > cached by CI*, CI build ~5 mins (*-T16*).* > > > > > > > > > > > > > > > > > > > > *Feature description:* > > > > > > > > > > > > > > > > > > > > The idea is to calculate checksum for inputs and save outputs in > > cache. > > > > > > > > > > [image: image2019-8-27_20-0-14.png] > > > > > > > > > > Each node checksum calculated with: > > > > > > > > > > > > > > > > > > > > · Effective POM hash > > > > > > > > > > · Sources hash > > > > > > > > > > · Dependencies hash (dependencies within multi-module > > project) > > > > > > > > > > > > > > > > > > > > Project sources inputs are searched inside project + all paths from > > > > > plugins configuration: > > > > > > > > > > [image: image2019-8-30_10-28-56.png] > > > > > > > > > > How does it work in practice: > > > > > > > > > > > > > > > > > > > > 1. CI: runs builds and stores outputs in shared cache > > > > > > > > > > 2. CI: reuse outputs for same inputs, so time is decreasing > > > > > > > > > > 3. Locally: when I checkout branch and run ‘install’ for whole > > > > > project, I get all actual snapshots from remote cache for this branch > > > > > > > > > > 4. Locally: if I change multiple modules in tree, only changed > > > > > subtree is rebuilt > > > > > > > > > > > > > > > > > > > > Impact on current Maven codebase is very localized (MojoExecutor, > > where > > > > we > > > > > injected cache controller). > > > > > > > > > > Caching can be activated/deactivated by property, so current maven > > flow > > > > > will work as is. > > > > > > > > > > > > > > > > > > > > And the big plus is that you don’t need to re-work your current > > project. > > > > > Caching should work out of box, just need to add config in .mvn > > folder. > > > > > > > > > > > > > > > > > > > > Please let us know what do you think. We are ready to invest in this > > > > > feature and address any further feedback. > > > > > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > > > Max > > > > > > > > > > > > > > > > > > > > > > > > > --- > > > > > This e-mail may contain confidential and/or privileged information. > > If > > > > you > > > > > are not the intended recipient (or have received this e-mail in > > error) > > > > > please notify the sender immediately and delete this e-mail. Any > > > > > unauthorized copying, disclosure or distribution of the material in > > this > > > > > e-mail is strictly forbidden. > > > > > > > > > > Please refer to https://www.db.com/disclosures for additional EU > > > > > corporate and regulatory disclosures and to > > > > > http://www.db.com/unitedkingdom/content/privacy.htm for information > > > > about > > > > > privacy. > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > For additional commands, e-mail: dev-h...@maven.apache.org > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
