We checked and price of 550$ per user makes us think twice of what's the best way to proceed here :-) Regarding plugin api - yes, changes are desirable to make maven model cache-friendly. Both in plugin invocation model and Mojo#execute input/output apis. But it is possible to work with current model with declarative approach.
Thanks in advance On 2019/09/14 10:45:24, Tibor Digana <tibordig...@apache.org> wrote: > But I do not understand why the Maven should be responsible for the project > cahe control/management of "/target" directories. > It is a responsibility of the build manager which is the Jenkins. > The Jenkins has the ability to archive files and such property already > exists in the Jenkins. > > So the Jenkins has a full knowledge about: > > 1. how long the workspace content retains intact > 2. what commit hash is for the last build/job/branch > 3. and what commit was successful > > If the target directories retain intact (or renewed from archive) in the > workspace for very long time and the workspace was reused by the next build > then I would say that the improvement should work as it is on CI level. > > Maybe what is necessary is only that improvement in Maven where we would > obtain the list of modules or directories of changes in the current commit. > Then the Maven can highly optimize its own build steps and build only those > modules which have been changed and their dependent modules. > So the interface between CI and Maven is needed in a kind of extension or > the class MavenCli can be extended with some new entrypoint. > > But I do not hink that Maven has to take care of responsibilities of CI > (project cache mgmt), that's not our task I would say and we as Maven would > never know all about the miscellaneous CI specifics and therefore we would > not cope with CI related troubles. > > Cheers > Tibor17 > > > > On Sat, Sep 14, 2019 at 11:08 AM Robert Scholte <rfscho...@apache.org> > wrote: > > > On Fri, 13 Sep 2019 23:37:15 +0200, Romain Manni-Bucau > > <rmannibu...@gmail.com> wrote: > > > > > There are multiple possible incremental support: > > > > > > 1. Scm related: do a status and rebuild downstream reactor > > > 2. Full and module build graph: seems it is the one you target, ie bypass > > > modules without change. Note that it only works if upstream graph is > > > taken > > > into account. > > > 3. Full build: each mojo has incremental support so the full build gets > > > it. > > > Issue is that it requires each mojo to know if it needs to be executed or > > > give enough info to the mojo executor to do so (gradle requires all > > > inputs/outputs to assume this state - which is still just an heuristic > > > and > > > not 100% reliable). > > > > > > In current state, 2. sounds like a good option since 3 can require a > > > loot > > > of work for external plugins (today's builds have a lot more of not maven > > > provide plugins than core plugins). > > > Now, we should be able to activate it or not so having a cacheLocation > > > config in settings.xml can be good. > > > > > > Side notes: > > > > > > 1. having it on by default will break builds - reactor is deterministic > > > and > > > bypassing a module can break a build since it can init maven properties - > > > for ex - for next modules > > > 2. You cant find all in/out paths from the pom in general so your algo is > > > not generic, a meta config can be needed in .mvn > > > 3. We should let a mojo be able to disable that to replace default logic > > > (surefire is a good example where it must be refined and it can save > > > hours > > > there ;)) > > > 4. Let's try to impl it as a mvn extension first then if it works well on > > > multiple big project get it to core? > > > > Did anyone Google for "maven extension build cache"? There are already > > commercial solutions for it. > > Even though I would like to see improvements in this area, the old > > architecture of Maven makes it quite hard to move to that situation. > > First > > of all it requires changes to the Plugin API (without breaking backwards > > compatibility) to have support out of the box. > > > > Robert > > > > > > > > Romain > > > > > > > > > > > > Le ven. 13 sept. 2019 à 23:18, Tibor Digana <tibordig...@apache.org> a > > > écrit : > > > > > >> In theory, the incremental compiler would make it faster. > > >> But this can be told only if you present a demo project with has trivial > > >> tests taking much less time to complete than the compiler. > > >> > > >> In reality the tests in huge projects take significantly longer time > > >> than > > >> the compiler. > > >> Some developers say "switch off all the tests" in the release phase but > > >> that's wrong because then the quality goes down and methodologies are > > >> broken. > > >> > > >> I can see a big problem that we do not have an interface between > > >> Surefire > > >> and Compiler plugin negotiating which tests have been modified including > > >> modules and classes in the entire structure. > > >> > > >> Having incremental compiler is easy, just use compiler:3.8.1 or use the > > >> Takari compiler. > > >> But IMO the biggest benefit in performance would be after having the > > >> truly > > >> incremental test executor. > > >> > > >> On Fri, Sep 13, 2019 at 10:46 PM Maximilian Novikov < > > >> maximilian.novi...@db.com> wrote: > > >> > > >> > Hi All, > > >> > > > >> > > > >> > > > >> > *We want to create upstream change to Maven* to support true > > >> incremental > > >> > build for big-sized projects. > > >> > > > >> > To raise a pull request we have to pass long chain of Deutsche Bank’s > > >> > internal procedures. So, *before starting the process we would like to > > >> > get your feedback regarding this feature*. > > >> > > > >> > > > >> > > > >> > *Motivation:* > > >> > > > >> > > > >> > > > >> > Our project is hosted in mono-repo and contains ~600 modules. All > > >> modules > > >> > has the same SNAPSHOT version. > > >> > > > >> > There are lot of test automation around this, everything is tested > > >> before > > >> > merge into release branch. > > >> > > > >> > > > >> > > > >> > Current setup helps us to simplify build/release/dependency management > > >> for > > >> > 10+ teams those contribute into codebase. We can release everything in > > >> > 1-click. > > >> > > > >> > The major drawback of such approach is build time: *full local build > > >> took > > >> > 45-60 min (*-T8)*, CI build ~25min(*-T16*)*. > > >> > > > >> > > > >> > > > >> > To speed-up our build we needed 2 features: incremental build and > > >> shared > > >> > cache. > > >> > > > >> > Initially we started to think about migration to Gradle or Bazel. As > > >> > migration costs for the mentioned tools were too high, we decided to > > >> add > > >> > similar functionality into Maven. > > >> > > > >> > > > >> > > > >> > Current results we get: *1-2 mins for local build(*-T8*)* if build was > > >> > cached by CI*, CI build ~5 mins (*-T16*).* > > >> > > > >> > > > >> > > > >> > *Feature description:* > > >> > > > >> > > > >> > > > >> > The idea is to calculate checksum for inputs and save outputs in > > >> cache. > > >> > > > >> > [image: image2019-8-27_20-0-14.png] > > >> > > > >> > Each node checksum calculated with: > > >> > > > >> > > > >> > > > >> > · Effective POM hash > > >> > > > >> > · Sources hash > > >> > > > >> > · Dependencies hash (dependencies within multi-module project) > > >> > > > >> > > > >> > > > >> > Project sources inputs are searched inside project + all paths from > > >> > plugins configuration: > > >> > > > >> > [image: image2019-8-30_10-28-56.png] > > >> > > > >> > How does it work in practice: > > >> > > > >> > > > >> > > > >> > 1. CI: runs builds and stores outputs in shared cache > > >> > > > >> > 2. CI: reuse outputs for same inputs, so time is decreasing > > >> > > > >> > 3. Locally: when I checkout branch and run ‘install’ for whole > > >> > project, I get all actual snapshots from remote cache for this branch > > >> > > > >> > 4. Locally: if I change multiple modules in tree, only changed > > >> > subtree is rebuilt > > >> > > > >> > > > >> > > > >> > Impact on current Maven codebase is very localized (MojoExecutor, > > >> where > > >> we > > >> > injected cache controller). > > >> > > > >> > Caching can be activated/deactivated by property, so current maven > > >> flow > > >> > will work as is. > > >> > > > >> > > > >> > > > >> > And the big plus is that you don’t need to re-work your current > > >> project. > > >> > Caching should work out of box, just need to add config in .mvn > > >> folder. > > >> > > > >> > > > >> > > > >> > Please let us know what do you think. We are ready to invest in this > > >> > feature and address any further feedback. > > >> > > > >> > > > >> > > > >> > Kind regards, > > >> > > > >> > Max > > >> > > > >> > > > >> > > > >> > > > >> > --- > > >> > This e-mail may contain confidential and/or privileged information. If > > >> you > > >> > are not the intended recipient (or have received this e-mail in error) > > >> > please notify the sender immediately and delete this e-mail. Any > > >> > unauthorized copying, disclosure or distribution of the material in > > >> this > > >> > e-mail is strictly forbidden. > > >> > > > >> > Please refer to https://www.db.com/disclosures for additional EU > > >> > corporate and regulatory disclosures and to > > >> > http://www.db.com/unitedkingdom/content/privacy.htm for information > > >> about > > >> > privacy. > > >> > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > For additional commands, e-mail: dev-h...@maven.apache.org > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
