Re: Reproducible Builds for Maven

[Mark Derricutt]

Tomo Suzuki wrote on 23/09/19 3:56 PM:

Does your approach use such file to record library versions?
I don't know about what Hervé is doing, but internally we have a tool I 
wrote for handling this, we have a pom.deps file that looks like:


    repository http://nexus.XX as public;

    import smx3:smx3.upstream.bill-of-materials:1.1.22;

    resolve highest org.jetbrains:annotations:[16.0.3,17.0.0) via public;
    resolve highest 
org.apache.maven.plugins:maven-jar-plugin:[3.1.2,4.0.0) via public;
    resolve highest org.apache.cxf:cxf-codegen-plugin:[3.3.3,4.0.0) via 
public;


which when we resolve, will find the highest, snapshot, or lowest 
version in a given range - also allowing filtering out annoying things 
like beta/alpha/CR from central, and rewriting the pom.xml's.


Our tooling also has an 'import' option shown above that lets us 
standardize the versions we resolving, and breaking it up - so we have 
'upstream.bill-of-materials' and 'upstream.testing.bill-of-materials`.


As part of this we also add in  to ban all transitive build 
deps, and [] range all version references.


I keep meaning to push for open sourcing it, but just haven't had the time.

Mark


--
Sent from Postbox 

Re: [VOTE] Maven incremental build for BIG-sized projects with local and remote caching [I]

[Romain Manni-Bucau]

> no go for this change

It is likely the easiest way to interpret (and reduce the short term
investment) it but I don't read it this way, more as "it needs more
investment to work and be a core feature usable by everyone so let step the
inclusion with intermediate deliveries" (but it is still a "go go go IMHO).
I really love the first step being an extension, letting people test it and
give feedback on the broken cases until it is mature and integrable more
deeply.

Romain

Le lun. 23 sept. 2019 à 15:10, Maximilian Novikov 
a écrit :

> Classification: For internal use only
>
> Hi Enrico,
>
> It's not open sourced for now. We need to spend much efforts according
> with DB policies/procedures to make it open source.
> That's why we started this discussion, we don't want to waste time if
> change conceptually wrong(from Maven Dev Community perspective) and will be
> rejected.
>
> To sum up discussion: no go for this change.
>
> In some time we will wrap caching module in extension.
>
> Regards,
> Max
>
> -Original Message-
> From: Enrico Olivelli [mailto:eolive...@gmail.com]
> Sent: Saturday, September 21, 2019 9:50 AM
> To: Maven Developers List 
> Subject: Re: [VOTE] Maven incremental build for BIG-sized projects with
> local and remote caching
>
> Hi Maximilian,
> is there anyway to see this work ? is it already open source? (I am sorry,
> maybe I missed some email with links)
>
> Enrico
>
> Il giorno ven 20 set 2019 alle ore 19:30 Alexander Ashitkin <
> ashitkin.a...@gmail.com> ha scritto:
>
> > Hi Martijn
> > thanks for positive feedback.
> >
> > Regarding IDE part, yes you're right on integration part, but still
> > there important cases when cache helps:
> > 1) you need to navigate less in project as top level targets fast
> > enough to not drill down
> > 2) if you need to build a part of project (say only rest of wicket)
> > you need to provide up-to-date rest dependencies which are not active
> > in the subproject - and caches restores missing pieces for you without
> > rebuilding remaining part of the project
> > 3) If you need to test project and invoke test - cache saves your time
> > (as gradle does) on unchanged pieces
> > 4) and because tests run faster you can try run slow tests which often
> > too expensive in rapid development
> >
> > So maven integration in Intellij works nice. There is nothing super
> > smart here, just sharing how i benefit from the cache in everyday ide
> > work
> >
> > Thank you!
> >
> > On 2019/09/19 11:28:48, Martijn Dashorst 
> > wrote:
> > > On Thu, Sep 19, 2019 at 7:48 AM Alexander Ashitkin
> > >  wrote:
> > > > Configuration:
> > > > * verify -T4 -P default,all-shapshots-repos
> > > > * my project config (might be suboptimal for wicket)
> > > > * scala tests disabled in 2 modules (caused bytecode version
> > > > conflict
> > on my machine)
> > > >
> > > > Results
> > > > Clean state (cache disabled):   15:58 min
> > > > Second run, target up to date (cache disabled):  10:20 min
> > > > Fully cached (no changes):
> 17.507
> > s
> > > > wicketstuff-jwicket-tooltip-wtooltips changed:  34.936 s
> > > > wicketstuff-rest-utils changed:
>  54.040
> > s
> > > >
> > > > If you want to try other modules - please let me know.
> > >
> > > Nice results!
> > >
> > > > regarding ide - it's a usual maven installation, so any ide with
> > > > maven
> > integration should benefit from cache them maven action invoked
> > >
> > > My instinct says that an IDE as Eclipse won't benefit much from it,
> > > as it has its own build lifecycle. Only when you invoke a
> > > commandline Maven action (such as generate-sources) one might have a
> benefit.
> > >
> > > So in the day-to-day life the caching might not be as beneficial for
> > > developers, but commandline builds happen often enough to make this
> > > matter.
> > >
> > > Martijn
> > >
> > > 
> > > - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For
> > > additional commands, e-mail: dev-h...@maven.apache.org
> > >
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For
> > additional commands, e-mail: dev-h...@maven.apache.org
> >
> >
>
>
> ---
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and delete this e-mail. Any
> unauthorized copying, disclosure or distribution of the material in this
> e-mail is strictly forbidden.
>
> Please refer to https://www.db.com/disclosures for additional EU
> corporate and regulatory disclosures and to
> http://www.db.com/unitedkingdom/content/privacy.htm for information about
> privacy.
>

RE: [VOTE] Maven incremental build for BIG-sized projects with local and remote caching [I]

[Maximilian Novikov]

Classification: For internal use only

Hi Enrico,

It's not open sourced for now. We need to spend much efforts according with DB 
policies/procedures to make it open source.
That's why we started this discussion, we don't want to waste time if change 
conceptually wrong(from Maven Dev Community perspective) and will be rejected.

To sum up discussion: no go for this change.

In some time we will wrap caching module in extension.

Regards,
Max

-Original Message-
From: Enrico Olivelli [mailto:eolive...@gmail.com]
Sent: Saturday, September 21, 2019 9:50 AM
To: Maven Developers List 
Subject: Re: [VOTE] Maven incremental build for BIG-sized projects with local 
and remote caching

Hi Maximilian,
is there anyway to see this work ? is it already open source? (I am sorry, 
maybe I missed some email with links)

Enrico

Il giorno ven 20 set 2019 alle ore 19:30 Alexander Ashitkin < 
ashitkin.a...@gmail.com> ha scritto:

> Hi Martijn
> thanks for positive feedback.
>
> Regarding IDE part, yes you're right on integration part, but still
> there important cases when cache helps:
> 1) you need to navigate less in project as top level targets fast
> enough to not drill down
> 2) if you need to build a part of project (say only rest of wicket)
> you need to provide up-to-date rest dependencies which are not active
> in the subproject - and caches restores missing pieces for you without
> rebuilding remaining part of the project
> 3) If you need to test project and invoke test - cache saves your time
> (as gradle does) on unchanged pieces
> 4) and because tests run faster you can try run slow tests which often
> too expensive in rapid development
>
> So maven integration in Intellij works nice. There is nothing super
> smart here, just sharing how i benefit from the cache in everyday ide
> work
>
> Thank you!
>
> On 2019/09/19 11:28:48, Martijn Dashorst 
> wrote:
> > On Thu, Sep 19, 2019 at 7:48 AM Alexander Ashitkin
> >  wrote:
> > > Configuration:
> > > * verify -T4 -P default,all-shapshots-repos
> > > * my project config (might be suboptimal for wicket)
> > > * scala tests disabled in 2 modules (caused bytecode version
> > > conflict
> on my machine)
> > >
> > > Results
> > > Clean state (cache disabled):   15:58 min
> > > Second run, target up to date (cache disabled):  10:20 min
> > > Fully cached (no changes):  17.507
> s
> > > wicketstuff-jwicket-tooltip-wtooltips changed:  34.936 s
> > > wicketstuff-rest-utils changed: 54.040
> s
> > >
> > > If you want to try other modules - please let me know.
> >
> > Nice results!
> >
> > > regarding ide - it's a usual maven installation, so any ide with
> > > maven
> integration should benefit from cache them maven action invoked
> >
> > My instinct says that an IDE as Eclipse won't benefit much from it,
> > as it has its own build lifecycle. Only when you invoke a
> > commandline Maven action (such as generate-sources) one might have a 
> > benefit.
> >
> > So in the day-to-day life the caching might not be as beneficial for
> > developers, but commandline builds happen often enough to make this
> > matter.
> >
> > Martijn
> >
> > 
> > - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For
> > additional commands, e-mail: dev-h...@maven.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For
> additional commands, e-mail: dev-h...@maven.apache.org
>
>


---
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to https://www.db.com/disclosures for additional EU corporate and 
regulatory disclosures and to 
http://www.db.com/unitedkingdom/content/privacy.htm for information about 
privacy.

Re: [VOTE] Maven incremental build for BIG-sized projects with local and remote caching

[Alexander Bubenchikov]

Hi Alexander and Maximilian. As a former Deutsche Bank RTC employee, I
cannot bypass this topic.

First of all, I'm not speaking for community, neither Jetbrains.
I know how painfull is to get all approvals for open-source contribution in
Deutsche Bank, but may be you can share you solution as a maven extension
first?
May be I've missed link to github, but discussion looks strange for me - we
are discussing pros and cons of solution, which is not public yet.

>so any ide with maven integration should benefit from cache them maven
action invoked
AFAIK, m2e use its own lifecycle (please, correct me, if I'm wrong). Not
sure, how this change will affect it's performance.

But IDEA executes maven tasks (actually, in a quite hacky-way) for
resolving dependencies, source folders, artifacts, etc.  Compiler plugin is
not being runned usually by IDEA

Compilation and tests runs of imported maven projects are performed by IDEA
itself, and IDEA does not use maven compilation results when run tests. No
maven code and maven extensions are used after importing, so for IDEA this
change should be irrelevant, as I got it (the only exception  case is if
you use "delegate maven" option, but this is a point of another discussion)

Thanks, Alex.



On Thu, Sep 19, 2019 at 8:48 AM Alexander Ashitkin 
wrote:

> Sorry if duplicated, looks like my yesterday reply didn't come through.
> Sharing results.
>
> Configuration:
> * verify -T4 -P default,all-shapshots-repos
> * my project config (might be suboptimal for wicket)
> * scala tests disabled in 2 modules (caused bytecode version conflict on
> my machine)
>
> Results
> Clean state (cache disabled):   15:58 min
> Second run, target up to date (cache disabled):  10:20 min
> Fully cached (no changes):  17.507 s
> wicketstuff-jwicket-tooltip-wtooltips changed:  34.936 s
> wicketstuff-rest-utils changed: 54.040 s
>
>
> For wicketstuff-jwicket-tooltip-wtooltips i didnt check invalidated
> modules, for wicketstuff-rest-utils
>  [wicketstuff-rest-lambda, wicketstuff-restannotations,
> wicketstuff-restannotations-json, wicketstuff-restannotations-examples]
> were invalidated and rebuilt
>
> If you want to try other modules - please let me know.
>
> regarding ide - it's a usual maven installation, so any ide with maven
> integration should benefit from cache them maven action invoked
>
> Thank you
> Aleks
>
>
> On 2019/09/17 12:29:11, Martijn Dashorst 
> wrote:
> > This seems like it would benefit a lot of projects (at least it would
> ours).
> >
> > How would this work in coordination with IDE's? m2e has (afaict, but
> > haven't looked closely) its own lifecycle management to bridge eclipse
> and
> > maven. AFIAK only Netbeans uses maven directly?
> >
> > If you want to benchmark a public big repo, you can use Wicket Stuff
> Core (
> > https://github.com/wicketstuff/core). It has 237 modules, and the build
> > takes quite a while to compile and package. The project levels are not
> > deep, but there's some nesting.
> >
> > Martijn
> >
> >
> > On Fri, Sep 13, 2019 at 10:46 PM Maximilian Novikov <
> > maximilian.novi...@db.com> wrote:
> >
> > > Hi All,
> > >
> > >
> > >
> > > *We want to create upstream change to Maven* to support true
> incremental
> > > build for big-sized projects.
> > >
> > > To raise a pull request we have to pass long chain of Deutsche Bank’s
> > > internal procedures. So, *before starting the process we would like to
> > > get your feedback regarding this feature*.
> > >
> > >
> > >
> > > *Motivation:*
> > >
> > >
> > >
> > > Our project is hosted in mono-repo and contains ~600 modules. All
> modules
> > > has the same SNAPSHOT version.
> > >
> > > There are lot of test automation around this, everything is tested
> before
> > > merge into release branch.
> > >
> > >
> > >
> > > Current setup helps us to simplify build/release/dependency management
> for
> > > 10+ teams those contribute into codebase. We can release everything in
> > > 1-click.
> > >
> > > The major drawback of such approach is build time: *full local build
> took
> > > 45-60 min (*-T8)*, CI build ~25min(*-T16*)*.
> > >
> > >
> > >
> > > To speed-up our build we needed 2 features: incremental build and
> shared
> > > cache.
> > >
> > > Initially we started to think about migration to Gradle or Bazel. As
> > > migration costs for the mentioned tools were too high, we decided to
> add
> > > similar functionality into Maven.
> > >
> > >
> > >
> > > Current results we get: *1-2 mins for local build(*-T8*)* if build was
> > > cached by CI*, CI build ~5 mins (*-T16*).*
> > >
> > >
> > >
> > > *Feature description:*
> > >
> > >
> > >
> > > The idea is to calculate checksum for inputs and save outputs in cache.
> > >
> > > [image: image2019-8-27_20-0-14.png]
> > >
> > > Each node checksum calculated with:
> > >
> > >
> > >
> > > · Effective POM hash
> > >
> > > · Sources hash
> > >
> > > · 

Re: Reproducible Builds for Maven

[Emmanuel Bourg]

Le 23/09/2019 à 01:52, Hervé BOUTEMY a écrit :
> after a few years of testing, thinking, procrastination and hard work (thank 
> you Thomas for your talk at Devoxx France 2016 [1]), I think I achieved a key 
> step this week-end toward native Reproducible Builds with Maven [2]: Maven 
> core itself can be built in a reproducible way!

An important milestone on a very long journey. Well done!

Emmanuel Bourg

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org