Re: Reproducible builds: a maven thing or not?

[Christoph Läubrich]

Tycho has a BuildTimestampProvider [1] that with implementation of 
several flavors:


- maven build timestamp (default)
- fixed timestamps
- jgit
- inherit from parent
- ...

So if maven would offer something similar a build extension can be used 
to implement any strategy but still be there some default choices in 
maven directly.


[1] 
https://github.com/eclipse-tycho/tycho/blob/main/tycho-spi/src/main/java/org/eclipse/tycho/build/BuildTimestampProvider.java


Am 01.04.24 um 20:25 schrieb Romain Manni-Bucau:

Hi all,

We've got a lot of work around reproducible builds - whatever we think
about it since there are good reasons to say it is pointless and some to
say it is useful so I don't want to enter into the debate - but the notion
is 100% hosted in plugins today.
It has a few pitfalls for me:

1. plugins assume the (output) timestamp is static for a build and rebuild
2. projects often need a build timestamp related to last modifications
(frontends, jar.lastModifiedTime usage, ...)

Both statements don't go well with a hardcoded value - manually
maintainedso not maintained and broken IRL.

A workaround used by some projects is to wire the git last commit time - or
equivalent - to this value but then a plugin injects the value which means
another plugin can read another value or no value and corrupt the
reproducibility of the build.

Related to that, the work around the PR #30 on maven-artifact-plugin shows
that if using a dynamic value the artifact plugin fails or at least emits a
warning - which is not acceptable for a valid case IMHO.

Originally I just thought about relaxing this warning but Hervé thinks it
is wrong and due to the limitations mentionned earlier I think he is right
too but this means we can't let a plugin handle the "output timestamp" so I
wonder if we should move it to the new maven 4 service layer.

The only difficulty is to let it have some pluggability, default would be a
hardcoded value in properties but we should be able to use git last commit
time - likely reading pack without jgit dependency - or another strategy
potentially (ideally, as pointed out by Hervé, a different value per module
would be highly relevant and could need some configuration, the common
example being: "use the last modified time from git of files in src", ie
ignore pom modifications since it does not end in the output for example).

So my question is the following one: do we add output timestamp to maven 4
API and let maven 3 like that or do we consider reproducible builds are not
a key thing for maven and we let it in plugins (~=outside and artifact
plugin is considered not a core plugin - not sure of the status today)?

I'm happy both ways but I'm way less happy with a broken setup if we
consider it is supported by maven.

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book




-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Reproducible Builds for Maven

[Mark Derricutt]

On 28 Sep 2019, at 18:21, Hervé BOUTEMY wrote:

> I'll share shortly a discussion on a choice we need to do together to define 
> how to configure reproducible builds (property name and value/format of 
> current source-date-epoch defined in PoC)

Now that support for this has been released as part of the source/jar plugins, 
but not yet the release plugin, I whipped up this morning a quick plugin to use 
in the meantime:

  https://github.com/talios/reproducible-maven-plugin

Basically to be included in preparationGoals/completionGoals using the 
apply/clear goal in each.

So far based on very quick, rudimentary tests this seems to work well (altho 
I've not yet done it as part of a release, but using apply and then 
building/checking my jars.

I'd be keen on any feedback...

Interestingly, whilst the class files INSIDE the jar have the correct date, the 
jar this has todays - which makes sense, but I wonder if it shouldn't be the 
same time for one extra step of "the exact same output"...

Mark



---
"The ease with which a change can be implemented has no relevance at all to 
whether it is the right change for the (Java) Platform for all time."  
Mark Reinhold.

Mark Derricutt
http://www.theoryinpractice.net
http://www.chaliceofblood.net
http://plus.google.com/+MarkDerricutt
http://twitter.com/talios
http://facebook.com/mderricutt


signature.asc
Description: OpenPGP digital signature

Re: Reproducible Builds for Maven

[Hervé BOUTEMY]

last updates:
- tar.gz archives are now also reproducible (in addition to .zip)
- src archives are also built and reproducible (notice that the result is the 
same on every JDK version of a platform. Notice 2: if you don't get the same 
result than CI, check that you don't have IDE configuration files that went 
into your local source archives...)
- artifacts built on ASF CI are available, for people to download and compare 
if you get a different result:
https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/reproducible/lastSuccessfulBuild/artifact/org/apache/maven/apache-maven/3.6.3-SNAPSHOT/

I'll share shortly a discussion on a choice we need to do together to define 
how to configure reproducible builds (property name and value/format of current 
source-date-epoch defined in PoC)

Once this decision is made, we can start release packaging plugins that support 
"native" reproducible builds
https://reproducible-builds.org/

Regards,

Hervé

Le lundi 23 septembre 2019, 01:52:48 CEST Hervé BOUTEMY a écrit :
> after a few years of testing, thinking, procrastination and hard work (thank
> you Thomas for your talk at Devoxx France 2016 [1]), I think I achieved a
> key step this week-end toward native Reproducible Builds with Maven [2]:
> Maven core itself can be built in a reproducible way!
> 
> It means that if you build "reproducible" branch of Maven core, you'll get
> the same apache-maven-3.6.3-SNAPSHOT-bin.zip than me or the ASF CI server
> [3]. The precise result depends only on 2 key facts:
> - do you build on Windows or any Unix? This impacts newlines...
> - what JDK major version do you use to build? This affects generated .class
> (notice: AFAIK minor JDK version does not have any impact, nor platform)
> 
> This branch is only a PoC: it uses unreleased packaging plugins that give
> reproducible results (versions in .RB-SNAPSHOT), and I had to tweak a
> little bit the build for remaining reproduciblity issues with sisu and
> plexus plugins. There are many details to decide before releasing these
> plugins and making every build reproducible by default. But the current
> steps proves that is is feasible.
> 
> Interested in joining the effort to bring this feature to releases for end
> users?
> 
> Regards,
> 
> Hervé
> 
> 
> [1]
> http://zlika.github.io/presentations/devoxx_fr_2016/reproducible-builds/sli
> des_fr.html
> 
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
> 
> [3]
> https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/r
> eproducible/
> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org





-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Reproducible Builds for Maven

[Hervé BOUTEMY]

Le mardi 24 septembre 2019, 02:28:15 CEST Mark Derricutt a écrit :
> Tomo Suzuki wrote on 23/09/19 3:56 PM:
> > Does your approach use such file to record library versions?
> 
> I don't know about what Hervé is doing,
I added an "out of scope" paragraph: managing version ranges in a stable way is 
not in the scope
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318#Reproducible/VerifiableBuilds-Outofscope


> but internally we have a tool I
> wrote for handling this, we have a pom.deps file that looks like:
nice separation of concerns: stable versions chosen vs updates controlled with 
ranges

with such an approach, version ranges could become something I love :)

Regards,

Hervé

> 
>  repository http://nexus.XX as public;
> 
>  import smx3:smx3.upstream.bill-of-materials:1.1.22;
> 
>  resolve highest org.jetbrains:annotations:[16.0.3,17.0.0) via public;
>  resolve highest
> org.apache.maven.plugins:maven-jar-plugin:[3.1.2,4.0.0) via public;
>  resolve highest org.apache.cxf:cxf-codegen-plugin:[3.3.3,4.0.0) via
> public;
> 
> which when we resolve, will find the highest, snapshot, or lowest
> version in a given range - also allowing filtering out annoying things
> like beta/alpha/CR from central, and rewriting the pom.xml's.
> 
> Our tooling also has an 'import' option shown above that lets us
> standardize the versions we resolving, and breaking it up - so we have
> 'upstream.bill-of-materials' and 'upstream.testing.bill-of-materials`.
> 
> As part of this we also add in  to ban all transitive build
> deps, and [] range all version references.
> 
> I keep meaning to push for open sourcing it, but just haven't had the time.
> 
> Mark





-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Reproducible Builds for Maven

[Tomo Suzuki]

Cool. Thanks.

Re: Reproducible Builds for Maven

[Mark Derricutt]

On 24 Sep 2019, at 23:37, Tomo Suzuki wrote:

> versions, rather than ranges. Would you share the background why your tool
> records the ranges?

The full examples is at:

  https://github.com/HalBuilder/halbuilder-support-4.x/blob/master/pom.deps

It resolves the locked down versions, but also retains the desired ranges for 
controlled updates.

We tend to keep ranges between major versions, i.e. [1.0.0,2.0.0) for a 
semblance of semver.

When I reresolve the bill of materials, I find I'll often look at the git diff 
and see what new versions of libraries have been updated, and decide which ( 
and when ) we pull them in to use - often committing those changes individually.



---
"The ease with which a change can be implemented has no relevance at all to 
whether it is the right change for the (Java) Platform for all time."  
Mark Reinhold.

Mark Derricutt
http://www.theoryinpractice.net
http://www.chaliceofblood.net
http://plus.google.com/+MarkDerricutt
http://twitter.com/talios
http://facebook.com/mderricutt


signature.asc
Description: OpenPGP digital signature

Re: Reproducible Builds for Maven

[Mark Derricutt]

Oh right! I missed the second half of our pom.deps file:

   blacklisted org.hibernate:hibernate-ehcache:4.2.19.Final from 
smx3:smx3.bill-of-materials:2.1.1;
   deprecated org.hibernate:hibernate-search-engine:4.4.0.Final from 
smx3:smx3.bill-of-materials:2.1.1;
   resolved org.hibernate:hibernate-search-orm:4.4.0.Final from 
smx3:smx3.bill-of-materials:2.1.1;
   locked org.hibernate:hibernate-validator:4.3.1.Final from 
smx3:smx3.upstream.bill-of-materials:1.0.16;


I provide 4 types of resolution, deprecated/blacklisted means the build 
will fail, unless you specifically opt-in to deprecated/blacklisted 
dependencies.


On resolve, anything that's changed moves from `locked` to either 
resolved/deprecated/blacklisted, and on release anything resolved goes 
to locked.


In the end user projects, we "import" the bill of materials .deps 
artifact which pulls in any locked version, but I also have support for 
adding:


   allow unlocked /^.*someregex.*$/;

which will trigger a re-resolve, which we use for our own internal 
artifacts, so we pick up the latest internal releases IF we re-resolve.


The whole process seems to work well, it does cause some annoying round 
trip releases or quirks now and then, but for the most part - it's 
really solved a heap of issues where we've needed to back a quick 
backport fix in a production environment.


Since we publish the pom.deps file for each artifact, when doing the 
backports, we simply import the deps used in the distribution version 
we're patching, and manually allow any updated deps that we need to fix.


Tomo Suzuki wrote:


For reproducible builds, I expected the lock file contains specific
versions, rather than ranges. Would you share the background why your tool
records the ranges?

Re: Reproducible Builds for Maven

[Tomo Suzuki]

Hi Mark,

Thank you for response.

> resolve highest org.jetbrains:annotations:[16.0.3,17.0.0) via public;

For reproducible builds, I expected the lock file contains specific
versions, rather than ranges. Would you share the background why your tool
records the ranges?


-- 
Regards,
Tomo

Re: Reproducible Builds for Maven

[Mark Derricutt]

Tomo Suzuki wrote on 23/09/19 3:56 PM:

Does your approach use such file to record library versions?
I don't know about what Hervé is doing, but internally we have a tool I 
wrote for handling this, we have a pom.deps file that looks like:


    repository http://nexus.XX as public;

    import smx3:smx3.upstream.bill-of-materials:1.1.22;

    resolve highest org.jetbrains:annotations:[16.0.3,17.0.0) via public;
    resolve highest 
org.apache.maven.plugins:maven-jar-plugin:[3.1.2,4.0.0) via public;
    resolve highest org.apache.cxf:cxf-codegen-plugin:[3.3.3,4.0.0) via 
public;


which when we resolve, will find the highest, snapshot, or lowest 
version in a given range - also allowing filtering out annoying things 
like beta/alpha/CR from central, and rewriting the pom.xml's.


Our tooling also has an 'import' option shown above that lets us 
standardize the versions we resolving, and breaking it up - so we have 
'upstream.bill-of-materials' and 'upstream.testing.bill-of-materials`.


As part of this we also add in  to ban all transitive build 
deps, and [] range all version references.


I keep meaning to push for open sourcing it, but just haven't had the time.

Mark


--
Sent from Postbox 

Re: Reproducible Builds for Maven

[Emmanuel Bourg]

Le 23/09/2019 à 01:52, Hervé BOUTEMY a écrit :
> after a few years of testing, thinking, procrastination and hard work (thank 
> you Thomas for your talk at Devoxx France 2016 [1]), I think I achieved a key 
> step this week-end toward native Reproducible Builds with Maven [2]: Maven 
> core itself can be built in a reproducible way!

An important milestone on a very long journey. Well done!

Emmanuel Bourg

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Reproducible Builds for Maven

[Hervé BOUTEMY]

Le lundi 23 septembre 2019, 05:56:06 CEST Tomo Suzuki a écrit :
> Sounds nice!
don't hesitate to build for yourself, check that you get the same sha512 and 
report: this will help me either confirm "it works", or find little remaining 
issues.

> 
> > The precise result depends only on 2 key facts
> 
> When I hear “reproducible builds”, I think of  lock files that remember
> library versions used.
> Gradle’s approach:
> https://docs.gradle.org/current/userguide/dependency_locking.html
> 
> Does your approach use such file to record library versions?
no, we don't need such a lock file since we don't use version ranges: the 
dependency resolution is already stable

Here, "Reproducible builds are a set of software development practices that 
create an independently-verifiable path from source to binary code."
see https://reproducible-builds.org/

For Java, one key non-reproducible aspect for example is the timestamp of zip 
entries in jar files.

Regards,

Hervé 

> 
> Regards,
> Tomo





-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Reproducible Builds for Maven

[Tomo Suzuki]

Sounds nice!

> The precise result depends only on 2 key facts

When I hear “reproducible builds”, I think of  lock files that remember
library versions used.
Gradle’s approach:
https://docs.gradle.org/current/userguide/dependency_locking.html

Does your approach use such file to record library versions?

Regards,
Tomo



-- 
Regards,
Tomo