Re: [DISCUSS] configuration for Reproducible Builds
Hervé When will you set this value? During release:prepare and modify the pom? Enrico Il sab 28 set 2019, 17:55 Hervé BOUTEMY ha scritto: > Achieving Reproducible Builds require only one parameter: plugins that > create > zip or tar archives require a fixed timestamp for entries > > Putting that parameter as a pom property with a well known name and value > format permits to share the configuration between every packaging plugin. > This also has the advantage that child poms will inherit from parent > value, > and eventually override. > > The question is: *what property name and what value format should we keep?* > > For the PoC, I chose to extrapolate from a convention from Reproducible > Builds > project, which is very Linux-oriented: SOURCE_DATE_EPOCH environment > variable, > that I transformed into source-date-epoch property name, keeping the "date > + > %s" value > https://reproducible-builds.org/docs/source-date-epoch/ > > > But I feel we can do a more user-readable solution by choosing another > name > and format, like "reproducible-build-timestamp" with an ISO-8601 combined > date > and time representation > > > WDYT? Any other idea? > > Regards, > > Hervé > > > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > >
Re: [DISCUSS] Maven 3.7.0
On Sat, 28 Sep 2019 16:53:00 +0200, Enrico Olivelli wrote: Robert, Il sab 28 set 2019, 14:04 Robert Scholte ha scritto: Hi, TLDR; introduce maven.experimental.buildconsumer and push Java requirement to Java 8 now that Maven 3.6.2 is out for a couple of weeks, it seems like we didn't face real regressions. The only one might be tricky is the issue related to Tycho. However, I think we're ready to push Maven to the next level. For those actively reading this list, they should recognize the need for splitting up the pom as it is on the local system versus the pom being uploaded. Once we truly control this mechanism we can think of improvements on model 5.0.0 and new fileformats. I've created and implemented MNG-6656[1]. It also contains a zip with an example (original, patched, README) to understand what's happening. This is really cool, I hope we get something like this very soon. One overall comment from me is about using XML and particularly SAX. We will have our Maven XML library but the core principle is that all of the transformations are in a streaming fashion, there is no overall view of the whole document, and you cannot go backward and you can't see the tags after the current point. SAX is more memory efficient but if this will be a base for the future we should take into account future needs. The choice for using XMLFilters is to be able to keep order of elements and also keep the comments (assuming the distributed pom is still an important source of information, otherwise we could decide to just recreate a new pom). Validating *after* the resolved pom (phase 2, but before inheritence) will probably need extra care, since there might be an issue with the input location. So please have a good look. As said, it will only be activated with the special flag and we simply need to start somewhere. It is all about collection first experiences, see what works and what doesn't. Robert I will review carefully the patch when the approach is agreed by the community. I have already taken a first look, if you create the pull requests I can add comments Enrico In order to make this successful, we need IDEs and CI Servers to understand and support these changes. The likely need to implement one of the interfaces[2]. The new interface uses Java8 Functions (and especially SAXEventFactory is way easier to read+maintain with Java 8). I've tried to keep Maven Java 7 compatible, but that was too hard to do. So I'd like to use this opportunity to move Maven forward and start requiring Java 8. There are some other improvements I'd like to add (those messages will follow), so this will imply that it will take some time before we do a new release. WDTY, Robert [1] https://issues.apache.org/jira/browse/MNG-6656 [2] https://github.com/apache/maven/compare/MNG-6656?expand=1 - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] Maven 3.7.0
Hi Mickael, On 28.09.19 17:37, Mickael Istria wrote: On Sat, Sep 28, 2019 at 5:35 PM Karl Heinz Marbaise wrote: now that Maven 3.6.2 is out for a couple of weeks, it seems like we didn't face real regressions. The only one might be tricky is the issue related to Tycho. Feedback of Michael Istria states different? Or do I miss a thing? I think we should trust the various users who face this issue, and assume the issue exist until proven otherwise. This is likely a bug in Tycho and/or Polyglot Maven, and I believe this reveals that some of the important Tycho tests are not performed against latest Maven snapshots. I've started a thread on the tycho-...@eclipse.org mailing-list on htis topic. Ah Ok...now I understand your post on tycho-dev list.. Thanks that explains it... Of course we should wait for the feedback.. Kind regards Karl Heinz Marbaise - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] Maven 3.7.0
did someone confirm that it is related to Plexus to Tycho switch in MNG-6685? Regards, Hervé Le samedi 28 septembre 2019, 16:42:27 CEST Robert Scholte a écrit : > https://issues.apache.org/jira/browse/MNG-6765 > > I guess it is more about the pom-less part than the tycho-part. > > On Sat, 28 Sep 2019 15:55:28 +0200, Mickael Istria > > wrote: > > On Sat, Sep 28, 2019 at 2:04 PM Robert Scholte > > > > wrote: > >> The only one might be tricky is the issue related to Tycho. > > > > What issue is this? Tycho integration-tests are being run against Maven > > snapshots daily and no issue was spot nor report on Tycho side as far as > > I > > know. > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
[DISCUSS] configuration for Reproducible Builds
Achieving Reproducible Builds require only one parameter: plugins that create zip or tar archives require a fixed timestamp for entries Putting that parameter as a pom property with a well known name and value format permits to share the configuration between every packaging plugin. This also has the advantage that child poms will inherit from parent value, and eventually override. The question is: *what property name and what value format should we keep?* For the PoC, I chose to extrapolate from a convention from Reproducible Builds project, which is very Linux-oriented: SOURCE_DATE_EPOCH environment variable, that I transformed into source-date-epoch property name, keeping the "date + %s" value https://reproducible-builds.org/docs/source-date-epoch/ But I feel we can do a more user-readable solution by choosing another name and format, like "reproducible-build-timestamp" with an ISO-8601 combined date and time representation WDYT? Any other idea? Regards, Hervé - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] Maven 3.7.0
On Sat, Sep 28, 2019 at 5:35 PM Karl Heinz Marbaise wrote: > > > now that Maven 3.6.2 is out for a couple of weeks, it seems like we > > didn't face real regressions. > > The only one might be tricky is the issue related to Tycho. > > Feedback of Michael Istria states different? Or do I miss a thing? I think we should trust the various users who face this issue, and assume the issue exist until proven otherwise. This is likely a bug in Tycho and/or Polyglot Maven, and I believe this reveals that some of the important Tycho tests are not performed against latest Maven snapshots. I've started a thread on the tycho-...@eclipse.org mailing-list on htis topic.
Re: [DISCUSS] Maven 3.7.0
Hi, On 28.09.19 14:05, Robert Scholte wrote: Hi, TLDR; introduce maven.experimental.buildconsumer and push Java requirement to Java 8 now that Maven 3.6.2 is out for a couple of weeks, it seems like we didn't face real regressions. The only one might be tricky is the issue related to Tycho. Feedback of Michael Istria states different? Or do I miss a thing? However, I think we're ready to push Maven to the next level. Yes that's very important to go that step... For those actively reading this list, they should recognize the need for splitting up the pom as it is on the local system versus the pom being uploaded. Once we truly control this mechanism we can think of improvements on model 5.0.0 and new fileformats. Yes that will open up several parts we are thinking about for a long time...(Build vs. Consumer POM). I've created and implemented MNG-6656[1]. It also contains a zip with an example (original, patched, README) to understand what's happening. In order to make this successful, we need IDEs and CI Servers to understand and support these changes. The likely need to implement one of the interfaces[2]. The new interface uses Java8 Functions (and especially SAXEventFactory is way easier to read+maintain with Java 8). I've tried to keep Maven Java 7 compatible, but that was too hard to do. This is a waste of time simply ... So I'd like to use this opportunity to move Maven forward and start requiring Java 8. It's really time to get up to Java 8 at minimum There are some other improvements I'd like to add (those messages will follow), so this will imply that it will take some time before we do a new release. Great go forward for Maven 3.7.0... Kind regards Karl Heinz Marbaise - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] Maven 3.7.0
Robert, Il sab 28 set 2019, 14:04 Robert Scholte ha scritto: > Hi, > > TLDR; introduce maven.experimental.buildconsumer and push Java > requirement > to Java 8 > > now that Maven 3.6.2 is out for a couple of weeks, it seems like we > didn't > face real regressions. > The only one might be tricky is the issue related to Tycho. > > However, I think we're ready to push Maven to the next level. > > For those actively reading this list, they should recognize the need for > splitting up the pom as it is on the local system versus the pom being > uploaded. Once we truly control this mechanism we can think of > improvements on model 5.0.0 and new fileformats. > > I've created and implemented MNG-6656[1]. It also contains a zip with an > example (original, patched, README) to understand what's happening. > This is really cool, I hope we get something like this very soon. One overall comment from me is about using XML and particularly SAX. We will have our Maven XML library but the core principle is that all of the transformations are in a streaming fashion, there is no overall view of the whole document, and you cannot go backward and you can't see the tags after the current point. SAX is more memory efficient but if this will be a base for the future we should take into account future needs. I will review carefully the patch when the approach is agreed by the community. I have already taken a first look, if you create the pull requests I can add comments Enrico > In order to make this successful, we need IDEs and CI Servers to > understand and support these changes. The likely need to implement one of > the interfaces[2]. > The new interface uses Java8 Functions (and especially SAXEventFactory is > way easier to read+maintain with Java 8). I've tried to keep Maven Java 7 > compatible, but that was too hard to do. > So I'd like to use this opportunity to move Maven forward and start > requiring Java 8. > > There are some other improvements I'd like to add (those messages will > follow), so this will imply that it will take some time before we do a > new > release. > > WDTY, > Robert > > [1] https://issues.apache.org/jira/browse/MNG-6656 > [2] https://github.com/apache/maven/compare/MNG-6656?expand=1 > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > >
Re: [DISCUSS] Maven 3.7.0
https://issues.apache.org/jira/browse/MNG-6765 I guess it is more about the pom-less part than the tycho-part. On Sat, 28 Sep 2019 15:55:28 +0200, Mickael Istria wrote: On Sat, Sep 28, 2019 at 2:04 PM Robert Scholte wrote: The only one might be tricky is the issue related to Tycho. What issue is this? Tycho integration-tests are being run against Maven snapshots daily and no issue was spot nor report on Tycho side as far as I know. - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [DISCUSS] Maven 3.7.0
On Sat, Sep 28, 2019 at 2:04 PM Robert Scholte wrote: > The only one might be tricky is the issue related to Tycho. What issue is this? Tycho integration-tests are being run against Maven snapshots daily and no issue was spot nor report on Tycho side as far as I know.
Re: [DISCUSS] Maven 3.7.0
Hello guys, For the user community these two issues are important: https://issues.apache.org/jira/browse/MNG-6169 https://issues.apache.org/jira/browse/MNG-6548 The Tycho project is the user as well. The J8 is internal code improvement/change => lower priority than the user's priority => release order/priorities/dedicated time spent in development. Have a nice day. Cheers Tibor17 On Sat, Sep 28, 2019 at 2:08 PM Gary Gregory wrote: > I would say that fixing the Tycho issue comes first. > > Gary > > On Sat, Sep 28, 2019 at 8:04 AM Robert Scholte > wrote: > > > Hi, > > > > TLDR; introduce maven.experimental.buildconsumer and push Java > > requirement > > to Java 8 > > > > now that Maven 3.6.2 is out for a couple of weeks, it seems like we > > didn't > > face real regressions. > > The only one might be tricky is the issue related to Tycho. > > > > However, I think we're ready to push Maven to the next level. > > > > For those actively reading this list, they should recognize the need for > > splitting up the pom as it is on the local system versus the pom being > > uploaded. Once we truly control this mechanism we can think of > > improvements on model 5.0.0 and new fileformats. > > > > I've created and implemented MNG-6656[1]. It also contains a zip with an > > example (original, patched, README) to understand what's happening. > > > > In order to make this successful, we need IDEs and CI Servers to > > understand and support these changes. The likely need to implement one of > > the interfaces[2]. > > The new interface uses Java8 Functions (and especially SAXEventFactory is > > way easier to read+maintain with Java 8). I've tried to keep Maven Java 7 > > compatible, but that was too hard to do. > > So I'd like to use this opportunity to move Maven forward and start > > requiring Java 8. > > > > There are some other improvements I'd like to add (those messages will > > follow), so this will imply that it will take some time before we do a > > new > > release. > > > > WDTY, > > Robert > > > > [1] https://issues.apache.org/jira/browse/MNG-6656 > > [2] https://github.com/apache/maven/compare/MNG-6656?expand=1 > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > For additional commands, e-mail: dev-h...@maven.apache.org > > > > >
Re: [DISCUSS] Maven 3.7.0
I would say that fixing the Tycho issue comes first. Gary On Sat, Sep 28, 2019 at 8:04 AM Robert Scholte wrote: > Hi, > > TLDR; introduce maven.experimental.buildconsumer and push Java > requirement > to Java 8 > > now that Maven 3.6.2 is out for a couple of weeks, it seems like we > didn't > face real regressions. > The only one might be tricky is the issue related to Tycho. > > However, I think we're ready to push Maven to the next level. > > For those actively reading this list, they should recognize the need for > splitting up the pom as it is on the local system versus the pom being > uploaded. Once we truly control this mechanism we can think of > improvements on model 5.0.0 and new fileformats. > > I've created and implemented MNG-6656[1]. It also contains a zip with an > example (original, patched, README) to understand what's happening. > > In order to make this successful, we need IDEs and CI Servers to > understand and support these changes. The likely need to implement one of > the interfaces[2]. > The new interface uses Java8 Functions (and especially SAXEventFactory is > way easier to read+maintain with Java 8). I've tried to keep Maven Java 7 > compatible, but that was too hard to do. > So I'd like to use this opportunity to move Maven forward and start > requiring Java 8. > > There are some other improvements I'd like to add (those messages will > follow), so this will imply that it will take some time before we do a > new > release. > > WDTY, > Robert > > [1] https://issues.apache.org/jira/browse/MNG-6656 > [2] https://github.com/apache/maven/compare/MNG-6656?expand=1 > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > >
[DISCUSS] Maven 3.7.0
Hi, TLDR; introduce maven.experimental.buildconsumer and push Java requirement to Java 8 now that Maven 3.6.2 is out for a couple of weeks, it seems like we didn't face real regressions. The only one might be tricky is the issue related to Tycho. However, I think we're ready to push Maven to the next level. For those actively reading this list, they should recognize the need for splitting up the pom as it is on the local system versus the pom being uploaded. Once we truly control this mechanism we can think of improvements on model 5.0.0 and new fileformats. I've created and implemented MNG-6656[1]. It also contains a zip with an example (original, patched, README) to understand what's happening. In order to make this successful, we need IDEs and CI Servers to understand and support these changes. The likely need to implement one of the interfaces[2]. The new interface uses Java8 Functions (and especially SAXEventFactory is way easier to read+maintain with Java 8). I've tried to keep Maven Java 7 compatible, but that was too hard to do. So I'd like to use this opportunity to move Maven forward and start requiring Java 8. There are some other improvements I'd like to add (those messages will follow), so this will imply that it will take some time before we do a new release. WDTY, Robert [1] https://issues.apache.org/jira/browse/MNG-6656 [2] https://github.com/apache/maven/compare/MNG-6656?expand=1 - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Reproducible Builds for Maven
last updates: - tar.gz archives are now also reproducible (in addition to .zip) - src archives are also built and reproducible (notice that the result is the same on every JDK version of a platform. Notice 2: if you don't get the same result than CI, check that you don't have IDE configuration files that went into your local source archives...) - artifacts built on ASF CI are available, for people to download and compare if you get a different result: https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/reproducible/lastSuccessfulBuild/artifact/org/apache/maven/apache-maven/3.6.3-SNAPSHOT/ I'll share shortly a discussion on a choice we need to do together to define how to configure reproducible builds (property name and value/format of current source-date-epoch defined in PoC) Once this decision is made, we can start release packaging plugins that support "native" reproducible builds https://reproducible-builds.org/ Regards, Hervé Le lundi 23 septembre 2019, 01:52:48 CEST Hervé BOUTEMY a écrit : > after a few years of testing, thinking, procrastination and hard work (thank > you Thomas for your talk at Devoxx France 2016 [1]), I think I achieved a > key step this week-end toward native Reproducible Builds with Maven [2]: > Maven core itself can be built in a reproducible way! > > It means that if you build "reproducible" branch of Maven core, you'll get > the same apache-maven-3.6.3-SNAPSHOT-bin.zip than me or the ASF CI server > [3]. The precise result depends only on 2 key facts: > - do you build on Windows or any Unix? This impacts newlines... > - what JDK major version do you use to build? This affects generated .class > (notice: AFAIK minor JDK version does not have any impact, nor platform) > > This branch is only a PoC: it uses unreleased packaging plugins that give > reproducible results (versions in .RB-SNAPSHOT), and I had to tweak a > little bit the build for remaining reproduciblity issues with sisu and > plexus plugins. There are many details to decide before releasing these > plugins and making every build reproducible by default. But the current > steps proves that is is feasible. > > Interested in joining the effort to bring this feature to releases for end > users? > > Regards, > > Hervé > > > [1] > http://zlika.github.io/presentations/devoxx_fr_2016/reproducible-builds/sli > des_fr.html > > [2] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 > > [3] > https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/r > eproducible/ > > > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Reproducible Builds for Maven
Le mardi 24 septembre 2019, 02:28:15 CEST Mark Derricutt a écrit : > Tomo Suzuki wrote on 23/09/19 3:56 PM: > > Does your approach use such file to record library versions? > > I don't know about what Hervé is doing, I added an "out of scope" paragraph: managing version ranges in a stable way is not in the scope https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318#Reproducible/VerifiableBuilds-Outofscope > but internally we have a tool I > wrote for handling this, we have a pom.deps file that looks like: nice separation of concerns: stable versions chosen vs updates controlled with ranges with such an approach, version ranges could become something I love :) Regards, Hervé > > repository http://nexus.XX as public; > > import smx3:smx3.upstream.bill-of-materials:1.1.22; > > resolve highest org.jetbrains:annotations:[16.0.3,17.0.0) via public; > resolve highest > org.apache.maven.plugins:maven-jar-plugin:[3.1.2,4.0.0) via public; > resolve highest org.apache.cxf:cxf-codegen-plugin:[3.3.3,4.0.0) via > public; > > which when we resolve, will find the highest, snapshot, or lowest > version in a given range - also allowing filtering out annoying things > like beta/alpha/CR from central, and rewriting the pom.xml's. > > Our tooling also has an 'import' option shown above that lets us > standardize the versions we resolving, and breaking it up - so we have > 'upstream.bill-of-materials' and 'upstream.testing.bill-of-materials`. > > As part of this we also add in to ban all transitive build > deps, and [] range all version references. > > I keep meaning to push for open sourcing it, but just haven't had the time. > > Mark - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
