Hi Karl, Thanks very much for your reply!
> Not only for displaying...you can also use it to update them... Understood. But I actually don't want to always auto-update everything to latest releases. The point of the BOM is that the versions it references have been tested to work together. Especially for things like major breaking releases of third party components, we need to do those updates manually and carefully. > Maybe I misunderstand a thing here but the first part can be answered > by using your version control? Make a diff between latest release tag > and the current trunk/master ? If a component needs a release is > something which only can being answered by a human... Right. That is actually what our tooling was doing until now. However, the process is laborious and slow. For each of our ~200 first-party components whose GAVs are listed in the BOM, the tooling needs to: - Fetch the POM for the GAV in question. - Extract the <scm> section from the (effective) POM. - Clone the indicated SCM locally (in our case: always a Git repository from GitHub). - Do the relevant git command(s) to decide whether master has really changed since the last release. It's the cloning step that is especially expensive here, even if you limit cloning depth. I came up with a (I hope) much simpler solution which does not rely on the SCM at all, but only on Maven and our remote repository manager: 1) Extract the component's versions/release tag from maven-metadata.xml of the relevant GA in the remote MRM. 2) Check the timestamp of that release's POM artifact in the remote MRM. 3) Extract the versions/lastUpdated tag from maven-metadata.xml of the relevant GA in the remote MRM. 4) Compare the release timestamp vs. the lastUpdated timestamp; if >30min different, real changes have occurred since the last release. In this way, I no longer have to clone 200 repositories just to confirm what the MRM already stores in its metadata. I also do not rely on the <scm> tag being correct, nor even that the artifact's sources reside in public SCM anywhere. So I can now report on third-party components as well. Furthermore, I am in the process of updating step 1 above to lean on "mvn -U -s settings.xml -Dverbose=true versions:display-dependency-updates" instead, since that also rolls in an update of our MRM's public content group including everything it proxies. (The settings.xml here defines our MRM as the sole mirror.) So then, the MRM metadata is (fingers crossed!) guaranteed to be up-to-date during steps 2-4. > But might it be a good idea for a plugin ? If we could get more > concrete I'm willing to implement such things if this will help... I am writing a shell script right now, but may be willing to contribute code to a plugin if this sort of thing is useful enough to a sufficient number of people. > What is the problem with that? Your repository manager should be > configured in that way that SNAPSHOT's will automatically being > removed after a time ? My point is that if I run "mvn -U -DallowSnapshots=true versions:display-dependency-updates" it will always tell me that everything has a newer snapshot version available, because every release concludes with a "bump to next development cycle" commit, which triggers a CI build that deploys a newer artifact than the release artifact. What I need to know is whether a new version (snapshot or otherwise) has been pushed _after some reasonable grace period_ since the last release version was cut. The lastUpdated tag of the maven-metadata.xml is actually perfect for this, as described above. > Can you list some of those use cases please which you are believe in ? Sure! My two most major ones right now are: - My jrun script (https://github.com/ctrueden/jrun) bootstraps a runtime environment for a given "endpoint" which is defined as a GAV, or even simply a GA with V defaulting to RELEASE. If RELEASE were to stop working, I would have to implement special code to discern the latest release in some other way—I guess by checking the remote's maven-metadata.xml and looking at the versions/release tag myself. - We use LATEST to override the version of components—including whole sub-collections of components at a time—for easier SNAPSHOT coupling for local development between components. E.g., in Eclipse, the dependency will automatically switch from being a library/JAR dependency to being a project dependency. See this profile for an example: https://github.com/ scijava/pom-scijava/blob/pom-scijava-14.0.0/pom.xml#L2634-L2683. See also http://imagej.net/Architecture#Using_snapshot_couplings_during_development. If LATEST goes away, I guess I can use [0,) everywhere, but that is pretty non-intuitive by comparison. My intuition also strongly tells me that there are other valid use cases here; I can think harder about it if you are still not convinced. Maybe others here can respond as well with their use cases? Moreover, I don't see the value of discontinuing these special keywords. It breaks backwards compatibility for no technical or social advantage that I can see. I am not trying to be difficult—if there was a rationale written somewhere for why these keywords are bad, I would love to see it!—but the only rationale I have actually found are vague and brief statements repeated second-hand that these keywords are somehow harmful to reproducibility. But as I argued in this thread's first post: reproducibility is still in danger thanks to version ranges and snapshots. The better solution here to encourage reproducibility is to actually encourage it via warnings—see below. > And what do you think is constructive to address the problem of > reproducibility ? Generally speaking, I think it should be illegal to deploy release versions whose dependencies result in irreproducible builds. To enforce that, we went so far as to write a new enforcer rule (part of org.scijava:scijava-maven-plugin): https://github.com/scijava/scijava-maven-plugin/blob/ scijava-maven-plugin-1.0.0/src/main/java/org/scijava/maven/plugin/enforcer/ RequireReproducibleBuilds.java#L53-L69 https://github.com/scijava/scijava-maven-plugin/blob/ scijava-maven-plugin-1.0.0/src/main/java/org/scijava/ maven/plugin/SnapshotFinder.java#L46-L61 This rule fails the build if it finds a SNAPSHOT parent, or any SNAPSHOT dependencies, recursively throughout the dependency tree. Because of this, we actually found some problematic releases on Central which have snapshot dependencies deep in their dependency trees. I cannot recall whether this rule currently fails the build if there are any version ranges, but I think it ought to do so. Our projects all require reproducible builds always on the master branch. Among other benefits, this allows "git bisect" to work reliably. See http://imagej.net/Architecture#Reproducible_builds for further discussion. I think that by default, Maven should warn if a build is irreproducible like this. That way, people new to Maven will be less likely to suffer pain later due to ignorance of this issue. People need to know when their builds are vulnerable. As things stand, I think many people just use a "snapshots all the time" approach because it is initially easier, without realizing how this will burn them in the future. That said, I also think there should be a property to disable the warning, since there are valid use cases (see above) for using rolling versions. Regards, Curtis -- Curtis Rueden LOCI software architect - https://loci.wisc.edu/software ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden On Mon, Apr 24, 2017 at 2:24 PM, Karl Heinz Marbaise <[email protected]> wrote: > Hi, > > On 24/04/17 19:46, Curtis Rueden wrote: > >> Hi Jesse, >> >> Prefer to harden the version # in a corporate pom. Then use >>> versions-m-p to detect and update bulk dependencies across all our >>> projects. We manage about 350 dependencies in the corporate pom >>> >> >> Definitely agreed. I am managing a similar number, and indeed versions-m-p >> is super nice for displaying which dependencies have new versions >> available. (Must remember to feed -U to mvn, though!) >> >> > Not only for displaying...you can also use it to update them... > > Yes I know there are issues in that plugin (I'm currently working on an > larger update https://github.com/mojohaus/versions-maven-plugin/commits/ma > ster)... > > Also updating to newer release versions can be done with that and > correctly checked in into version control. > > I am still looking for an easy way to answer the other question: "Have >> there been changes to this component since the last release" -- i.e. "Does >> this component need a new release" >> > > Maybe I misunderstand a thing here but the first part can be answered by > using your version control? Make a diff between latest release tag and the > current trunk/master ? If a component needs a release is something which > only can being answered by a human... > > But might it be a good idea for a plugin ? If we could get more concrete > I'm willing to implement such things if this will help... > > > > -- since we use release-ready master > >> branches. >> > > > I think the allowSnapshots property is not sufficient in my case, > >> since the CI will always deploy a new SNAPSHOT in response to the "Bump to >> next development cycle" commit which does nothing else besides bump the >> version on master after each release. >> > > What is the problem with that? Your repository manager should be > configured in that way that SNAPSHOT's will automatically being removed > after a time ? > > > > >> And even if somehow the versions-m-p had a magicallySolveAllMyProblems >> flag >> here, >> > > I still believe there are other use cases where RELEASE and LATEST >> are useful -- and that deprecating/removing them does not do anything >> constructive to address the problem of irreproducible builds. >> > > Can you list some of those use cases please which you are believe in ? > > And what do you think is constructive to address the problem of > reproducibility ? > > > > >> Regards, >> Curtis >> >> P.S. to Rick Huff: Thanks for taking the time to reply. :-) >> >> -- >> Curtis Rueden >> LOCI software architect - https://loci.wisc.edu/software >> ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden >> >> >> On Mon, Apr 24, 2017 at 12:37 PM, jieryn <[email protected]> wrote: >> >> Prefer to harden the version # in a corporate pom. Then use >>> versions-m-p to detect and update bulk dependencies across all our >>> projects. We manage about 350 dependencies in the corporate pom, and >>> that doesn't even include the huge number that are scope=import for >>> Arquillian and Selenium, etc. >>> >>> On Mon, Apr 24, 2017 at 12:58 PM, Curtis Rueden <[email protected]> >>> wrote: >>> >>>> I would like to argue for the inclusion / restoration / continued >>>>> support of the RELEASE and LATEST tags. >>>>> >>>> >>>> Really? No one else cares enough to respond? >>>> >>>> I am very often running into use cases where the easiest solution seems >>>> >>> to >>> >>>> be LATEST and/or RELEASE. I have to manage a large Bill of Materials POM >>>> and I need to know things like "which components have new releases >>>> available" and "which components have SNAPSHOT builds since the last >>>> release." How do other people answer these questions without custom >>>> tooling, and without using LATEST or RELEASE tags? >>>> >>> > You can simply let run versions-maven-plugin get a report about updates or > just let update versions-maven-plugin the pom and if you checkin the pom > the version control will find out if something has changed or not ? > > Apart from that maybe I misunderstand a thing here, but using the > versions-maven-plugin is custom tooling for you? > > > > > >>>> Regards, >>>> Curtis >>>> >>>> -- >>>> Curtis Rueden >>>> LOCI software architect - https://loci.wisc.edu/software >>>> ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden >>>> >>>> >>>> On Thu, Apr 13, 2017 at 10:51 AM, Curtis Rueden <[email protected]> >>>> >>> wrote: >>> >>>> >>>> Hi Maven developers, >>>>> >>>>> I would like to argue for the inclusion / restoration / continued >>>>> >>>> support >>> >>>> of the RELEASE and LATEST tags. >>>>> >>>>> Reasons: >>>>> >>>>> 1) There are many valid use cases for them. For example, my script jrun >>>>> [1] uses RELEASE right now to make it easier to launch a Maven GAV. >>>>> Launching e.g. the Jython REPL is as easy as "jrun >>>>> org.python:jython-standalone" from the CLI. But this relies on RELEASE >>>>> working. I know that Maven is traditionally viewed as primarily a >>>>> build-time tool, but it is also extremely useful for synthesizing >>>>> >>>> runtime >>> >>>> environments, and IMHO underutilized in this regard. >>>>> >>>>> 2) For LATEST, you can achieve basically the same behavior using a >>>>> >>>> version >>> >>>> range like [0,). There is no other way (that I know of) to achieve what >>>>> >>>> the >>> >>>> RELEASE tag does. >>>>> >>>>> 3) The argument that they harm reproducibility is totally valid. But so >>>>> >>>> do >>> >>>> SNAPSHOTs, and so do version ranges. And those have not been >>>>> >>>> deprecated. So >>> >>>> why are RELEASE and LATEST eschewed so heavily? >>>>> >>>> > I agree fully to that, cause I've learned that using version ranges has > the same problem as you already mentioned. I would like to deprecate the > version ranges as well.... > > The SNAPSHOT's a very usefull but you can run in problems too...yes... > > > >>>>> Orthogonally: I think it would be awesome to warn about irreproducible >>>>> builds, be they from SNAPSHOTs, version ranges, or these special >>>>> >>>> keywords. >>> >>>> People need to know when their builds are vulnerable. But there should >>>>> probably also be a property to disable this warning, for the cases when >>>>> >>>> the >>> >>>> developer intentionally wants/needs to use them, and knows what they are >>>>> doing. >>>>> >>>> > My experience is that they simply don't understand the consequence using > LATEST/RELEASE nor the usage of version ranges...and often astonished of > build failures later and having problems reproducing builds... > > Kind regards > Karl Heinz Marbaise > > If the issue is just that no ones has time to work on e.g. >> >>> MNG-6206, >>> >>>> I could try to make some time for it—I feel strongly enough about this >>>>> issue. >>>>> >>>> > > > > >>>>> Thanks for any insight, workarounds, counterarguments, agreement. >>>>> (Especially if you agree: please speak up, so the core Maven devs know >>>>> >>>> I'm >>> >>>> not just an outlier here!) >>>>> >>>>> Regards, >>>>> Curtis >>>>> >>>>> [1] https://github.com/ctrueden/jrun >>>>> >>>>> -- >>>>> Curtis Rueden >>>>> LOCI software architect - https://loci.wisc.edu/software >>>>> ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden >>>>> >>>>> > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
