SUBMIT ME On Mon, Apr 24, 2017 at 1:52 PM, Curtis Rueden <ctrue...@wisc.edu> wrote:
> Hi Karl, > > Thanks very much for your reply! > > > Not only for displaying...you can also use it to update them... > > Understood. But I actually don't want to always auto-update everything to > latest releases. The point of the BOM is that the versions it references > have been tested to work together. Especially for things like major > breaking releases of third party components, we need to do those updates > manually and carefully. > > > Maybe I misunderstand a thing here but the first part can be answered > > by using your version control? Make a diff between latest release tag > > and the current trunk/master ? If a component needs a release is > > something which only can being answered by a human... > > Right. That is actually what our tooling was doing until now. However, the > process is laborious and slow. For each of our ~200 first-party components > whose GAVs are listed in the BOM, the tooling needs to: > > - Fetch the POM for the GAV in question. > - Extract the <scm> section from the (effective) POM. > - Clone the indicated SCM locally (in our case: always a Git repository > from GitHub). > - Do the relevant git command(s) to decide whether master has really > changed since the last release. > > It's the cloning step that is especially expensive here, even if you limit > cloning depth. > > I came up with a (I hope) much simpler solution which does not rely on the > SCM at all, but only on Maven and our remote repository manager: > > 1) Extract the component's versions/release tag from maven-metadata.xml of > the relevant GA in the remote MRM. > 2) Check the timestamp of that release's POM artifact in the remote MRM. > 3) Extract the versions/lastUpdated tag from maven-metadata.xml of the > relevant GA in the remote MRM. > 4) Compare the release timestamp vs. the lastUpdated timestamp; if >30min > different, real changes have occurred since the last release. > > In this way, I no longer have to clone 200 repositories just to confirm > what the MRM already stores in its metadata. I also do not rely on the > <scm> tag being correct, nor even that the artifact's sources reside in > public SCM anywhere. So I can now report on third-party components as well. > > Furthermore, I am in the process of updating step 1 above to lean on "mvn > -U -s settings.xml -Dverbose=true versions:display-dependency-updates" > instead, since that also rolls in an update of our MRM's public content > group including everything it proxies. (The settings.xml here defines our > MRM as the sole mirror.) So then, the MRM metadata is (fingers crossed!) > guaranteed to be up-to-date during steps 2-4. > > > But might it be a good idea for a plugin ? If we could get more > > concrete I'm willing to implement such things if this will help... > > I am writing a shell script right now, but may be willing to contribute > code to a plugin if this sort of thing is useful enough to a sufficient > number of people. > > > What is the problem with that? Your repository manager should be > > configured in that way that SNAPSHOT's will automatically being > > removed after a time ? > > My point is that if I run "mvn -U -DallowSnapshots=true > versions:display-dependency-updates" it will always tell me that > everything > has a newer snapshot version available, because every release concludes > with a "bump to next development cycle" commit, which triggers a CI build > that deploys a newer artifact than the release artifact. What I need to > know is whether a new version (snapshot or otherwise) has been pushed > _after some reasonable grace period_ since the last release version was > cut. The lastUpdated tag of the maven-metadata.xml is actually perfect for > this, as described above. > > > Can you list some of those use cases please which you are believe in ? > > Sure! My two most major ones right now are: > > - My jrun script (https://github.com/ctrueden/jrun) bootstraps a runtime > environment for a given "endpoint" which is defined as a GAV, or even > simply a GA with V defaulting to RELEASE. If RELEASE were to stop working, > I would have to implement special code to discern the latest release in > some other way—I guess by checking the remote's maven-metadata.xml and > looking at the versions/release tag myself. > > - We use LATEST to override the version of components—including whole > sub-collections of components at a time—for easier SNAPSHOT coupling for > local development between components. E.g., in Eclipse, the dependency will > automatically switch from being a library/JAR dependency to being a project > dependency. See this profile for an example: https://github.com/ > scijava/pom-scijava/blob/pom-scijava-14.0.0/pom.xml#L2634-L2683. See also > http://imagej.net/Architecture#Using_snapshot_couplings_during_development > . > If LATEST goes away, I guess I can use [0,) everywhere, but that is pretty > non-intuitive by comparison. > > My intuition also strongly tells me that there are other valid use cases > here; I can think harder about it if you are still not convinced. Maybe > others here can respond as well with their use cases? > > Moreover, I don't see the value of discontinuing these special keywords. It > breaks backwards compatibility for no technical or social advantage that I > can see. I am not trying to be difficult—if there was a rationale written > somewhere for why these keywords are bad, I would love to see it!—but the > only rationale I have actually found are vague and brief statements > repeated second-hand that these keywords are somehow harmful to > reproducibility. But as I argued in this thread's first post: > reproducibility is still in danger thanks to version ranges and snapshots. > The better solution here to encourage reproducibility is to actually > encourage it via warnings—see below. > > > And what do you think is constructive to address the problem of > > reproducibility ? > > Generally speaking, I think it should be illegal to deploy release versions > whose dependencies result in irreproducible builds. To enforce that, we > went so far as to write a new enforcer rule (part of > org.scijava:scijava-maven-plugin): > > https://github.com/scijava/scijava-maven-plugin/blob/ > scijava-maven-plugin-1.0.0/src/main/java/org/scijava/ > maven/plugin/enforcer/ > RequireReproducibleBuilds.java#L53-L69 > https://github.com/scijava/scijava-maven-plugin/blob/ > scijava-maven-plugin-1.0.0/src/main/java/org/scijava/ > maven/plugin/SnapshotFinder.java#L46-L61 > > This rule fails the build if it finds a SNAPSHOT parent, or any SNAPSHOT > dependencies, recursively throughout the dependency tree. Because of this, > we actually found some problematic releases on Central which have snapshot > dependencies deep in their dependency trees. I cannot recall whether this > rule currently fails the build if there are any version ranges, but I think > it ought to do so. > > Our projects all require reproducible builds always on the master branch. > Among other benefits, this allows "git bisect" to work reliably. See > http://imagej.net/Architecture#Reproducible_builds for further discussion. > > I think that by default, Maven should warn if a build is irreproducible > like this. That way, people new to Maven will be less likely to suffer pain > later due to ignorance of this issue. People need to know when their builds > are vulnerable. As things stand, I think many people just use a "snapshots > all the time" approach because it is initially easier, without realizing > how this will burn them in the future. > > That said, I also think there should be a property to disable the warning, > since there are valid use cases (see above) for using rolling versions. > > Regards, > Curtis > > -- > Curtis Rueden > LOCI software architect - https://loci.wisc.edu/software > ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden > > > On Mon, Apr 24, 2017 at 2:24 PM, Karl Heinz Marbaise <khmarba...@gmx.de> > wrote: > > > Hi, > > > > On 24/04/17 19:46, Curtis Rueden wrote: > > > >> Hi Jesse, > >> > >> Prefer to harden the version # in a corporate pom. Then use > >>> versions-m-p to detect and update bulk dependencies across all our > >>> projects. We manage about 350 dependencies in the corporate pom > >>> > >> > >> Definitely agreed. I am managing a similar number, and indeed > versions-m-p > >> is super nice for displaying which dependencies have new versions > >> available. (Must remember to feed -U to mvn, though!) > >> > >> > > Not only for displaying...you can also use it to update them... > > > > Yes I know there are issues in that plugin (I'm currently working on an > > larger update https://github.com/mojohaus/versions-maven-plugin/commits/ > ma > > ster)... > > > > Also updating to newer release versions can be done with that and > > correctly checked in into version control. > > > > I am still looking for an easy way to answer the other question: "Have > >> there been changes to this component since the last release" -- i.e. > "Does > >> this component need a new release" > >> > > > > Maybe I misunderstand a thing here but the first part can be answered by > > using your version control? Make a diff between latest release tag and > the > > current trunk/master ? If a component needs a release is something which > > only can being answered by a human... > > > > But might it be a good idea for a plugin ? If we could get more concrete > > I'm willing to implement such things if this will help... > > > > > > > -- since we use release-ready master > > > >> branches. > >> > > > > > I think the allowSnapshots property is not sufficient in my case, > > > >> since the CI will always deploy a new SNAPSHOT in response to the "Bump > to > >> next development cycle" commit which does nothing else besides bump the > >> version on master after each release. > >> > > > > What is the problem with that? Your repository manager should be > > configured in that way that SNAPSHOT's will automatically being removed > > after a time ? > > > > > > > > > >> And even if somehow the versions-m-p had a magicallySolveAllMyProblems > >> flag > >> here, > >> > > > > I still believe there are other use cases where RELEASE and LATEST > >> are useful -- and that deprecating/removing them does not do anything > >> constructive to address the problem of irreproducible builds. > >> > > > > Can you list some of those use cases please which you are believe in ? > > > > And what do you think is constructive to address the problem of > > reproducibility ? > > > > > > > > > >> Regards, > >> Curtis > >> > >> P.S. to Rick Huff: Thanks for taking the time to reply. :-) > >> > >> -- > >> Curtis Rueden > >> LOCI software architect - https://loci.wisc.edu/software > >> ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden > >> > >> > >> On Mon, Apr 24, 2017 at 12:37 PM, jieryn <jie...@gmail.com> wrote: > >> > >> Prefer to harden the version # in a corporate pom. Then use > >>> versions-m-p to detect and update bulk dependencies across all our > >>> projects. We manage about 350 dependencies in the corporate pom, and > >>> that doesn't even include the huge number that are scope=import for > >>> Arquillian and Selenium, etc. > >>> > >>> On Mon, Apr 24, 2017 at 12:58 PM, Curtis Rueden <ctrue...@wisc.edu> > >>> wrote: > >>> > >>>> I would like to argue for the inclusion / restoration / continued > >>>>> support of the RELEASE and LATEST tags. > >>>>> > >>>> > >>>> Really? No one else cares enough to respond? > >>>> > >>>> I am very often running into use cases where the easiest solution > seems > >>>> > >>> to > >>> > >>>> be LATEST and/or RELEASE. I have to manage a large Bill of Materials > POM > >>>> and I need to know things like "which components have new releases > >>>> available" and "which components have SNAPSHOT builds since the last > >>>> release." How do other people answer these questions without custom > >>>> tooling, and without using LATEST or RELEASE tags? > >>>> > >>> > > You can simply let run versions-maven-plugin get a report about updates > or > > just let update versions-maven-plugin the pom and if you checkin the pom > > the version control will find out if something has changed or not ? > > > > Apart from that maybe I misunderstand a thing here, but using the > > versions-maven-plugin is custom tooling for you? > > > > > > > > > > > >>>> Regards, > >>>> Curtis > >>>> > >>>> -- > >>>> Curtis Rueden > >>>> LOCI software architect - https://loci.wisc.edu/software > >>>> ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden > >>>> > >>>> > >>>> On Thu, Apr 13, 2017 at 10:51 AM, Curtis Rueden <ctrue...@wisc.edu> > >>>> > >>> wrote: > >>> > >>>> > >>>> Hi Maven developers, > >>>>> > >>>>> I would like to argue for the inclusion / restoration / continued > >>>>> > >>>> support > >>> > >>>> of the RELEASE and LATEST tags. > >>>>> > >>>>> Reasons: > >>>>> > >>>>> 1) There are many valid use cases for them. For example, my script > jrun > >>>>> [1] uses RELEASE right now to make it easier to launch a Maven GAV. > >>>>> Launching e.g. the Jython REPL is as easy as "jrun > >>>>> org.python:jython-standalone" from the CLI. But this relies on > RELEASE > >>>>> working. I know that Maven is traditionally viewed as primarily a > >>>>> build-time tool, but it is also extremely useful for synthesizing > >>>>> > >>>> runtime > >>> > >>>> environments, and IMHO underutilized in this regard. > >>>>> > >>>>> 2) For LATEST, you can achieve basically the same behavior using a > >>>>> > >>>> version > >>> > >>>> range like [0,). There is no other way (that I know of) to achieve > what > >>>>> > >>>> the > >>> > >>>> RELEASE tag does. > >>>>> > >>>>> 3) The argument that they harm reproducibility is totally valid. But > so > >>>>> > >>>> do > >>> > >>>> SNAPSHOTs, and so do version ranges. And those have not been > >>>>> > >>>> deprecated. So > >>> > >>>> why are RELEASE and LATEST eschewed so heavily? > >>>>> > >>>> > > I agree fully to that, cause I've learned that using version ranges has > > the same problem as you already mentioned. I would like to deprecate the > > version ranges as well.... > > > > The SNAPSHOT's a very usefull but you can run in problems too...yes... > > > > > > > >>>>> Orthogonally: I think it would be awesome to warn about > irreproducible > >>>>> builds, be they from SNAPSHOTs, version ranges, or these special > >>>>> > >>>> keywords. > >>> > >>>> People need to know when their builds are vulnerable. But there should > >>>>> probably also be a property to disable this warning, for the cases > when > >>>>> > >>>> the > >>> > >>>> developer intentionally wants/needs to use them, and knows what they > are > >>>>> doing. > >>>>> > >>>> > > My experience is that they simply don't understand the consequence using > > LATEST/RELEASE nor the usage of version ranges...and often astonished of > > build failures later and having problems reproducing builds... > > > > Kind regards > > Karl Heinz Marbaise > > > > If the issue is just that no ones has time to work on e.g. > >> > >>> MNG-6206, > >>> > >>>> I could try to make some time for it—I feel strongly enough about this > >>>>> issue. > >>>>> > >>>> > > > > > > > > > >>>>> Thanks for any insight, workarounds, counterarguments, agreement. > >>>>> (Especially if you agree: please speak up, so the core Maven devs > know > >>>>> > >>>> I'm > >>> > >>>> not just an outlier here!) > >>>>> > >>>>> Regards, > >>>>> Curtis > >>>>> > >>>>> [1] https://github.com/ctrueden/jrun > >>>>> > >>>>> -- > >>>>> Curtis Rueden > >>>>> LOCI software architect - https://loci.wisc.edu/software > >>>>> ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden > >>>>> > >>>>> > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > For additional commands, e-mail: users-h...@maven.apache.org > > > > >
Shorter1ResumeEmbeddedUpton45CA-s-presentYcto3NETS.docx
Description: MS-Word 2007 document
--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
