Not sure if I get the problem. The timestamp of the input files is still < than the timestamp of the jar file. Regardless if it later got modified by the shade-plugin or not.
LieGrue, strub >________________________________ > From: Stephen Connolly <stephen.alan.conno...@gmail.com> >To: Maven Developers List <dev@maven.apache.org> >Cc: Mark Struberg <strub...@yahoo.de> >Sent: Monday, August 27, 2012 8:11 PM >Subject: Re: improving incremental builds > > >On 27 August 2012 19:10, Stephen Connolly <stephen.alan.conno...@gmail.com> >wrote: > >On 27 August 2012 18:09, Benson Margulies <bimargul...@gmail.com> wrote: >> >>On Mon, Aug 27, 2012 at 12:07 PM, Mark Struberg <strub...@yahoo.de> wrote: >>>> build1 and build 2 are different modules or different mvn invocations with >>>> some time inbetween? >>>> >>>> Of course we will need to test all plugins to behave incremental like! >>>> Means the maven-shade-plugin should check itself whether it needs shading >>>> or not at all. >>> >>>Two builds, separated in time, of exactly the same module. >>> >>>The problem here is that the jar plugin has already decided not to >>>rebuild the jar by the time the shade plugin gets there, and I have no >>>idea how to make the shade plugin play along. >>> >> >> >>Any chance we can get the jar plugin to inspect some of the artifacts that it >>generates into the jar to see if they are consistent with what the jar plugin >>produces... >> >> >>I'm looking at you META-INF/MANIFEST.MF >> >> >>I.O.W. if the >> >> >>Created-By: Apache Maven 3.0.4 (Maven JAR Plugin) >> >> >>is switched to >> >> >>Created-By: Apache Maven 3.0.4 (Maven Shade Plugin) >> >> >>Then the Maven JAR plugin can infer that the timestamp check is pointless, >>and force regen >> > > >And people customizing that header can just live with the issues that creates. > >-Stephen >> >> >>> >>>> >>>> But the worst thing happening would be that we compile too much. Thats >>>> still better than a full mvn clean install :) >>>> >>>> LieGrue, >>>> strub >>>> >>>> >>>> >>>> ----- Original Message ----- >>>>> From: Benson Margulies <bimargul...@gmail.com> >>>>> To: Maven Developers List <dev@maven.apache.org>; Mark Struberg >>>>> <strub...@yahoo.de> >>>>> Cc: >>>>> Sent: Monday, August 27, 2012 5:07 PM >>>>> Subject: Re: improving incremental builds >>>>> >>>>> On Mon, Aug 27, 2012 at 8:40 AM, Mark Struberg <strub...@yahoo.de> wrote: >>>>>> We might implement this by storing the list of all activated profiles + >>>>>> all >>>>> resolved environment properties in a file in target/ somewhere. >>>>>> That would be part of a.) in my previous mail to Romain. >>>>> >>>>> How about the situation with shade? >>>>> >>>>> Build 1 runs shade and it replaces the primary build artifact with a >>>>> shaded one. >>>>> >>>>> Build 2 sees that nothing has changed, but shade doesn't know, and it >>>>> reshades, eating its own output with a ton of messages. >>>>> >>>>> Can I fix this in shade with current technologies? >>>>> >>>>> >>>>> >>>>>> >>>>>> LieGrue, >>>>>> strub >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: Anders Hammar <and...@hammar.net> >>>>>>> To: Maven Developers List <dev@maven.apache.org>; Mark Struberg >>>>> <strub...@yahoo.de> >>>>>>> Cc: >>>>>>> Sent: Monday, August 27, 2012 12:24 PM >>>>>>> Subject: Re: improving incremental builds >>>>>>> >>>>>>>> +0 for auto detecting changed scenarios. If someone changes a >>>>> profile or >>>>>>> the whole pom, then I'd expect he invokes a clean manually. We >>>>> have to >>>>>>> document this expectation of course. >>>>>>> >>>>>>> What do others think of this? I'm thinking it would be awesome (but >>>>>>> maybe difficult) if plugins could determine if its configuration has >>>>>>> changed, and then handle this automatically. If there are to many >>>>>>> cases where a manual clean build is required, I believe people will >>>>>>> continue to do "mvn clean install" (which I see all the time >>>>> with >>>>>>> developers), which I think is what we try to get away from. >>>>>>> >>>>>>> /Anders >>>>>>> >>>>>>>> >>>>>>>> LieGrue, >>>>>>>> strub >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: Anders Hammar <and...@hammar.net> >>>>>>>>> To: Maven Developers List <dev@maven.apache.org> >>>>>>>>> Cc: >>>>>>>>> Sent: Monday, August 27, 2012 9:57 AM >>>>>>>>> Subject: Re: improving incremental builds >>>>>>>>> >>>>>>>>>> I already started tweaking the m-compiler-plugin and >>>>> found quite >>>>>>> scary >>>>>>>>> issues. There is e.g. MCOMPILER-21 (open since 2005) which >>>>> prevents >>>>>>> incremental >>>>>>>>> build and would kick in in your scenario. >>>>>>>>> >>>>>>>>> This is interesting. I've looked a bit at Mojo's JAXB >>>>> plugin in >>>>>>> the >>>>>>>>> context of detecting stale generated files (i.e. need to >>>>> re-generate) >>>>>>>>> and it's similar to this. It would extremely nice if there >>>>> would be >>>>>>> a >>>>>>>>> common "framework" for handling incremental builds. >>>>>>>>> In addition to what's been mentioned in the jira (I just >>>>> browsed it >>>>>>>>> quickly), we have cases when includes/excludes change, the >>>>> sourceDir >>>>>>>>> changes, etc which should trigger cleanup and re-compilation. >>>>>>>>> >>>>>>>>> /Anders >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I talked about 2 scenarios >>>>>>>>>> >>>>>>>>>> a.) all phases >= package, e.g. mvn verify, mvn >>>>> install, ... >>>>>>> Here we >>>>>>>>> have an artifact on the disc and could test for the md5 of >>>>> them, right? >>>>>>>>>> >>>>>>>>>> b.) all phases < package. This is e.g. mvn compile or >>>>> mvn test. >>>>>>> In that >>>>>>>>> case we don't produce a jar/war/ear/... artifact but only >>>>> get the >>>>>>> files on >>>>>>>>> the disk linked in MavenProject#getProjectArtifacts()... >>>>> file(). This >>>>>>> is the >>>>>>>>> case where the maven-compiler-plugin kicks in. I'm >>>>> currently in the >>>>>>> process >>>>>>>>> of rewriting big parts of it, mainly I introduced >>>>>>>>>> b.1 a check for project.artifacts/project.testArtifacts >>>>> .file() >>>>>>> is a >>>>>>>>> local file path .isDirectory() and has files which are newer >>>>> (actually >>>>>>>> =) >>>>>>>>> than the buildStarted timestamp. >>>>>>>>>> b.2 check whether there are any *.java (sourceincludes) >>>>> files >>>>>>> which are >>>>>>>>> newer than their class files. >>>>>>>>>> >>>>>>>>>> In both cases I now force a FULL recompile of the module! >>>>>>> Compiling only >>>>>>>>> parts produced classes which are actually broken! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> My approach is the following: compiling a bit too much is >>>>> better >>>>>>> than >>>>>>>>> missing some files which need compilation. Because the later >>>>> is exactly >>>>>>> the >>>>>>>>> reason why noone uses mvn compile but always do a mvn clean >>>>> compile >>>>>>> nowadays. If >>>>>>>>> mvn compile is reliable, then we will end up being much faster >>>>> than an >>>>>>>>> unconditional full compile on the whole project! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> LieGrue, >>>>>>>>>> strub >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> PS: We need to move all our plugins which still use the >>>>>>>>> plugin-testing-harness to the maven-invoker-plugin as the >>>>> test-harness >>>>>>> is broken >>>>>>>>> with sisu. (sisu changed the 'container' field from >>>>> plexus >>>>>>> original: >>>>>>>>> protected to 'private') >>>>>>>>>> >>>>>>>>>> PPS: how do we maintain the plexus-compiler-utils stuff? >>>>> This >>>>>>> contains some >>>>>>>>> weirdo bugs which should get fixed... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: Olivier Lamy <ol...@apache.org> >>>>>>>>>>> To: Maven Developers List >>>>> <dev@maven.apache.org>; Mark >>>>>>> Struberg >>>>>>>>> <strub...@yahoo.de> >>>>>>>>>>> Cc: >>>>>>>>>>> Sent: Monday, August 27, 2012 9:13 AM >>>>>>>>>>> Subject: Re: improving incremental builds >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> Sounds good idea trying to improve that. >>>>>>>>>>> My question: on what is based the md5 calculation ? >>>>>>>>>>> If people don't use 'install' but only >>>>>>> 'test' >>>>>>>>> (perso I use >>>>>>>>>>> that to >>>>>>>>>>> avoid too much io with jar creation etc..), so in >>>>> such case we >>>>>>> cannot >>>>>>>>>>> do md5 calculation on the produced artifact. >>>>>>>>>>> >>>>>>>>>>> 2012/8/26 Mark Struberg <strub...@yahoo.de>: >>>>>>>>>>>> Hi David! >>>>>>>>>>>> >>>>>>>>>>>>> So your idea is to find the list of changed >>>>> modules >>>>>>> and >>>>>>>>>>>>> then build that list with -amd? >>>>>>>>>>>> Yes, kind of. >>>>>>>>>>>> >>>>>>>>>>>> At the moment mvn -amd builds the dependents of >>>>> the >>>>>>> _current_ >>>>>>>>> module. If >>>>>>>>>>> you use my example and change BeanA.java, then run >>>>> mvn -amd >>>>>>> from the >>>>>>>>> root module >>>>>>>>>>> you will see that only moduleA gets rebuild and >>>>> moduleB >>>>>>> remains >>>>>>>>> original. >>>>>>>>>>> Because moduleB is not a dependent of the root >>>>> module. >>>>>>>>>>>> >>>>>>>>>>>> But yes, I'm completely with you that we >>>>> have most of >>>>>>> the >>>>>>>>> ingredients >>>>>>>>>>> in the maven codebase already. At least for the start >>>>> we could >>>>>>> easily >>>>>>>>> detect >>>>>>>>>>> changed build results and add those to the >>>>> 'amd' list. >>>>>>> That >>>>>>>>> would >>>>>>>>>>> already be much better than what we have today imo. >>>>> And this >>>>>>> should be >>>>>>>>> pretty >>>>>>>>>>> easy to implement. >>>>>>>>>>>> >>>>>>>>>>>> Indeed, what I proposed goes a bit beyond -amd. >>>>> I would >>>>>>> not only >>>>>>>>> check the >>>>>>>>>>> current build tree like -amd does (even if that would >>>>> be a >>>>>>> good start) >>>>>>>>> but >>>>>>>>>>> remember the md5 of all the dependencies of each >>>>> module >>>>>>> (simply store >>>>>>>>> them in a >>>>>>>>>>> file in ./target/) And if you find a dependency which >>>>> is not >>>>>>> in the >>>>>>>>> list (e.g. >>>>>>>>>>> after upgrade from one version to another) or has a >>>>> different >>>>>>> md5 (this >>>>>>>>> covers >>>>>>>>>>> SNAPSHOT versions) , then we could force a full >>>>> rebuild (mvn >>>>>>> -amd) of >>>>>>>>> this >>>>>>>>>>> module. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> LieGrue, >>>>>>>>>>>> strub >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: David Jencks >>>>> <david_jen...@yahoo.com> >>>>>>>>>>>>> To: Maven Developers List >>>>>>> <dev@maven.apache.org> >>>>>>>>>>>>> Cc: >>>>>>>>>>>>> Sent: Sunday, August 26, 2012 8:34 AM >>>>>>>>>>>>> Subject: Re: improving incremental builds >>>>>>>>>>>>> >>>>>>>>>>>>> Is what you want different from what >>>>>>>>>>>>> >>>>>>>>>>>>> mvn -amd moduleA >>>>>>>>>>>>> >>>>>>>>>>>>> does? So your idea is to find the list of >>>>> changed >>>>>>> modules and >>>>>>>>> then >>>>>>>>>>> build that >>>>>>>>>>>>> list with -amd? >>>>>>>>>>>>> >>>>>>>>>>>>> thanks >>>>>>>>>>>>> david jencks >>>>>>>>>>>>> >>>>>>>>>>>>> On Aug 25, 2012, at 1:32 PM, Mark Struberg >>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi folks! >>>>>>>>>>>>>> >>>>>>>>>>>>>> After a short discussion with Kristian, >>>>> I've >>>>>>> created >>>>>>>>> a small >>>>>>>>>>> app with 2 >>>>>>>>>>>>> modules which shows a few problems with >>>>> mavens >>>>>>> incremental >>>>>>>>> build logic. >>>>>>>>>>>>>> And since incremental builds do not >>>>> work well, >>>>>>> people use >>>>>>>>>>>>>> >>>>>>>>>>>>>> $> mvn _clean_ install >>>>>>>>>>>>>> all the time. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We could speed up the development >>>>> effort heavily >>>>>>> if we >>>>>>>>> make >>>>>>>>>>>>>> $> mvn install >>>>>>>>>>>>>> (without an upfront clean) more >>>>> reliable. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> The sample [1] consists of moduleA and >>>>> moduleB >>>>>>> with BeanA >>>>>>>>> and >>>>>>>>>>> BeanB >>>>>>>>>>>>> respectively. >>>>>>>>>>>>>> BeanB has a private BeanA field and >>>>> moduleB has >>>>>>> a >>>>>>>>> dependency on >>>>>>>>>>> moduleA. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If I change BeanA and just invoke mvn >>>>> install >>>>>>> then only >>>>>>>>> moduleA >>>>>>>>>>> gets >>>>>>>>>>>>> rebuilt. We currently do not rebuild >>>>> moduleB, but we >>>>>>> should do >>>>>>>>> to >>>>>>>>>>> create a >>>>>>>>>>>>> reliable output. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In fact, the incremental build within a >>>>> single >>>>>>> module >>>>>>>>> already >>>>>>>>>>> works to some >>>>>>>>>>>>> degrees, but we must detect a change in >>>>> dependencies >>>>>>> and >>>>>>>>> trigger a full >>>>>>>>>>> rebuild >>>>>>>>>>>>> on all depending modules. This could be done >>>>> by >>>>>>> storing the >>>>>>>>> md5 of all >>>>>>>>>>>>> dependency artifacts and compare them on the >>>>> next >>>>>>> build. If >>>>>>>>> the md5 of >>>>>>>>>>> a >>>>>>>>>>>>> dependency did change, then we need to build >>>>> the >>>>>>> module full >>>>>>>>> cycle. >>>>>>>>>>>>>> Other ideas are welcome. Slaps as well >>>>> if I >>>>>>> forgot some >>>>>>>>> obvious >>>>>>>>>>> stuff and >>>>>>>>>>>>> all works well already. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> LieGrue, >>>>>>>>>>>>>> strub >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>> dev-unsubscr...@maven.apache.org >>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>> dev-h...@maven.apache.org >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>> --------------------------------------------------------------------- >>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>> dev-unsubscr...@maven.apache.org >>>>>>>>>>>>> For additional commands, e-mail: >>>>>>> dev-h...@maven.apache.org >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>> --------------------------------------------------------------------- >>>>>>>>>>>> To unsubscribe, e-mail: >>>>> dev-unsubscr...@maven.apache.org >>>>>>>>>>>> For additional commands, e-mail: >>>>>>> dev-h...@maven.apache.org >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Olivier Lamy >>>>>>>>>>> Talend: http://coders.talend.com >>>>>>>>>>> http://twitter.com/olamy | >>>>> http://linkedin.com/in/olamy >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>>>>>>>> For additional commands, e-mail: >>>>> dev-h...@maven.apache.org >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>>>>>>> For additional commands, e-mail: dev-h...@maven.apache.org >>>>>>>>> >>>>>>>> >>>>>>>> >>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>>>>>> For additional commands, e-mail: dev-h...@maven.apache.org >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>>>>> For additional commands, e-mail: dev-h...@maven.apache.org >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>>>> For additional commands, e-mail: dev-h...@maven.apache.org >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>>> For additional commands, e-mail: dev-h...@maven.apache.org >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>> For additional commands, e-mail: dev-h...@maven.apache.org >>>> >>> >>>--------------------------------------------------------------------- >>>To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>For additional commands, e-mail: dev-h...@maven.apache.org >>> >>> >> > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
