On 27 August 2012 18:09, Benson Margulies <bimargul...@gmail.com> wrote:
> On Mon, Aug 27, 2012 at 12:07 PM, Mark Struberg <strub...@yahoo.de> wrote: > > build1 and build 2 are different modules or different mvn invocations > with some time inbetween? > > > > Of course we will need to test all plugins to behave incremental like! > Means the maven-shade-plugin should check itself whether it needs shading > or not at all. > > Two builds, separated in time, of exactly the same module. > > The problem here is that the jar plugin has already decided not to > rebuild the jar by the time the shade plugin gets there, and I have no > idea how to make the shade plugin play along. > Any chance we can get the jar plugin to inspect some of the artifacts that it generates into the jar to see if they are consistent with what the jar plugin produces... I'm looking at you META-INF/MANIFEST.MF I.O.W. if the Created-By: Apache Maven 3.0.4 (Maven JAR Plugin) is switched to Created-By: Apache Maven 3.0.4 (Maven Shade Plugin) Then the Maven JAR plugin can infer that the timestamp check is pointless, and force regen -Stephen > > > > > > But the worst thing happening would be that we compile too much. Thats > still better than a full mvn clean install :) > > > > LieGrue, > > strub > > > > > > > > ----- Original Message ----- > >> From: Benson Margulies <bimargul...@gmail.com> > >> To: Maven Developers List <dev@maven.apache.org>; Mark Struberg < > strub...@yahoo.de> > >> Cc: > >> Sent: Monday, August 27, 2012 5:07 PM > >> Subject: Re: improving incremental builds > >> > >> On Mon, Aug 27, 2012 at 8:40 AM, Mark Struberg <strub...@yahoo.de> > wrote: > >>> We might implement this by storing the list of all activated profiles > + all > >> resolved environment properties in a file in target/ somewhere. > >>> That would be part of a.) in my previous mail to Romain. > >> > >> How about the situation with shade? > >> > >> Build 1 runs shade and it replaces the primary build artifact with a > shaded one. > >> > >> Build 2 sees that nothing has changed, but shade doesn't know, and it > >> reshades, eating its own output with a ton of messages. > >> > >> Can I fix this in shade with current technologies? > >> > >> > >> > >>> > >>> LieGrue, > >>> strub > >>> > >>> > >>> > >>> ----- Original Message ----- > >>>> From: Anders Hammar <and...@hammar.net> > >>>> To: Maven Developers List <dev@maven.apache.org>; Mark Struberg > >> <strub...@yahoo.de> > >>>> Cc: > >>>> Sent: Monday, August 27, 2012 12:24 PM > >>>> Subject: Re: improving incremental builds > >>>> > >>>>> +0 for auto detecting changed scenarios. If someone changes a > >> profile or > >>>> the whole pom, then I'd expect he invokes a clean manually. We > >> have to > >>>> document this expectation of course. > >>>> > >>>> What do others think of this? I'm thinking it would be awesome (but > >>>> maybe difficult) if plugins could determine if its configuration has > >>>> changed, and then handle this automatically. If there are to many > >>>> cases where a manual clean build is required, I believe people will > >>>> continue to do "mvn clean install" (which I see all the time > >> with > >>>> developers), which I think is what we try to get away from. > >>>> > >>>> /Anders > >>>> > >>>>> > >>>>> LieGrue, > >>>>> strub > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> ----- Original Message ----- > >>>>>> From: Anders Hammar <and...@hammar.net> > >>>>>> To: Maven Developers List <dev@maven.apache.org> > >>>>>> Cc: > >>>>>> Sent: Monday, August 27, 2012 9:57 AM > >>>>>> Subject: Re: improving incremental builds > >>>>>> > >>>>>>> I already started tweaking the m-compiler-plugin and > >> found quite > >>>> scary > >>>>>> issues. There is e.g. MCOMPILER-21 (open since 2005) which > >> prevents > >>>> incremental > >>>>>> build and would kick in in your scenario. > >>>>>> > >>>>>> This is interesting. I've looked a bit at Mojo's JAXB > >> plugin in > >>>> the > >>>>>> context of detecting stale generated files (i.e. need to > >> re-generate) > >>>>>> and it's similar to this. It would extremely nice if there > >> would be > >>>> a > >>>>>> common "framework" for handling incremental builds. > >>>>>> In addition to what's been mentioned in the jira (I just > >> browsed it > >>>>>> quickly), we have cases when includes/excludes change, the > >> sourceDir > >>>>>> changes, etc which should trigger cleanup and re-compilation. > >>>>>> > >>>>>> /Anders > >>>>>> > >>>>>>> > >>>>>>> I talked about 2 scenarios > >>>>>>> > >>>>>>> a.) all phases >= package, e.g. mvn verify, mvn > >> install, ... > >>>> Here we > >>>>>> have an artifact on the disc and could test for the md5 of > >> them, right? > >>>>>>> > >>>>>>> b.) all phases < package. This is e.g. mvn compile or > >> mvn test. > >>>> In that > >>>>>> case we don't produce a jar/war/ear/... artifact but only > >> get the > >>>> files on > >>>>>> the disk linked in MavenProject#getProjectArtifacts()... > >> file(). This > >>>> is the > >>>>>> case where the maven-compiler-plugin kicks in. I'm > >> currently in the > >>>> process > >>>>>> of rewriting big parts of it, mainly I introduced > >>>>>>> b.1 a check for project.artifacts/project.testArtifacts > >> .file() > >>>> is a > >>>>>> local file path .isDirectory() and has files which are newer > >> (actually > >>>>> =) > >>>>>> than the buildStarted timestamp. > >>>>>>> b.2 check whether there are any *.java (sourceincludes) > >> files > >>>> which are > >>>>>> newer than their class files. > >>>>>>> > >>>>>>> In both cases I now force a FULL recompile of the module! > >>>> Compiling only > >>>>>> parts produced classes which are actually broken! > >>>>>>> > >>>>>>> > >>>>>>> My approach is the following: compiling a bit too much is > >> better > >>>> than > >>>>>> missing some files which need compilation. Because the later > >> is exactly > >>>> the > >>>>>> reason why noone uses mvn compile but always do a mvn clean > >> compile > >>>> nowadays. If > >>>>>> mvn compile is reliable, then we will end up being much faster > >> than an > >>>>>> unconditional full compile on the whole project! > >>>>>>> > >>>>>>> > >>>>>>> LieGrue, > >>>>>>> strub > >>>>>>> > >>>>>>> > >>>>>>> PS: We need to move all our plugins which still use the > >>>>>> plugin-testing-harness to the maven-invoker-plugin as the > >> test-harness > >>>> is broken > >>>>>> with sisu. (sisu changed the 'container' field from > >> plexus > >>>> original: > >>>>>> protected to 'private') > >>>>>>> > >>>>>>> PPS: how do we maintain the plexus-compiler-utils stuff? > >> This > >>>> contains some > >>>>>> weirdo bugs which should get fixed... > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ----- Original Message ----- > >>>>>>>> From: Olivier Lamy <ol...@apache.org> > >>>>>>>> To: Maven Developers List > >> <dev@maven.apache.org>; Mark > >>>> Struberg > >>>>>> <strub...@yahoo.de> > >>>>>>>> Cc: > >>>>>>>> Sent: Monday, August 27, 2012 9:13 AM > >>>>>>>> Subject: Re: improving incremental builds > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> Sounds good idea trying to improve that. > >>>>>>>> My question: on what is based the md5 calculation ? > >>>>>>>> If people don't use 'install' but only > >>>> 'test' > >>>>>> (perso I use > >>>>>>>> that to > >>>>>>>> avoid too much io with jar creation etc..), so in > >> such case we > >>>> cannot > >>>>>>>> do md5 calculation on the produced artifact. > >>>>>>>> > >>>>>>>> 2012/8/26 Mark Struberg <strub...@yahoo.de>: > >>>>>>>>> Hi David! > >>>>>>>>> > >>>>>>>>>> So your idea is to find the list of changed > >> modules > >>>> and > >>>>>>>>>> then build that list with -amd? > >>>>>>>>> Yes, kind of. > >>>>>>>>> > >>>>>>>>> At the moment mvn -amd builds the dependents of > >> the > >>>> _current_ > >>>>>> module. If > >>>>>>>> you use my example and change BeanA.java, then run > >> mvn -amd > >>>> from the > >>>>>> root module > >>>>>>>> you will see that only moduleA gets rebuild and > >> moduleB > >>>> remains > >>>>>> original. > >>>>>>>> Because moduleB is not a dependent of the root > >> module. > >>>>>>>>> > >>>>>>>>> But yes, I'm completely with you that we > >> have most of > >>>> the > >>>>>> ingredients > >>>>>>>> in the maven codebase already. At least for the start > >> we could > >>>> easily > >>>>>> detect > >>>>>>>> changed build results and add those to the > >> 'amd' list. > >>>> That > >>>>>> would > >>>>>>>> already be much better than what we have today imo. > >> And this > >>>> should be > >>>>>> pretty > >>>>>>>> easy to implement. > >>>>>>>>> > >>>>>>>>> Indeed, what I proposed goes a bit beyond -amd. > >> I would > >>>> not only > >>>>>> check the > >>>>>>>> current build tree like -amd does (even if that would > >> be a > >>>> good start) > >>>>>> but > >>>>>>>> remember the md5 of all the dependencies of each > >> module > >>>> (simply store > >>>>>> them in a > >>>>>>>> file in ./target/) And if you find a dependency which > >> is not > >>>> in the > >>>>>> list (e.g. > >>>>>>>> after upgrade from one version to another) or has a > >> different > >>>> md5 (this > >>>>>> covers > >>>>>>>> SNAPSHOT versions) , then we could force a full > >> rebuild (mvn > >>>> -amd) of > >>>>>> this > >>>>>>>> module. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> LieGrue, > >>>>>>>>> strub > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ----- Original Message ----- > >>>>>>>>>> From: David Jencks > >> <david_jen...@yahoo.com> > >>>>>>>>>> To: Maven Developers List > >>>> <dev@maven.apache.org> > >>>>>>>>>> Cc: > >>>>>>>>>> Sent: Sunday, August 26, 2012 8:34 AM > >>>>>>>>>> Subject: Re: improving incremental builds > >>>>>>>>>> > >>>>>>>>>> Is what you want different from what > >>>>>>>>>> > >>>>>>>>>> mvn -amd moduleA > >>>>>>>>>> > >>>>>>>>>> does? So your idea is to find the list of > >> changed > >>>> modules and > >>>>>> then > >>>>>>>> build that > >>>>>>>>>> list with -amd? > >>>>>>>>>> > >>>>>>>>>> thanks > >>>>>>>>>> david jencks > >>>>>>>>>> > >>>>>>>>>> On Aug 25, 2012, at 1:32 PM, Mark Struberg > >> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi folks! > >>>>>>>>>>> > >>>>>>>>>>> After a short discussion with Kristian, > >> I've > >>>> created > >>>>>> a small > >>>>>>>> app with 2 > >>>>>>>>>> modules which shows a few problems with > >> mavens > >>>> incremental > >>>>>> build logic. > >>>>>>>>>>> And since incremental builds do not > >> work well, > >>>> people use > >>>>>>>>>>> > >>>>>>>>>>> $> mvn _clean_ install > >>>>>>>>>>> all the time. > >>>>>>>>>>> > >>>>>>>>>>> We could speed up the development > >> effort heavily > >>>> if we > >>>>>> make > >>>>>>>>>>> $> mvn install > >>>>>>>>>>> (without an upfront clean) more > >> reliable. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> The sample [1] consists of moduleA and > >> moduleB > >>>> with BeanA > >>>>>> and > >>>>>>>> BeanB > >>>>>>>>>> respectively. > >>>>>>>>>>> BeanB has a private BeanA field and > >> moduleB has > >>>> a > >>>>>> dependency on > >>>>>>>> moduleA. > >>>>>>>>>>> > >>>>>>>>>>> If I change BeanA and just invoke mvn > >> install > >>>> then only > >>>>>> moduleA > >>>>>>>> gets > >>>>>>>>>> rebuilt. We currently do not rebuild > >> moduleB, but we > >>>> should do > >>>>>> to > >>>>>>>> create a > >>>>>>>>>> reliable output. > >>>>>>>>>>> > >>>>>>>>>>> In fact, the incremental build within a > >> single > >>>> module > >>>>>> already > >>>>>>>> works to some > >>>>>>>>>> degrees, but we must detect a change in > >> dependencies > >>>> and > >>>>>> trigger a full > >>>>>>>> rebuild > >>>>>>>>>> on all depending modules. This could be done > >> by > >>>> storing the > >>>>>> md5 of all > >>>>>>>>>> dependency artifacts and compare them on the > >> next > >>>> build. If > >>>>>> the md5 of > >>>>>>>> a > >>>>>>>>>> dependency did change, then we need to build > >> the > >>>> module full > >>>>>> cycle. > >>>>>>>>>>> Other ideas are welcome. Slaps as well > >> if I > >>>> forgot some > >>>>>> obvious > >>>>>>>> stuff and > >>>>>>>>>> all works well already. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> LieGrue, > >>>>>>>>>>> strub > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> > >>>> --------------------------------------------------------------------- > >>>>>>>>>>> To unsubscribe, e-mail: > >>>> dev-unsubscr...@maven.apache.org > >>>>>>>>>>> For additional commands, e-mail: > >>>>>> dev-h...@maven.apache.org > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>> > >> --------------------------------------------------------------------- > >>>>>>>>>> To unsubscribe, e-mail: > >>>> dev-unsubscr...@maven.apache.org > >>>>>>>>>> For additional commands, e-mail: > >>>> dev-h...@maven.apache.org > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>> > >> --------------------------------------------------------------------- > >>>>>>>>> To unsubscribe, e-mail: > >> dev-unsubscr...@maven.apache.org > >>>>>>>>> For additional commands, e-mail: > >>>> dev-h...@maven.apache.org > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Olivier Lamy > >>>>>>>> Talend: http://coders.talend.com > >>>>>>>> http://twitter.com/olamy | > >> http://linkedin.com/in/olamy > >>>>>>>> > >>>>>>> > >>>>>>> > >>>> --------------------------------------------------------------------- > >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > >>>>>>> For additional commands, e-mail: > >> dev-h...@maven.apache.org > >>>>>>> > >>>>>> > >>>>>> > >> --------------------------------------------------------------------- > >>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > >>>>>> For additional commands, e-mail: dev-h...@maven.apache.org > >>>>>> > >>>>> > >>>>> > >> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > >>>>> For additional commands, e-mail: dev-h...@maven.apache.org > >>>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > >>>> For additional commands, e-mail: dev-h...@maven.apache.org > >>>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > >>> For additional commands, e-mail: dev-h...@maven.apache.org > >>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > >> For additional commands, e-mail: dev-h...@maven.apache.org > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > For additional commands, e-mail: dev-h...@maven.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > >
