build1 and build 2 are different modules or different mvn invocations with some time inbetween?
Of course we will need to test all plugins to behave incremental like! Means the maven-shade-plugin should check itself whether it needs shading or not at all. But the worst thing happening would be that we compile too much. Thats still better than a full mvn clean install :) LieGrue, strub ----- Original Message ----- > From: Benson Margulies <bimargul...@gmail.com> > To: Maven Developers List <dev@maven.apache.org>; Mark Struberg > <strub...@yahoo.de> > Cc: > Sent: Monday, August 27, 2012 5:07 PM > Subject: Re: improving incremental builds > > On Mon, Aug 27, 2012 at 8:40 AM, Mark Struberg <strub...@yahoo.de> wrote: >> We might implement this by storing the list of all activated profiles + all > resolved environment properties in a file in target/ somewhere. >> That would be part of a.) in my previous mail to Romain. > > How about the situation with shade? > > Build 1 runs shade and it replaces the primary build artifact with a shaded > one. > > Build 2 sees that nothing has changed, but shade doesn't know, and it > reshades, eating its own output with a ton of messages. > > Can I fix this in shade with current technologies? > > > >> >> LieGrue, >> strub >> >> >> >> ----- Original Message ----- >>> From: Anders Hammar <and...@hammar.net> >>> To: Maven Developers List <dev@maven.apache.org>; Mark Struberg > <strub...@yahoo.de> >>> Cc: >>> Sent: Monday, August 27, 2012 12:24 PM >>> Subject: Re: improving incremental builds >>> >>>> +0 for auto detecting changed scenarios. If someone changes a > profile or >>> the whole pom, then I'd expect he invokes a clean manually. We > have to >>> document this expectation of course. >>> >>> What do others think of this? I'm thinking it would be awesome (but >>> maybe difficult) if plugins could determine if its configuration has >>> changed, and then handle this automatically. If there are to many >>> cases where a manual clean build is required, I believe people will >>> continue to do "mvn clean install" (which I see all the time > with >>> developers), which I think is what we try to get away from. >>> >>> /Anders >>> >>>> >>>> LieGrue, >>>> strub >>>> >>>> >>>> >>>> >>>> ----- Original Message ----- >>>>> From: Anders Hammar <and...@hammar.net> >>>>> To: Maven Developers List <dev@maven.apache.org> >>>>> Cc: >>>>> Sent: Monday, August 27, 2012 9:57 AM >>>>> Subject: Re: improving incremental builds >>>>> >>>>>> I already started tweaking the m-compiler-plugin and > found quite >>> scary >>>>> issues. There is e.g. MCOMPILER-21 (open since 2005) which > prevents >>> incremental >>>>> build and would kick in in your scenario. >>>>> >>>>> This is interesting. I've looked a bit at Mojo's JAXB > plugin in >>> the >>>>> context of detecting stale generated files (i.e. need to > re-generate) >>>>> and it's similar to this. It would extremely nice if there > would be >>> a >>>>> common "framework" for handling incremental builds. >>>>> In addition to what's been mentioned in the jira (I just > browsed it >>>>> quickly), we have cases when includes/excludes change, the > sourceDir >>>>> changes, etc which should trigger cleanup and re-compilation. >>>>> >>>>> /Anders >>>>> >>>>>> >>>>>> I talked about 2 scenarios >>>>>> >>>>>> a.) all phases >= package, e.g. mvn verify, mvn > install, ... >>> Here we >>>>> have an artifact on the disc and could test for the md5 of > them, right? >>>>>> >>>>>> b.) all phases < package. This is e.g. mvn compile or > mvn test. >>> In that >>>>> case we don't produce a jar/war/ear/... artifact but only > get the >>> files on >>>>> the disk linked in MavenProject#getProjectArtifacts()... > file(). This >>> is the >>>>> case where the maven-compiler-plugin kicks in. I'm > currently in the >>> process >>>>> of rewriting big parts of it, mainly I introduced >>>>>> b.1 a check for project.artifacts/project.testArtifacts > .file() >>> is a >>>>> local file path .isDirectory() and has files which are newer > (actually >>>> =) >>>>> than the buildStarted timestamp. >>>>>> b.2 check whether there are any *.java (sourceincludes) > files >>> which are >>>>> newer than their class files. >>>>>> >>>>>> In both cases I now force a FULL recompile of the module! >>> Compiling only >>>>> parts produced classes which are actually broken! >>>>>> >>>>>> >>>>>> My approach is the following: compiling a bit too much is > better >>> than >>>>> missing some files which need compilation. Because the later > is exactly >>> the >>>>> reason why noone uses mvn compile but always do a mvn clean > compile >>> nowadays. If >>>>> mvn compile is reliable, then we will end up being much faster > than an >>>>> unconditional full compile on the whole project! >>>>>> >>>>>> >>>>>> LieGrue, >>>>>> strub >>>>>> >>>>>> >>>>>> PS: We need to move all our plugins which still use the >>>>> plugin-testing-harness to the maven-invoker-plugin as the > test-harness >>> is broken >>>>> with sisu. (sisu changed the 'container' field from > plexus >>> original: >>>>> protected to 'private') >>>>>> >>>>>> PPS: how do we maintain the plexus-compiler-utils stuff? > This >>> contains some >>>>> weirdo bugs which should get fixed... >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: Olivier Lamy <ol...@apache.org> >>>>>>> To: Maven Developers List > <dev@maven.apache.org>; Mark >>> Struberg >>>>> <strub...@yahoo.de> >>>>>>> Cc: >>>>>>> Sent: Monday, August 27, 2012 9:13 AM >>>>>>> Subject: Re: improving incremental builds >>>>>>> >>>>>>> Hi, >>>>>>> Sounds good idea trying to improve that. >>>>>>> My question: on what is based the md5 calculation ? >>>>>>> If people don't use 'install' but only >>> 'test' >>>>> (perso I use >>>>>>> that to >>>>>>> avoid too much io with jar creation etc..), so in > such case we >>> cannot >>>>>>> do md5 calculation on the produced artifact. >>>>>>> >>>>>>> 2012/8/26 Mark Struberg <strub...@yahoo.de>: >>>>>>>> Hi David! >>>>>>>> >>>>>>>>> So your idea is to find the list of changed > modules >>> and >>>>>>>>> then build that list with -amd? >>>>>>>> Yes, kind of. >>>>>>>> >>>>>>>> At the moment mvn -amd builds the dependents of > the >>> _current_ >>>>> module. If >>>>>>> you use my example and change BeanA.java, then run > mvn -amd >>> from the >>>>> root module >>>>>>> you will see that only moduleA gets rebuild and > moduleB >>> remains >>>>> original. >>>>>>> Because moduleB is not a dependent of the root > module. >>>>>>>> >>>>>>>> But yes, I'm completely with you that we > have most of >>> the >>>>> ingredients >>>>>>> in the maven codebase already. At least for the start > we could >>> easily >>>>> detect >>>>>>> changed build results and add those to the > 'amd' list. >>> That >>>>> would >>>>>>> already be much better than what we have today imo. > And this >>> should be >>>>> pretty >>>>>>> easy to implement. >>>>>>>> >>>>>>>> Indeed, what I proposed goes a bit beyond -amd. > I would >>> not only >>>>> check the >>>>>>> current build tree like -amd does (even if that would > be a >>> good start) >>>>> but >>>>>>> remember the md5 of all the dependencies of each > module >>> (simply store >>>>> them in a >>>>>>> file in ./target/) And if you find a dependency which > is not >>> in the >>>>> list (e.g. >>>>>>> after upgrade from one version to another) or has a > different >>> md5 (this >>>>> covers >>>>>>> SNAPSHOT versions) , then we could force a full > rebuild (mvn >>> -amd) of >>>>> this >>>>>>> module. >>>>>>>> >>>>>>>> >>>>>>>> LieGrue, >>>>>>>> strub >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: David Jencks > <david_jen...@yahoo.com> >>>>>>>>> To: Maven Developers List >>> <dev@maven.apache.org> >>>>>>>>> Cc: >>>>>>>>> Sent: Sunday, August 26, 2012 8:34 AM >>>>>>>>> Subject: Re: improving incremental builds >>>>>>>>> >>>>>>>>> Is what you want different from what >>>>>>>>> >>>>>>>>> mvn -amd moduleA >>>>>>>>> >>>>>>>>> does? So your idea is to find the list of > changed >>> modules and >>>>> then >>>>>>> build that >>>>>>>>> list with -amd? >>>>>>>>> >>>>>>>>> thanks >>>>>>>>> david jencks >>>>>>>>> >>>>>>>>> On Aug 25, 2012, at 1:32 PM, Mark Struberg > wrote: >>>>>>>>> >>>>>>>>>> Hi folks! >>>>>>>>>> >>>>>>>>>> After a short discussion with Kristian, > I've >>> created >>>>> a small >>>>>>> app with 2 >>>>>>>>> modules which shows a few problems with > mavens >>> incremental >>>>> build logic. >>>>>>>>>> And since incremental builds do not > work well, >>> people use >>>>>>>>>> >>>>>>>>>> $> mvn _clean_ install >>>>>>>>>> all the time. >>>>>>>>>> >>>>>>>>>> We could speed up the development > effort heavily >>> if we >>>>> make >>>>>>>>>> $> mvn install >>>>>>>>>> (without an upfront clean) more > reliable. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The sample [1] consists of moduleA and > moduleB >>> with BeanA >>>>> and >>>>>>> BeanB >>>>>>>>> respectively. >>>>>>>>>> BeanB has a private BeanA field and > moduleB has >>> a >>>>> dependency on >>>>>>> moduleA. >>>>>>>>>> >>>>>>>>>> If I change BeanA and just invoke mvn > install >>> then only >>>>> moduleA >>>>>>> gets >>>>>>>>> rebuilt. We currently do not rebuild > moduleB, but we >>> should do >>>>> to >>>>>>> create a >>>>>>>>> reliable output. >>>>>>>>>> >>>>>>>>>> In fact, the incremental build within a > single >>> module >>>>> already >>>>>>> works to some >>>>>>>>> degrees, but we must detect a change in > dependencies >>> and >>>>> trigger a full >>>>>>> rebuild >>>>>>>>> on all depending modules. This could be done > by >>> storing the >>>>> md5 of all >>>>>>>>> dependency artifacts and compare them on the > next >>> build. If >>>>> the md5 of >>>>>>> a >>>>>>>>> dependency did change, then we need to build > the >>> module full >>>>> cycle. >>>>>>>>>> Other ideas are welcome. Slaps as well > if I >>> forgot some >>>>> obvious >>>>>>> stuff and >>>>>>>>> all works well already. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> LieGrue, >>>>>>>>>> strub >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>> >>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: >>> dev-unsubscr...@maven.apache.org >>>>>>>>>> For additional commands, e-mail: >>>>> dev-h...@maven.apache.org >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> > --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: >>> dev-unsubscr...@maven.apache.org >>>>>>>>> For additional commands, e-mail: >>> dev-h...@maven.apache.org >>>>>>>>> >>>>>>>> >>>>>>>> >>>>> > --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: > dev-unsubscr...@maven.apache.org >>>>>>>> For additional commands, e-mail: >>> dev-h...@maven.apache.org >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Olivier Lamy >>>>>>> Talend: http://coders.talend.com >>>>>>> http://twitter.com/olamy | > http://linkedin.com/in/olamy >>>>>>> >>>>>> >>>>>> >>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>>>> For additional commands, e-mail: > dev-h...@maven.apache.org >>>>>> >>>>> >>>>> > --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>>> For additional commands, e-mail: dev-h...@maven.apache.org >>>>> >>>> >>>> > --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>>> For additional commands, e-mail: dev-h...@maven.apache.org >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >>> For additional commands, e-mail: dev-h...@maven.apache.org >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> For additional commands, e-mail: dev-h...@maven.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
