On 27 August 2012 19:10, Stephen Connolly <stephen.alan.conno...@gmail.com>wrote:
> On 27 August 2012 18:09, Benson Margulies <bimargul...@gmail.com> wrote: > >> On Mon, Aug 27, 2012 at 12:07 PM, Mark Struberg <strub...@yahoo.de> >> wrote: >> > build1 and build 2 are different modules or different mvn invocations >> with some time inbetween? >> > >> > Of course we will need to test all plugins to behave incremental like! >> Means the maven-shade-plugin should check itself whether it needs shading >> or not at all. >> >> Two builds, separated in time, of exactly the same module. >> >> The problem here is that the jar plugin has already decided not to >> rebuild the jar by the time the shade plugin gets there, and I have no >> idea how to make the shade plugin play along. >> > > Any chance we can get the jar plugin to inspect some of the artifacts that > it generates into the jar to see if they are consistent with what the jar > plugin produces... > > I'm looking at you META-INF/MANIFEST.MF > > I.O.W. if the > > Created-By: Apache Maven 3.0.4 (Maven JAR Plugin) > > is switched to > > Created-By: Apache Maven 3.0.4 (Maven Shade Plugin) > > Then the Maven JAR plugin can infer that the timestamp check is pointless, > and force regen > > And people customizing that header can just live with the issues that creates. > -Stephen > > >> >> >> > >> > But the worst thing happening would be that we compile too much. Thats >> still better than a full mvn clean install :) >> > >> > LieGrue, >> > strub >> > >> > >> > >> > ----- Original Message ----- >> >> From: Benson Margulies <bimargul...@gmail.com> >> >> To: Maven Developers List <dev@maven.apache.org>; Mark Struberg < >> strub...@yahoo.de> >> >> Cc: >> >> Sent: Monday, August 27, 2012 5:07 PM >> >> Subject: Re: improving incremental builds >> >> >> >> On Mon, Aug 27, 2012 at 8:40 AM, Mark Struberg <strub...@yahoo.de> >> wrote: >> >>> We might implement this by storing the list of all activated >> profiles + all >> >> resolved environment properties in a file in target/ somewhere. >> >>> That would be part of a.) in my previous mail to Romain. >> >> >> >> How about the situation with shade? >> >> >> >> Build 1 runs shade and it replaces the primary build artifact with a >> shaded one. >> >> >> >> Build 2 sees that nothing has changed, but shade doesn't know, and it >> >> reshades, eating its own output with a ton of messages. >> >> >> >> Can I fix this in shade with current technologies? >> >> >> >> >> >> >> >>> >> >>> LieGrue, >> >>> strub >> >>> >> >>> >> >>> >> >>> ----- Original Message ----- >> >>>> From: Anders Hammar <and...@hammar.net> >> >>>> To: Maven Developers List <dev@maven.apache.org>; Mark Struberg >> >> <strub...@yahoo.de> >> >>>> Cc: >> >>>> Sent: Monday, August 27, 2012 12:24 PM >> >>>> Subject: Re: improving incremental builds >> >>>> >> >>>>> +0 for auto detecting changed scenarios. If someone changes a >> >> profile or >> >>>> the whole pom, then I'd expect he invokes a clean manually. We >> >> have to >> >>>> document this expectation of course. >> >>>> >> >>>> What do others think of this? I'm thinking it would be awesome (but >> >>>> maybe difficult) if plugins could determine if its configuration has >> >>>> changed, and then handle this automatically. If there are to many >> >>>> cases where a manual clean build is required, I believe people will >> >>>> continue to do "mvn clean install" (which I see all the time >> >> with >> >>>> developers), which I think is what we try to get away from. >> >>>> >> >>>> /Anders >> >>>> >> >>>>> >> >>>>> LieGrue, >> >>>>> strub >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> ----- Original Message ----- >> >>>>>> From: Anders Hammar <and...@hammar.net> >> >>>>>> To: Maven Developers List <dev@maven.apache.org> >> >>>>>> Cc: >> >>>>>> Sent: Monday, August 27, 2012 9:57 AM >> >>>>>> Subject: Re: improving incremental builds >> >>>>>> >> >>>>>>> I already started tweaking the m-compiler-plugin and >> >> found quite >> >>>> scary >> >>>>>> issues. There is e.g. MCOMPILER-21 (open since 2005) which >> >> prevents >> >>>> incremental >> >>>>>> build and would kick in in your scenario. >> >>>>>> >> >>>>>> This is interesting. I've looked a bit at Mojo's JAXB >> >> plugin in >> >>>> the >> >>>>>> context of detecting stale generated files (i.e. need to >> >> re-generate) >> >>>>>> and it's similar to this. It would extremely nice if there >> >> would be >> >>>> a >> >>>>>> common "framework" for handling incremental builds. >> >>>>>> In addition to what's been mentioned in the jira (I just >> >> browsed it >> >>>>>> quickly), we have cases when includes/excludes change, the >> >> sourceDir >> >>>>>> changes, etc which should trigger cleanup and re-compilation. >> >>>>>> >> >>>>>> /Anders >> >>>>>> >> >>>>>>> >> >>>>>>> I talked about 2 scenarios >> >>>>>>> >> >>>>>>> a.) all phases >= package, e.g. mvn verify, mvn >> >> install, ... >> >>>> Here we >> >>>>>> have an artifact on the disc and could test for the md5 of >> >> them, right? >> >>>>>>> >> >>>>>>> b.) all phases < package. This is e.g. mvn compile or >> >> mvn test. >> >>>> In that >> >>>>>> case we don't produce a jar/war/ear/... artifact but only >> >> get the >> >>>> files on >> >>>>>> the disk linked in MavenProject#getProjectArtifacts()... >> >> file(). This >> >>>> is the >> >>>>>> case where the maven-compiler-plugin kicks in. I'm >> >> currently in the >> >>>> process >> >>>>>> of rewriting big parts of it, mainly I introduced >> >>>>>>> b.1 a check for project.artifacts/project.testArtifacts >> >> .file() >> >>>> is a >> >>>>>> local file path .isDirectory() and has files which are newer >> >> (actually >> >>>>> =) >> >>>>>> than the buildStarted timestamp. >> >>>>>>> b.2 check whether there are any *.java (sourceincludes) >> >> files >> >>>> which are >> >>>>>> newer than their class files. >> >>>>>>> >> >>>>>>> In both cases I now force a FULL recompile of the module! >> >>>> Compiling only >> >>>>>> parts produced classes which are actually broken! >> >>>>>>> >> >>>>>>> >> >>>>>>> My approach is the following: compiling a bit too much is >> >> better >> >>>> than >> >>>>>> missing some files which need compilation. Because the later >> >> is exactly >> >>>> the >> >>>>>> reason why noone uses mvn compile but always do a mvn clean >> >> compile >> >>>> nowadays. If >> >>>>>> mvn compile is reliable, then we will end up being much faster >> >> than an >> >>>>>> unconditional full compile on the whole project! >> >>>>>>> >> >>>>>>> >> >>>>>>> LieGrue, >> >>>>>>> strub >> >>>>>>> >> >>>>>>> >> >>>>>>> PS: We need to move all our plugins which still use the >> >>>>>> plugin-testing-harness to the maven-invoker-plugin as the >> >> test-harness >> >>>> is broken >> >>>>>> with sisu. (sisu changed the 'container' field from >> >> plexus >> >>>> original: >> >>>>>> protected to 'private') >> >>>>>>> >> >>>>>>> PPS: how do we maintain the plexus-compiler-utils stuff? >> >> This >> >>>> contains some >> >>>>>> weirdo bugs which should get fixed... >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> ----- Original Message ----- >> >>>>>>>> From: Olivier Lamy <ol...@apache.org> >> >>>>>>>> To: Maven Developers List >> >> <dev@maven.apache.org>; Mark >> >>>> Struberg >> >>>>>> <strub...@yahoo.de> >> >>>>>>>> Cc: >> >>>>>>>> Sent: Monday, August 27, 2012 9:13 AM >> >>>>>>>> Subject: Re: improving incremental builds >> >>>>>>>> >> >>>>>>>> Hi, >> >>>>>>>> Sounds good idea trying to improve that. >> >>>>>>>> My question: on what is based the md5 calculation ? >> >>>>>>>> If people don't use 'install' but only >> >>>> 'test' >> >>>>>> (perso I use >> >>>>>>>> that to >> >>>>>>>> avoid too much io with jar creation etc..), so in >> >> such case we >> >>>> cannot >> >>>>>>>> do md5 calculation on the produced artifact. >> >>>>>>>> >> >>>>>>>> 2012/8/26 Mark Struberg <strub...@yahoo.de>: >> >>>>>>>>> Hi David! >> >>>>>>>>> >> >>>>>>>>>> So your idea is to find the list of changed >> >> modules >> >>>> and >> >>>>>>>>>> then build that list with -amd? >> >>>>>>>>> Yes, kind of. >> >>>>>>>>> >> >>>>>>>>> At the moment mvn -amd builds the dependents of >> >> the >> >>>> _current_ >> >>>>>> module. If >> >>>>>>>> you use my example and change BeanA.java, then run >> >> mvn -amd >> >>>> from the >> >>>>>> root module >> >>>>>>>> you will see that only moduleA gets rebuild and >> >> moduleB >> >>>> remains >> >>>>>> original. >> >>>>>>>> Because moduleB is not a dependent of the root >> >> module. >> >>>>>>>>> >> >>>>>>>>> But yes, I'm completely with you that we >> >> have most of >> >>>> the >> >>>>>> ingredients >> >>>>>>>> in the maven codebase already. At least for the start >> >> we could >> >>>> easily >> >>>>>> detect >> >>>>>>>> changed build results and add those to the >> >> 'amd' list. >> >>>> That >> >>>>>> would >> >>>>>>>> already be much better than what we have today imo. >> >> And this >> >>>> should be >> >>>>>> pretty >> >>>>>>>> easy to implement. >> >>>>>>>>> >> >>>>>>>>> Indeed, what I proposed goes a bit beyond -amd. >> >> I would >> >>>> not only >> >>>>>> check the >> >>>>>>>> current build tree like -amd does (even if that would >> >> be a >> >>>> good start) >> >>>>>> but >> >>>>>>>> remember the md5 of all the dependencies of each >> >> module >> >>>> (simply store >> >>>>>> them in a >> >>>>>>>> file in ./target/) And if you find a dependency which >> >> is not >> >>>> in the >> >>>>>> list (e.g. >> >>>>>>>> after upgrade from one version to another) or has a >> >> different >> >>>> md5 (this >> >>>>>> covers >> >>>>>>>> SNAPSHOT versions) , then we could force a full >> >> rebuild (mvn >> >>>> -amd) of >> >>>>>> this >> >>>>>>>> module. >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> LieGrue, >> >>>>>>>>> strub >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> ----- Original Message ----- >> >>>>>>>>>> From: David Jencks >> >> <david_jen...@yahoo.com> >> >>>>>>>>>> To: Maven Developers List >> >>>> <dev@maven.apache.org> >> >>>>>>>>>> Cc: >> >>>>>>>>>> Sent: Sunday, August 26, 2012 8:34 AM >> >>>>>>>>>> Subject: Re: improving incremental builds >> >>>>>>>>>> >> >>>>>>>>>> Is what you want different from what >> >>>>>>>>>> >> >>>>>>>>>> mvn -amd moduleA >> >>>>>>>>>> >> >>>>>>>>>> does? So your idea is to find the list of >> >> changed >> >>>> modules and >> >>>>>> then >> >>>>>>>> build that >> >>>>>>>>>> list with -amd? >> >>>>>>>>>> >> >>>>>>>>>> thanks >> >>>>>>>>>> david jencks >> >>>>>>>>>> >> >>>>>>>>>> On Aug 25, 2012, at 1:32 PM, Mark Struberg >> >> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Hi folks! >> >>>>>>>>>>> >> >>>>>>>>>>> After a short discussion with Kristian, >> >> I've >> >>>> created >> >>>>>> a small >> >>>>>>>> app with 2 >> >>>>>>>>>> modules which shows a few problems with >> >> mavens >> >>>> incremental >> >>>>>> build logic. >> >>>>>>>>>>> And since incremental builds do not >> >> work well, >> >>>> people use >> >>>>>>>>>>> >> >>>>>>>>>>> $> mvn _clean_ install >> >>>>>>>>>>> all the time. >> >>>>>>>>>>> >> >>>>>>>>>>> We could speed up the development >> >> effort heavily >> >>>> if we >> >>>>>> make >> >>>>>>>>>>> $> mvn install >> >>>>>>>>>>> (without an upfront clean) more >> >> reliable. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> The sample [1] consists of moduleA and >> >> moduleB >> >>>> with BeanA >> >>>>>> and >> >>>>>>>> BeanB >> >>>>>>>>>> respectively. >> >>>>>>>>>>> BeanB has a private BeanA field and >> >> moduleB has >> >>>> a >> >>>>>> dependency on >> >>>>>>>> moduleA. >> >>>>>>>>>>> >> >>>>>>>>>>> If I change BeanA and just invoke mvn >> >> install >> >>>> then only >> >>>>>> moduleA >> >>>>>>>> gets >> >>>>>>>>>> rebuilt. We currently do not rebuild >> >> moduleB, but we >> >>>> should do >> >>>>>> to >> >>>>>>>> create a >> >>>>>>>>>> reliable output. >> >>>>>>>>>>> >> >>>>>>>>>>> In fact, the incremental build within a >> >> single >> >>>> module >> >>>>>> already >> >>>>>>>> works to some >> >>>>>>>>>> degrees, but we must detect a change in >> >> dependencies >> >>>> and >> >>>>>> trigger a full >> >>>>>>>> rebuild >> >>>>>>>>>> on all depending modules. This could be done >> >> by >> >>>> storing the >> >>>>>> md5 of all >> >>>>>>>>>> dependency artifacts and compare them on the >> >> next >> >>>> build. If >> >>>>>> the md5 of >> >>>>>>>> a >> >>>>>>>>>> dependency did change, then we need to build >> >> the >> >>>> module full >> >>>>>> cycle. >> >>>>>>>>>>> Other ideas are welcome. Slaps as well >> >> if I >> >>>> forgot some >> >>>>>> obvious >> >>>>>>>> stuff and >> >>>>>>>>>> all works well already. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> LieGrue, >> >>>>>>>>>>> strub >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>> >> >>>> >> --------------------------------------------------------------------- >> >>>>>>>>>>> To unsubscribe, e-mail: >> >>>> dev-unsubscr...@maven.apache.org >> >>>>>>>>>>> For additional commands, e-mail: >> >>>>>> dev-h...@maven.apache.org >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>> >> >> --------------------------------------------------------------------- >> >>>>>>>>>> To unsubscribe, e-mail: >> >>>> dev-unsubscr...@maven.apache.org >> >>>>>>>>>> For additional commands, e-mail: >> >>>> dev-h...@maven.apache.org >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>> >> >> --------------------------------------------------------------------- >> >>>>>>>>> To unsubscribe, e-mail: >> >> dev-unsubscr...@maven.apache.org >> >>>>>>>>> For additional commands, e-mail: >> >>>> dev-h...@maven.apache.org >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> Olivier Lamy >> >>>>>>>> Talend: http://coders.talend.com >> >>>>>>>> http://twitter.com/olamy | >> >> http://linkedin.com/in/olamy >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>> >> --------------------------------------------------------------------- >> >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> >>>>>>> For additional commands, e-mail: >> >> dev-h...@maven.apache.org >> >>>>>>> >> >>>>>> >> >>>>>> >> >> --------------------------------------------------------------------- >> >>>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> >>>>>> For additional commands, e-mail: dev-h...@maven.apache.org >> >>>>>> >> >>>>> >> >>>>> >> >> --------------------------------------------------------------------- >> >>>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> >>>>> For additional commands, e-mail: dev-h...@maven.apache.org >> >>>>> >> >>>> >> >>>> >> --------------------------------------------------------------------- >> >>>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> >>>> For additional commands, e-mail: dev-h...@maven.apache.org >> >>>> >> >>> >> >>> --------------------------------------------------------------------- >> >>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> >>> For additional commands, e-mail: dev-h...@maven.apache.org >> >>> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> >> For additional commands, e-mail: dev-h...@maven.apache.org >> >> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> > For additional commands, e-mail: dev-h...@maven.apache.org >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> For additional commands, e-mail: dev-h...@maven.apache.org >> >> >
