Thanks!. 25 seconds is a winner. can decrease it down to 15 if re-compile of parent, math and mojo is turned off.
On Fri, Feb 5, 2010 at 10:49 PM, Drew Farris <drew.far...@gmail.com> wrote: > So, I'm running: mvn -o install -DskipTests=true at project root (in > mahout) > > Comment out or remove the maven-assembly-plugin definition in > core/pom.xml -- it reduced my core build time from 26s to 6s -- I can > submit a patch for this. > > Mahout math is still 17s here due to code generation. I'm wondering if > there's a way to modify the generation plugin to that it doesn't > re-generate if there are no changes to the templates. You can remove > the plugin definition from math/pom.xml and it doesn't seem to break > anything unless you're doing a clean. Brings down math compilation to > 3s without it. Total compile time is 22s. > > re: the job, I'll have to look into that further later. > > On Fri, Feb 5, 2010 at 12:06 PM, Robin Anil <robin.a...@gmail.com> wrote: > > Yes for editing i use eclipse in the same fashion. If i want to try out a > > job and see how it performs on hadoop I need job compiled fast. > > > > On another note. I think there will be a lot of dead code in the job(with > > all the jar files bundles) Is there an optimiser for that i.e to remove > > classes which mahout never use indirectly > > > > I see loading jar takes 10-20 seconds when initializing mapper or > reducer. > > It doesnt affect long running jobs but 20 sec overhead for processing > 64MB > > chunk sucks > > > > On Fri, Feb 5, 2010 at 10:19 PM, Ted Dunning <ted.dunn...@gmail.com> > wrote: > > > >> I usually do an initial compilation using mvn package. Then, during > >> development I use IntelliJ's incremental compilation which generally > only > >> takes a few seconds. Since that compilation doesn't handle things like > >> copying resources, I get caught out and surprised now and again, but > this > >> works almost all the time. > >> > >> On Fri, Feb 5, 2010 at 12:27 AM, Robin Anil <robin.a...@gmail.com> > wrote: > >> > >> > When developing mahout core/util/examples we dont need to generate > math > >> > often and dont need to tar gzip bzip2 the jar files. We are mostly > >> > concerned > >> > with the job file/ jar file. > >> > Cant there be another target like develop which does this. (waiting > 2-3 > >> > mins > >> > for a 2 line change is frustrating) > >> > > >> > Robin > >> > > >> > >> > >> > >> -- > >> Ted Dunning, CTO > >> DeepDyve > >> > > >
