Re: Any plans to move to build Nutchusing Maven?
Hi Steven, On 8/16/06 7:36 AM, steven shingler [EMAIL PROTECTED] wrote: (This thread moved from the User List.) OK Lukas, lets open it up to the dev list! :) Particularly, does the group feel moving to Maven would be _a good thing_ ? +1 I suggested this (however did not make any progress on realizing it ;) ) a while back. I think it makes a * lot of sense *. Maven's dependency system would significantly reduce the size of the CM'ed Nutch source code, as all the jars required by Nutch could be referenced externally (plugins are a different beast, but we're working on that). Additionally, Maven would allow automatic generation of a soft of nightly build Nutch site, showing recent commits, unit test results and more. Even if so, what are the problems? The main problem I see is the plugin system, and how to appropriate represent plugin dependencies in Maven (or just neglect to elegantly handle them, and treat them like invididual projects, like nutch, which requires CM'ing jar files). Additionally, I think it will probably require writing some custom Jelly scripts to do all the neat ant build stuff that Nutch does on the side (e.g., unpack Hadoop, etc.). There are currently two versions of Lucene in the Maven repos, but Hadoop would have to be added manually, I think. It would probably make most sense to run a Maven repo explicitly for Nutch off of the Lucene Nutch site. Something like (http://lucene.apache.org/nutch/maven/) might be sensible. Just my 2 cents. Cheers, Chris All thoughts gratefully received. Cheers Steven On 8/16/06, Lukas Vlcek [EMAIL PROTECTED] wrote: Hi, I would like to help. But first of all I would suggest to start wider discussion in dev list to get more feedback/suggestions. I think one problem can be that Nutch depends on both Lucene and Hadoop libraries and it won't be easy to maintain these dependencies if recent versions are not yet committed into some maven accesible repo. Regards, Lukas On 8/16/06, steven shingler [EMAIL PROTECTED] wrote: Well I'm up for giving it a try. My current work has me looking at both Nutch and Maven, so what better way to understand both projects :) I agree it is far from trivial - so if anyone here would like to collaborate on it, that would be great. Cheers, Steven On 8/15/06, Lukas Vlcek [EMAIL PROTECTED] wrote: Hi, I would warmly appreciate this activity. At least it would help more people to understand/join this great project. But I don't think this will be an easy step (this reminds me what N.Armstrong said on moon: That's one small step for [a] man, one giant leap for mankind.) :-) Regards, Lukas On 8/15/06, Sami Siren [EMAIL PROTECTED] wrote: steven shingler wrote: Hi all, I know this has come up at least once before, but I just thought I'd raise the question again: Are there any plans to move to building Nutch using Maven? Haven't heard of such activities, however if you or somebody else can put such thing together and it proves to be a good thing to do then I certainly don't have anything against it. -- Sami Siren
Re: Any plans to move to build Nutchusing Maven?
I am an outsider, I hope my comments do not cause grief. AFAIK, Maven is transitioning to version 2. The newer version is more attractive, but noticeably different than version 1 in implementation but, closer to home, in configuration. It might be wise to explore some of the comments from other, open sourced, projects regarding maven (about a month ago I was researching a maven problem and bumped into some negative comments that were unexpected.) This is a big step. Due diligence before commitment might be beneficial. With all those negatives, I want maven, especially 2, to be successful. I believe it will solve many tedious problems. Bill
Re: Any plans to move to build Nutchusing Maven?
Hi, I have almost no experience with maven subprojects but somehow I feel this could help us with Nutch plugins. Am I correct? In maven we can always call ant goals as well and Jelly is a fun to use. With maven one of the biggest benefit would be that eclipse (or other IDE) classpath settings hell will be over which could help naive users (like me) a lot. (and silently I hope that what I have said now holds for M2 too) Regards, Lukas On 8/16/06, William Surowiec [EMAIL PROTECTED] wrote: I am an outsider, I hope my comments do not cause grief. AFAIK, Maven is transitioning to version 2. The newer version is more attractive, but noticeably different than version 1 in implementation but, closer to home, in configuration. It might be wise to explore some of the comments from other, open sourced, projects regarding maven (about a month ago I was researching a maven problem and bumped into some negative comments that were unexpected.) This is a big step. Due diligence before commitment might be beneficial. With all those negatives, I want maven, especially 2, to be successful. I believe it will solve many tedious problems. Bill
Re: Any plans to move to build Nutchusing Maven?
Lukas Vlcek wrote: Hi, I have almost no experience with maven subprojects but somehow I feel this could help us with Nutch plugins. Am I correct? In maven we can always call ant goals as well and Jelly is a fun to use. With maven one of the biggest benefit would be that eclipse (or other IDE) classpath settings hell will be over which could help naive users (like me) a lot. (and silently I hope that what I have said now holds for M2 too) Just for clarification when I was talking about maven I meant maven 2, maven 1 is a dead end. -- Sami Siren
Re: Any plans to move to build Nutchusing Maven?
Le Mercredi 16 Août 2006 17:18, Sami Siren a écrit : Lukas Vlcek wrote: Hi, I have almost no experience with maven subprojects but somehow I feel this could help us with Nutch plugins. Am I correct? In maven we can always call ant goals as well and Jelly is a fun to use. With maven one of the biggest benefit would be that eclipse (or other IDE) classpath settings hell will be over which could help naive users (like me) a lot. (and silently I hope that what I have said now holds for M2 too) Just for clarification when I was talking about maven I meant maven 2, maven 1 is a dead end. Hi, I am an outsider too, but my experience with maven 2 was not as awful as expected. The problem with maven is that you would depend enormously of the repository service. If it's down, just hope that you have download everything before, otherwise you have to wait until it's up again. Two or three month ago, the main repositories were down, and even more, one was corrupted (because of a disk failure or something like that, I don't remember). So our team of 5 people lost 3 days... The ASF Cocoon project has mooved to maven and now the README explains that : Build times will vary dramatically depending on how good your connectivity to the Maven repository/mirror is, and how well it performs, and on whether you already have a loaded local Maven repository. A full build can take from 5 minutes to maybe 4 hours... http://svn.apache.org/viewvc/cocoon/trunk/README.txt?view=markup Now, for our ex-maven2ish project, we use a combinaison of ant tasks and ivy for the dependencies (http://jayasoft.org/ivy). The difference between maven and ivy is that ivy runs completely offline, and maven download only on demand. So if you do a svn up on your project, you probably immediately want to do a mvn install to download every dependencies. But you also should run every maven task before getting offline : maven jetty6:run, maven source:jar, maven deploy... As I said, I am a completely outsider, I am not a Nutch user, just a Lucene user that read sometimes this dev list to read some interesting discussion about search. I just wanted to add my unlucky maven user experience. Here are some notes about Sylvain, one of my coworker that had some trouble with maven : http://bluxte.net/blog/2006-05/21-25-04.html http://bluxte.net/blog/2006-06/30-32-27.html Nicolas
Re: Any plans to move to build Nutchusing Maven?
Hi, I have just noticed that there is going on some activity called Tika (http://code.google.com/p/tika/) and these guys are starting directly with Maven2 (am I right?). The more Nutch/Lucene/Hadoop/[Tika]/[?] thing grows the more sophisticated project management tool will be needed I think. Lukas On 8/16/06, Nicolas Lalevée [EMAIL PROTECTED] wrote: Le Mercredi 16 Août 2006 17:18, Sami Siren a écrit: Lukas Vlcek wrote: Hi, I have almost no experience with maven subprojects but somehow I feel this could help us with Nutch plugins. Am I correct? In maven we can always call ant goals as well and Jelly is a fun to use. With maven one of the biggest benefit would be that eclipse (or other IDE) classpath settings hell will be over which could help naive users (like me) a lot. (and silently I hope that what I have said now holds for M2 too) Just for clarification when I was talking about maven I meant maven 2, maven 1 is a dead end. Hi, I am an outsider too, but my experience with maven 2 was not as awful as expected. The problem with maven is that you would depend enormously of the repository service. If it's down, just hope that you have download everything before, otherwise you have to wait until it's up again. Two or three month ago, the main repositories were down, and even more, one was corrupted (because of a disk failure or something like that, I don't remember). So our team of 5 people lost 3 days... The ASF Cocoon project has mooved to maven and now the README explains that : Build times will vary dramatically depending on how good your connectivity to the Maven repository/mirror is, and how well it performs, and on whether you already have a loaded local Maven repository. A full build can take from 5 minutes to maybe 4 hours... http://svn.apache.org/viewvc/cocoon/trunk/README.txt?view=markup Now, for our ex-maven2ish project, we use a combinaison of ant tasks and ivy for the dependencies (http://jayasoft.org/ivy). The difference between maven and ivy is that ivy runs completely offline, and maven download only on demand. So if you do a svn up on your project, you probably immediately want to do a mvn install to download every dependencies. But you also should run every maven task before getting offline : maven jetty6:run, maven source:jar, maven deploy... As I said, I am a completely outsider, I am not a Nutch user, just a Lucene user that read sometimes this dev list to read some interesting discussion about search. I just wanted to add my unlucky maven user experience. Here are some notes about Sylvain, one of my coworker that had some trouble with maven : http://bluxte.net/blog/2006-05/21-25-04.html http://bluxte.net/blog/2006-06/30-32-27.html Nicolas
