So when you say "prune the dependencies", I am not sure what you are
talking about cause what I could think of is not working. let me
explain the situation again. nutch 2.x ivy file ( ivy/ivy.xml ) has this
in it:
<dependencies>
<dependency org="org.elasticsearch" name="elasticsearch" rev="0.19.4"
conf="*->default,sources"/>
<dependency org="org.apache.solr" name="solr-solrj" rev="3.4.0"
conf="*->default" />
<dependency org="org.slf4j" name="slf4j-log4j12" rev="1.6.1"
conf="*->master" />
hadoop 1.1.1 ships with slf4j 1.4.3. these 2 are not compatible. now, i
rather not mess with my hadoop cluster so I tried to downgrade slf4j in
nutch. I changed the above lines to :
<dependencies>
<!--
<dependency org="org.elasticsearch" name="elasticsearch" rev="0.19.4"
conf="*->default,sources"/>
-->
<dependency org="org.apache.zookeeper" name="zookeeper" rev="3.4.5"
conf="*->default"/>
<dependency org="org.apache.solr" name="solr-solrj" rev="3.6.2"
conf="*->default" />
<dependency org="org.slf4j" name="slf4j-log4j12" rev="1.4.3"
as you can see I am upgrading the solr and zookeeper and removing the
elasticsearch, and all of these changes work fine since I can see the
appropriate files in the build/lib directory after ant is done. but it
doesn't work for slf4j, and the files copied to build/lib ( and
subsequently in my job file ) are :
kaveh@d1r2n2:/source/nutch/nutch$ ll build/lib/slf*
-rw-r--r-- 1 kaveh kaveh 25496 Jul 5 2010 build/lib/slf4j-api-1.6.1.jar
-rw-r--r-- 1 kaveh kaveh 9753 Jul 5 2010
build/lib/slf4j-log4j12-1.6.1.jar
since i need the job file i can't just manually change the files in
build/lib, won't do me any good. now I don't know ant very well, and
that is mostly why I am asking this from you guys. I have to say that I
also changed the same thing in pom.xml as well:
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.4.3</version>
<optional>true</optional>
</dependency>
but I still end up with the 1.6.1 version. I don't know how exactly ant
and ivy and pom work together, so I am asking if there is any other
config file that I am missing, or why while it is working fine for solr
and zookeeper it is not affecting the slf4j?
thanks,
On 02/16/2013 09:42 AM, Lewis John Mcgibbney wrote:
A solution would be to manually prune the dependencies which are fetched
via Ivy. If old slf4j dependencies are fetched for Hadoop via Ivy then
maybe we need to make the exclusions explicit within ivy.xml. if you are
able , then please provide a patch which fixes this if it is really a
problem.
It is important to note that pom.xml will most likely be outdated. You
should build nutch with ant + ivy for the time being as this is stable.
Thank you
Lewis
On Saturday, February 16, 2013, kaveh minooie <[email protected]> wrote:
unfortunately your links have been removed from the email that i got so i
am not sure what [0] and [1] are, but this is what i am using :
kaveh@d1r2n2:/source/nutch/nutch.git$ git remote -v
origin https://github.com/apache/nutch.git (fetch)
origin https://github.com/apache/nutch.git (push)
kaveh@d1r2n2:/source/nutch/nutch.git$ git branch -v
* 2.x f02dcf6 NUTCH-XX remove unused db.max.inlinks from
nutch-default.xml
trunk a7a1b41 NUTCH-1521 CrawlDbFilter pass null url to urlNormalizers
kaveh@d1r2n2:/2locos/source/nutch/nutch.git$
i am using branch 2.x
On 02/15/2013 06:02 PM, Lewis John Mcgibbney wrote:
Hi Kaveh,
Two seconds please. First lets set some thing straight.
Nutch trunk is from here [0]
Nutch 2,x is from here [1]
Which one do you use?
On Fri, Feb 15, 2013 at 4:53 PM, kaveh minooie <[email protected]> wrote:
but here is my problem. I tried to build the nutch using ver 1.4.3 of
the
slf4j. i changed the version in both ivy.xml and pom.xml and cleaned my
ivy
cache but ant still fetches the version 1.6.1 when it builds the
project.
what am I missing?
We can progress with the problem once we know what's actually going on.
Thanks
Lewis
