Thanks Lewis
it turned out that slf4j 1.6.1 is being pulled by both hbase 0.90.4
and zookeeper 3.4.5:
Required by
Organisation Name Revision In Configurations Asked Revision
org.apache.nutch nutch working@localhost default, test, master
1.4.3
org.apache.hbase hbase 0.90.4 compile, runtime 1.6.1
org.apache.zookeeper zookeeper 3.4.5 default, compile, runtime,
master 1.6.1
Dependencies
Module Revision Status Resolver Default Licenses Size
log4j by log4j 1.2.15 release maven2 false The Apache Software License,
Version 2.0 0 kB
log4j by log4j 1.2.16 release maven2 false The Apache Software License,
Version 2.0 470 kB
slf4j-api by org.slf4j 1.6.1 release maven2 false 25 kB
I think I am better off upgrading the slf4j across my hadoop cluster :)
thanks,
On 02/18/2013 10:26 PM, Lewis John Mcgibbney wrote:
Hi,
So by the sounds of it the slf4j is being pulled transitively and you need
to determine from where.
You can use the ant report task which makes ivy generate a nice dependency
report for you. You can then see which direct dependency includes slf4j. I
know for a fact that all gora-core dependency will pull it.
Once you've highlighted where it is coming from you can make the relevant
exclusion ok.
On Saturday, February 16, 2013, kaveh minooie <[email protected]> wrote:
So when you say "prune the dependencies", I am not sure what you are
talking about cause what I could think of is not working. let me explain
the situation again. nutch 2.x ivy file ( ivy/ivy.xml ) has this in it:
<dependencies>
<dependency org="org.elasticsearch" name="elasticsearch" rev="0.19.4"
conf="*->default,sources"/>
<dependency org="org.apache.solr" name="solr-solrj" rev="3.4.0"
conf="*->default" />
<dependency org="org.slf4j" name="slf4j-log4j12" rev="1.6.1"
conf="*->master" />
hadoop 1.1.1 ships with slf4j 1.4.3. these 2 are not compatible. now, i
rather not mess with my hadoop cluster so I tried to downgrade slf4j in
nutch. I changed the above lines to :
<dependencies>
<!--
<dependency org="org.elasticsearch" name="elasticsearch" rev="0.19.4"
conf="*->default,sources"/>
-->
<dependency org="org.apache.zookeeper" name="zookeeper" rev="3.4.5"
conf="*->default"/>
<dependency org="org.apache.solr" name="solr-solrj" rev="3.6.2"
conf="*->default" />
<dependency org="org.slf4j" name="slf4j-log4j12" rev="1.4.3"
as you can see I am upgrading the solr and zookeeper and removing the
elasticsearch, and all of these changes work fine since I can see the
appropriate files in the build/lib directory after ant is done. but it
doesn't work for slf4j, and the files copied to build/lib ( and
subsequently in my job file ) are :
kaveh@d1r2n2:/source/nutch/nutch$ ll build/lib/slf*
-rw-r--r-- 1 kaveh kaveh 25496 Jul 5 2010 build/lib/slf4j-api-1.6.1.jar
-rw-r--r-- 1 kaveh kaveh 9753 Jul 5 2010
build/lib/slf4j-log4j12-1.6.1.jar
since i need the job file i can't just manually change the files in
build/lib, won't do me any good. now I don't know ant very well, and that
is mostly why I am asking this from you guys. I have to say that I also
changed the same thing in pom.xml as well:
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.4.3</version>
<optional>true</optional>
</dependency>
but I still end up with the 1.6.1 version. I don't know how exactly ant
and ivy and pom work together, so I am asking if there is any other config
file that I am missing, or why while it is working fine for solr and
zookeeper it is not affecting the slf4j?
thanks,
On 02/16/2013 09:42 AM, Lewis John Mcgibbney wrote:
A solution would be to manually prune the dependencies which are fetched
via Ivy. If old slf4j dependencies are fetched for Hadoop via Ivy then
maybe we need to make the exclusions explicit within ivy.xml. if you are
able , then please provide a patch which fixes this if it is really a
problem.
It is important to note that pom.xml will most likely be outdated. You
should build nutch with ant + ivy for the time being as this is stable.
Thank you
Lewis
On Saturday, February 16, 2013, kaveh minooie <[email protected]> wrote:
unfortunately your links have been removed from the email that i got so
i
am not sure what [0] and [1] are, but this is what i am using :
kaveh@d1r2n2:/source/nutch/nutch.git$ git remote -v
origin https://github.com/apache/nutch.git (fetch)
origin https://github.com/apache/nutch.git (push)
kaveh@d1r2n2:/source/nutch/nutch.git$ git branch -v
* 2.x f02dcf6 NUTCH-XX remove unused db.max.inlinks from
nutch-default.xml
trunk a7a1b41 NUTCH-1521 CrawlDbFilter pass null url to
urlNormalizers
kaveh@d1r2n2:/2locos/source/nutch/nutch.git$
i am using branch 2.x
On 02/15/2013 06:02 PM, Lewis John Mcgibbney wrote:
Hi Kaveh,
Two seconds please. First lets set some thing straight.
Nutch trunk is from here [0]
Nutch 2,x is from here [1]
Which one do you use?
On Fri, Feb 15, 2013 at 4:53 PM, kaveh minooie <[email protected]>
wrote:
but here is my problem. I tried to build the nutch using ver 1.4.3 of
the
slf4j. i changed the version in both ivy.xml and pom.xml and cleaned
my
ivy
cache but ant still fetches the version 1.6.1 when it builds the
project.
what am I missing?
We can progress with the problem once we know what's actually going on.
Thanks
Lewis
--
Kaveh Minooie
www.plutoz.com
