Hi Dennis, Thanks for trying to help.
I don't know what "fresh install" means exactly. Here is what I've done: 1) Downloaded latest version of Nutch from the SVN to a new folder. 2) Copied all the custom plugins I've written to the new folder 3) Edited all configuration files. 4) Executed "ant package". 5) Run the new Nutch... and got this error. What did I miss? Thanks, Eran On Wed, Dec 9, 2009 at 3:36 PM, Dennis Kubes <ku...@apache.org> wrote: > Did you do a fresh install of Nutch with Hadoop 0.20 or did you just copy > over the new jars? The sealing violation is multiple of the same jars being > loaded and the Jetty versions changed between 0.19 and 0.20 for Hadoop? > > Dennis > > > Eran Zinman wrote: > >> Hi Dennis, >> >> 1) I've initially tried to run on my existing DFS and it didn't work. I >> then >> made a backup of my DFS and performed a format and it still didn't work... >> >> 2) I'm using: >> >> java version "1.6.0_0" >> OpenJDK Runtime Environment (IcedTea6 1.4.1) (6b14-1.4.1-0ubuntu12) >> OpenJDK Client VM (build 14.0-b08, mixed mode, sharing) >> >> 3) My environment variables: >> >> ORBIT_SOCKETDIR=/tmp/orbit-eran >> SSH_AGENT_PID=3533 >> GPG_AGENT_INFO=/tmp/seahorse-Gq6lRI/S.gpg-agent:3557:1 >> TERM=xterm >> SHELL=/bin/bash >> >> XDG_SESSION_COOKIE=1a02c2275727547fa7209ad54a91276c-1260199857.905267-2000911890 >> GTK_RC_FILES=/etc/gtk/gtkrc:/home/eran/.gtkrc-1.2-gnome2 >> WINDOWID=54653392 >> GTK_MODULES=canberra-gtk-module >> USER=eran >> >> LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.svgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=0 >> > > 0;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36: > >> GNOME_KEYRING_SOCKET=/tmp/keyring-0Vt0yu/socket >> SSH_AUTH_SOCK=/tmp/keyring-0Vt0yu/socket.ssh >> SESSION_MANAGER=local/eran:/tmp/.ICE-unix/3387 >> USERNAME=eran >> DESKTOP_SESSION=default >> >> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games >> GDM_XSERVER_LOCATION=local >> PWD=/home/eran >> JAVA_HOME=/usr/lib/jvm/default-java/ >> LANG=en_US.UTF-8 >> GDM_LANG=en_US.UTF-8 >> GDMSESSION=default >> HISTCONTROL=ignoreboth >> SHLVL=1 >> HOME=/home/eran >> GNOME_DESKTOP_SESSION_ID=this-is-deprecated >> LOGNAME=eran >> XDG_DATA_DIRS=/usr/local/share/:/usr/share/:/usr/share/gdm/ >> >> DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-E4IJ0hMrD8,guid=c3caaf3e590c65a58904ca7f4b1d1fb3 >> LESSOPEN=| /usr/bin/lesspipe %s >> WINDOWPATH=7 >> DISPLAY=:0.0 >> LESSCLOSE=/usr/bin/lesspipe %s %s >> XAUTHORITY=/home/eran/.Xauthority >> COLORTERM=gnome-terminal >> _=/usr/bin/printenv >> >> Thanks, >> Eran >> >> >> On Wed, Dec 9, 2009 at 2:38 PM, Dennis Kubes <ku...@apache.org> wrote: >> >> 1) Is this a new or existing Hadoop cluster? >>> 2) What Java version are you using and what is your environment? >>> >>> Dennis >>> >>> >>> Eran Zinman wrote: >>> >>> Hi, >>>> >>>> Running new Nutch version status: >>>> >>>> 1. Nutch runs perfectly if Hadoop is disabled (i.e. running in normal >>>> mode). >>>> 2. Nutch doesn't work when I setup it to work with Hadoop either in a >>>> single >>>> or cluster setup. >>>> >>>> *I'm getting an exception: * >>>> ERROR namenode.NameNode - java.lang.SecurityException: sealing >>>> violation: >>>> can't seal package org.mortbay.util: already loaded >>>> >>>> I thought it might be a good idea that I'll attach my Hadoop conf files, >>>> so >>>> here they are: >>>> >>>> *core-site.xml* >>>> <configuration> >>>> <property> >>>> <name>fs.default.name</name> >>>> <value>hdfs://10.0.0.2:9000/</value> >>>> <description> >>>> The name of the default file system. Either the literal string >>>> "local" or a host:port for NDFS. >>>> </description> >>>> </property> >>>> </configuration> >>>> >>>> *mapred-site.xml* >>>> <configuration> >>>> <property> >>>> <name>mapred.job.tracker</name> >>>> <value>10.0.0.2:9001</value> >>>> <description> >>>> The host and port that the MapReduce job tracker runs at. If >>>> "local", then jobs are run in-process as a single map and >>>> reduce task. >>>> </description> >>>> </property> >>>> >>>> <property> >>>> <name>mapred.system.dir</name> >>>> <value>/my_crawler/filesystem/mapreduce/system</value> >>>> </property> >>>> >>>> <property> >>>> <name>mapred.local.dir</name> >>>> <value>/my_crawler/filesystem/mapreduce/local</value> >>>> </property> >>>> </configuration> >>>> >>>> *hdfs-site.xml* >>>> <configuration> >>>> <property> >>>> <name>dfs.name.dir</name> >>>> <value>/my_crawler/filesystem/name</value> >>>> </property> >>>> >>>> <property> >>>> <name>dfs.data.dir</name> >>>> <value>/my_crawler/filesystem/data</value> >>>> </property> >>>> >>>> <property> >>>> <name>dfs.replication</name> >>>> <value>2</value> >>>> </property> >>>> </configuration> >>>> >>>> Thanks, >>>> Eran >>>> >>>> On Wed, Dec 9, 2009 at 12:22 PM, Eran Zinman <zze...@gmail.com> wrote: >>>> >>>> Hi Andrzej, >>>> >>>>> Thanks for your help (as always). >>>>> >>>>> Still getting same exception when running on standalone Hadoop cluster. >>>>> Getting same exceptions as before - also in the datanode log I'm >>>>> getting: >>>>> >>>>> 2009-12-09 12:20:37,805 ERROR datanode.DataNode - java.io.IOException: >>>>> Call >>>>> to 10.0.0.2:9000 failed on local exception: java.io.IOException: >>>>> Connection reset by peer >>>>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:774) >>>>> at org.apache.hadoop.ipc.Client.call(Client.java:742) >>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) >>>>> at $Proxy4.getProtocolVersion(Unknown Source) >>>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) >>>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346) >>>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383) >>>>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314) >>>>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:269) >>>>> at >>>>> >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246) >>>>> at >>>>> >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368) >>>>> Caused by: java.io.IOException: Connection reset by peer >>>>> at sun.nio.ch.FileDispatcher.read0(Native Method) >>>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) >>>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) >>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:206) >>>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) >>>>> at >>>>> >>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) >>>>> at >>>>> >>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) >>>>> at java.io.FilterInputStream.read(FilterInputStream.java:116) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) >>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:237) >>>>> at java.io.DataInputStream.readInt(DataInputStream.java:370) >>>>> at >>>>> >>>>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) >>>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) >>>>> >>>>> Thanks, >>>>> Eran >>>>> >>>>> >>>>> On Wed, Dec 9, 2009 at 12:12 PM, Andrzej Bialecki <a...@getopt.org> >>>>> wrote: >>>>> >>>>> Eran Zinman wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>>> Sorry to bother you guys again, but it seems that no matter what I do >>>>>>> I >>>>>>> can't run the new version of Nutch with Hadoop 0.20. >>>>>>> >>>>>>> I am getting the following exceptions in my logs when I execute >>>>>>> bin/start-all.sh >>>>>>> >>>>>>> Do you use the scripts in place, i.e. without deploying the >>>>>>> nutch*.job >>>>>>> >>>>>> to >>>>>> a separate Hadoop cluster? Could you please try it with a standalone >>>>>> Hadoop >>>>>> cluster (even if it's a pseudo-distributed, i.e. single node)? >>>>>> >>>>>> >>>>>> -- >>>>>> Best regards, >>>>>> Andrzej Bialecki <>< >>>>>> ___. ___ ___ ___ _ _ __________________________________ >>>>>> [__ || __|__/|__||\/| Information Retrieval, Semantic Web >>>>>> ___|||__|| \| || | Embedded Unix, System Integration >>>>>> http://www.sigram.com Contact: info at sigram dot com >>>>>> >>>>>> >>>>>> >>>>>> >>
