Did you do a fresh install of Nutch with Hadoop 0.20 or did you just
copy over the new jars? The sealing violation is multiple of the same
jars being loaded and the Jetty versions changed between 0.19 and 0.20
for Hadoop?
Dennis
Eran Zinman wrote:
Hi Dennis,
1) I've initially tried to run on my existing DFS and it didn't work. I then
made a backup of my DFS and performed a format and it still didn't work...
2) I'm using:
java version "1.6.0_0"
OpenJDK Runtime Environment (IcedTea6 1.4.1) (6b14-1.4.1-0ubuntu12)
OpenJDK Client VM (build 14.0-b08, mixed mode, sharing)
3) My environment variables:
ORBIT_SOCKETDIR=/tmp/orbit-eran
SSH_AGENT_PID=3533
GPG_AGENT_INFO=/tmp/seahorse-Gq6lRI/S.gpg-agent:3557:1
TERM=xterm
SHELL=/bin/bash
XDG_SESSION_COOKIE=1a02c2275727547fa7209ad54a91276c-1260199857.905267-2000911890
GTK_RC_FILES=/etc/gtk/gtkrc:/home/eran/.gtkrc-1.2-gnome2
WINDOWID=54653392
GTK_MODULES=canberra-gtk-module
USER=eran
LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.svgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=0
0;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:
GNOME_KEYRING_SOCKET=/tmp/keyring-0Vt0yu/socket
SSH_AUTH_SOCK=/tmp/keyring-0Vt0yu/socket.ssh
SESSION_MANAGER=local/eran:/tmp/.ICE-unix/3387
USERNAME=eran
DESKTOP_SESSION=default
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
GDM_XSERVER_LOCATION=local
PWD=/home/eran
JAVA_HOME=/usr/lib/jvm/default-java/
LANG=en_US.UTF-8
GDM_LANG=en_US.UTF-8
GDMSESSION=default
HISTCONTROL=ignoreboth
SHLVL=1
HOME=/home/eran
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
LOGNAME=eran
XDG_DATA_DIRS=/usr/local/share/:/usr/share/:/usr/share/gdm/
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-E4IJ0hMrD8,guid=c3caaf3e590c65a58904ca7f4b1d1fb3
LESSOPEN=| /usr/bin/lesspipe %s
WINDOWPATH=7
DISPLAY=:0.0
LESSCLOSE=/usr/bin/lesspipe %s %s
XAUTHORITY=/home/eran/.Xauthority
COLORTERM=gnome-terminal
_=/usr/bin/printenv
Thanks,
Eran
On Wed, Dec 9, 2009 at 2:38 PM, Dennis Kubes <[email protected]> wrote:
1) Is this a new or existing Hadoop cluster?
2) What Java version are you using and what is your environment?
Dennis
Eran Zinman wrote:
Hi,
Running new Nutch version status:
1. Nutch runs perfectly if Hadoop is disabled (i.e. running in normal
mode).
2. Nutch doesn't work when I setup it to work with Hadoop either in a
single
or cluster setup.
*I'm getting an exception: *
ERROR namenode.NameNode - java.lang.SecurityException: sealing violation:
can't seal package org.mortbay.util: already loaded
I thought it might be a good idea that I'll attach my Hadoop conf files,
so
here they are:
*core-site.xml*
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://10.0.0.2:9000/</value>
<description>
The name of the default file system. Either the literal string
"local" or a host:port for NDFS.
</description>
</property>
</configuration>
*mapred-site.xml*
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>10.0.0.2:9001</value>
<description>
The host and port that the MapReduce job tracker runs at. If
"local", then jobs are run in-process as a single map and
reduce task.
</description>
</property>
<property>
<name>mapred.system.dir</name>
<value>/my_crawler/filesystem/mapreduce/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/my_crawler/filesystem/mapreduce/local</value>
</property>
</configuration>
*hdfs-site.xml*
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/my_crawler/filesystem/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/my_crawler/filesystem/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
Thanks,
Eran
On Wed, Dec 9, 2009 at 12:22 PM, Eran Zinman <[email protected]> wrote:
Hi Andrzej,
Thanks for your help (as always).
Still getting same exception when running on standalone Hadoop cluster.
Getting same exceptions as before - also in the datanode log I'm
getting:
2009-12-09 12:20:37,805 ERROR datanode.DataNode - java.io.IOException:
Call
to 10.0.0.2:9000 failed on local exception: java.io.IOException:
Connection reset by peer
at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
at org.apache.hadoop.ipc.Client.call(Client.java:742)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:269)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:206)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
Thanks,
Eran
On Wed, Dec 9, 2009 at 12:12 PM, Andrzej Bialecki <[email protected]> wrote:
Eran Zinman wrote:
Hi,
Sorry to bother you guys again, but it seems that no matter what I do I
can't run the new version of Nutch with Hadoop 0.20.
I am getting the following exceptions in my logs when I execute
bin/start-all.sh
Do you use the scripts in place, i.e. without deploying the nutch*.job
to
a separate Hadoop cluster? Could you please try it with a standalone
Hadoop
cluster (even if it's a pseudo-distributed, i.e. single node)?
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
