Hi,
I'm trying to get Hadoop running on Windows, and I've found that it
isn't exactly a simple process (yes, I'm using Cygwin).
I've had a quick look though the mailing list, and I see the occasional
question about it, generally with the response "use Cygwin", and no
further details.
Is anyone actually successfully running Hadoop on Windows/Cygwin? If so,
have you had similar problems to those outlined below? I think there's a
fair chance the first two of these are environmental local to my
machine, but it would be nice to be sure that if I keep working on this
it will work in the end...
1) Running bin/start-all.sh gives:
: no address associated with name
That turns out to be a Unix vs DOS line endings thing (!).
Running the following commands fixed that:
dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/masters
dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/slaves
2) Any script using hadoop-daemon.sh failed. This turned out to be
another Unix vs DOS line endings thing. The critical error looked like
this:
<machine-name>: head: cannot open
`/cygdrive/c/dev/prog/hadoop-0.14.1/bin/../logs/hadoop-nlothian-namenode
-<machine-name>\r.out' for reading: No such file or directory
(note that "\r.out" in the logfile name...)
The fix for that was to alter the hadoop-daemon.sh script. Replace:
export
HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.log
with
export
HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-`'hostname' | sed
s/\r//`.log
and
log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.out
with
log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`'hostname' |
sed s/\r//`.out
3) Now the start-all.sh script seems to run ok, but the Java processes
which are started on the slave machine (in this case the same machine as
the master) die soon after launching.
If I manually run:
bin/hadoop jobtracker
bin/hadoop tasktracker
bin/hadoop namenode
bin/hadoop datanode
I can succesfully submit jobs. However, when using the start-all.sh
script the submitted jobs fail to connect (because the jobtracker dies).
Has anyone else seen this? Is there a workaround?
Regards
Nick Lothian
IMPORTANT: This e-mail, including any attachments, may contain private or
confidential information.
If you think you may not be the intended recipient, or if you have received
this e-mail in error, please contact the sender immediately and delete all
copies of this e-mail.
If you are not the intended recipient, you must not reproduce any part of this
e-mail or disclose its contents to any other party.
This email represents the views of the individual sender, which do not
necessarily reflect those of education.au limited except where the sender
expressly states otherwise.
It is your responsibility to scan this email and any files transmitted with it
for viruses or any other defects. education.au limited will not be liable for
any loss, damage or consequence caused directly or indirectly by this email.
