On 2010-04-26 22:31, Joshua J Pavel wrote:
> Sending this out to close the thread if anyone else experiences this
> problem: nutch 1.0 is not AIX-friendly (0.9 is).
> I'm not 100% sure which command it may be, but by modifying my path so
> that /opt/freeware/bin has precedence, I no longer get the hadoop disk
> error.  While I though this means the problem comes from the nutch script,
> not the code itself, manually trying to set system calls
> to /opt/freeware/bin didn't fix it.  I assume until detailed debugging is
> done, further releases will also require a workaround similar to what I'm
> doing.

Ahhh ... now I understand. The problem lies in Hadoop's use of utilities
such as /bin/whoami, /bin/ls and /bin/df. These are used to obtain some
filesystem and permissions information that is otherwise not available
from JVM.

However, these utilities are expected to provide a POSIX-y output if on
Unix, or Cygwin output if on Windows. I guess the native commands in AIX
don't conform to either, so the output of these utilities can't be
parsed, which ultimately results in errors. Whereas the output of
/opt/freeware/bin utilities follows the POSIX format.

I'm not sure what was the difference in 0.9 that still made it work ...
perhaps the parsing of these outputs was more lenient, or some errors
were ignored.

In any case, we in Nutch can't do anything about this, we can just add
your workaround to the documentation. The problem should be reported to
the Hadoop project.

Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Reply via email to