On 2010-04-26 22:31, Joshua J Pavel wrote: > > Sending this out to close the thread if anyone else experiences this > problem: nutch 1.0 is not AIX-friendly (0.9 is). > > I'm not 100% sure which command it may be, but by modifying my path so > that /opt/freeware/bin has precedence, I no longer get the hadoop disk > error. While I though this means the problem comes from the nutch script, > not the code itself, manually trying to set system calls > to /opt/freeware/bin didn't fix it. I assume until detailed debugging is > done, further releases will also require a workaround similar to what I'm > doing.
Ahhh ... now I understand. The problem lies in Hadoop's use of utilities such as /bin/whoami, /bin/ls and /bin/df. These are used to obtain some filesystem and permissions information that is otherwise not available from JVM. However, these utilities are expected to provide a POSIX-y output if on Unix, or Cygwin output if on Windows. I guess the native commands in AIX don't conform to either, so the output of these utilities can't be parsed, which ultimately results in errors. Whereas the output of /opt/freeware/bin utilities follows the POSIX format. I'm not sure what was the difference in 0.9 that still made it work ... perhaps the parsing of these outputs was more lenient, or some errors were ignored. In any case, we in Nutch can't do anything about this, we can just add your workaround to the documentation. The problem should be reported to the Hadoop project. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com