This looks like a compilation problem on the native hadoop libraries. Please locate the libhadoop.so library on your system and run [shell] objdump -Tt libhadoop.so | grep -I fadvise
If you don't see something like the following *the GLIBC* part (that means the system where the share lib was compiled did not have it) 00000000000056a0 g F .text 00000000000000a3 Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise 0000000000000000 F *UND* 0000000000000000 posix_fadvise@@GLIBC_2.2.5 0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 posix_fadvise 00000000000056a0 g DF .text 00000000000000a3 Base Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise Note: objdump is from binutils- rpm (you can use yum install to install it if you don't have it) -----Original Message----- From: Jun Li [mailto:[email protected]] Sent: Friday, June 21, 2013 1:35 AM To: [email protected] Subject: how to turn on NativeIO.posixFadviseIfPossible Hi, I downloaded the current stable version from the Apache Hadoop web site, hadoop-1.1.2. My machine is an AMD-based machine and Redhat Enterprise 6.1. The detailed Linux kernel version is: 2.6.32-131.0.15.el6.x86_64 I ran the TestNativeIO.java under the distribution directory of "test/org/apache/hadoop/io/nativeio/TestNativeIO.java" and tried to understand how NativeIO.posixFadviseIfPossible behaves, in particular, to check whether "posix_fadvise" is turned on or not. I am interested in this call as it is used in Read-Ahead-Pool to cache data to the OS's buffer cache. The following is the test case that I ran: @Test public void testPosixFadvise() throws Exception { FileInputStream fis = new FileInputStream("/dev/zero"); try { NativeIO.posixFadviseIfPossible(fis.getFD(), 0, 0, NativeIO.POSIX_FADV_SEQUENTIAL); } catch (NativeIOException noe) { // we should just skip the unit test on machines where we don't // have fadvise support assumeTrue(false); } finally { fis.close(); } However, when I stepped into the code and reached "NativeIO.java" under the package of "org.apache.hadoop.io.nativeio", in the particular call below: public static void posixFadviseIfPossible( FileDescriptor fd, long offset, long len, int flags) throws NativeIOException { if (nativeLoaded && fadvisePossible) { try { posix_fadvise(fd, offset, len, flags); } catch (UnsupportedOperationException uoe) { fadvisePossible = false; } catch (UnsatisfiedLinkError ule) { fadvisePossible = false; } } } The call to "posix_fadvise" threw the "UnsupportedOperationException" exception. I further traced to the native library, and in the code "NativeIO.c", I found JNIEXPORT void JNICALL Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise( JNIEnv *env, jclass clazz, jobject fd_object, jlong offset, jlong len, jint flags) { #ifndef HAVE_POSIX_FADVISE THROW(env, "java/lang/UnsupportedOperationException", "fadvise support not available"); #else ... } I believe that the problem of throwing the exception is because "HAVE_POSIX_FADVISE" is not defined. I made sure that the native IO library is loaded properly in the Java code, as I can successfully run the other test cases in "TestNativeIO.java". So my question is: should I re-compile the "libhadoop" in order to get the version of the shared library that can have "HAVE_POSIX_FADVISE" turned on? Or by default, FADVISE is turned on already? Thank you!
