Hi Leo, Following your instruction on moving the sections in configure.ac, I am able to re-compile the code base and I can find that the "objdump" returns the correct fadvise symbols as you described in your earlier email. I then ran the TestNativeIO.java class and I believe that the Fadvise test case gets passed now.
Thank you very much for the help! Regards, Jun On Sat, Jun 22, 2013 at 4:42 PM, Leo Leung <[email protected]> wrote: > Hi Jun,**** > > ** ** > > Try moving the section between:**** > > dnl Check for headers needed by the native Group…. > […include everything in between inclusive …] > AC_FUNC_STRERROR_R > > **** > > to higher place (insert before)**** > > ** ** > > # Checks for libraries.**** > > dnl Check for ‘-ldl’ > > Hope this helps J**** > > ** ** > > ** ** > > *From:* Jun Li [mailto:[email protected]] > *Sent:* Saturday, June 22, 2013 3:26 AM > > *To:* [email protected] > *Subject:* Re: how to turn on NativeIO.posixFadviseIfPossible**** > > ** ** > > Hi Leo, > > Thanks for the detailed instruction. I started from the clean downloaded > hadoop1.1.12 copy and then I issue the command: ant -Dcompile.native=true. > > And with the configure.ac that comes with the downloaded package, without > any changes, the following is the compilation output: > > > [exec] checking fcntl.h usability... yes > [exec] checking fcntl.h presence... yes > [exec] checking for fcntl.h... yes > [exec] checking for stdlib.h... (cached) yes > [exec] checking for string.h... (cached) yes > [exec] checking for unistd.h... (cached) yes > [exec] checking for fcntl.h... (cached) yes > [exec] checking for posix_fadvise... no > [exec] checking for fcntl.h... (cached) yes > [exec] checking for sync_file_range... no > [exec] checking for an ANSI C-conforming const... yes > [exec] checking for memset... no > [exec] checking whether strerror_r is declared... yes > [exec] checking for strerror_r... no**** > > I have difficulty to follow the "+" and "-" relate patching sequence that > you stated in your email earlier. I have attached the "configure.ac" file > that I copied out of the hadoop1.1.12/src/native directory. Could you help > me to construct the configure.ac with the correct patching that you > described, using the one that I attached in this email, and send back to > me? Then I will use the patched one to build the native library.**** > > Regards, > > Jun**** > > ** ** > > On Fri, Jun 21, 2013 at 12:35 PM, Leo Leung <[email protected]> wrote:**** > > Hi Jun:**** > > **** > > Looks like it is not in the apache hadoop 1.x release binaries, this > probably will have to be answered by the apache hadoop 1.x release team*** > * > > **** > > To build the native libs:**** > > **** > > 1) Get the source code and install the dependencies to compile hadoop > **** > > 2) You’ll have to setup the env for it, such as JAVA_HOME. Gcc > installed etc.**** > > 3) Got to <hadoop-source dir> ant [veryclean] > compile-core-native >compile.out**** > > **** > > Verify with **** > > [branch-1]$ objdump -Tt ./build/native/Linux-amd64-64/lib/libhadoop.so | > grep fadv**** > > 00000000000056a0 g F .text 00000000000000a3 > Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise**** > > 0000000000000000 F *UND* 0000000000000000 > posix_fadvise@@GLIBC_2.2.5**** > > 0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 posix_fadvise > **** > > 00000000000056a0 g DF .text 00000000000000a3 Base > Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise > > **** > > If you get an error about “make distclean” error**** > > Just cd in src/native and run make distclean (to cleanup ) **** > > **** > > Example of ant output that shows checking for fadvise **** > > [exec] checking for fcntl.h... yes**** > > [exec] checking for stdlib.h... (cached) yes**** > > [exec] checking for string.h... (cached) yes**** > > [exec] checking for unistd.h... (cached) yes**** > > [exec] checking for fcntl.h... (cached) yes**** > > [exec] checking for posix_fadvise... yes**** > > [exec] checking for fcntl.h... (cached) yes**** > > [exec] checking for sync_file_range... yes**** > > **** > > If you see “no” for posix_fadvise or “sync_file_range” **** > > And you are sure you have posix_fdavise support (RHEL 6.x or CentOS 6.x)** > ** > > **** > > you may need to move the section on src/native/configure.ac**** > > Try to modify the configure.ac > --- src/native/configure.ac (revision 1495498)**** > > +++ src/native/configure.ac (working copy)**** > > @@ -47,6 +47,21 @@**** > > AC_PROG_CC**** > > AC_PROG_LIBTOOL**** > > **** > > +dnl check for posix_fadvise**** > > +AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(posix_fadvise)])**** > > +**** > > +dnl check for sync_file_range**** > > +AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(sync_file_range)])**** > > +**** > > +# Checks for typedefs, structures, and compiler characteristics.**** > > +AC_C_CONST**** > > +**** > > +# Checks for library functions.**** > > +AC_CHECK_FUNCS([memset])**** > > +**** > > +# Check for nonstandard STRERROR_R**** > > +AC_FUNC_STRERROR_R**** > > +**** > > # Checks for libraries.**** > > dnl Check for '-ldl'**** > > AC_CHECK_LIB([dl], [dlopen])**** > > @@ -104,21 +119,6 @@**** > > dnl Check for headers needed by the native Group resolution implementation > **** > > AC_CHECK_HEADERS([fcntl.h stdlib.h string.h unistd.h], [], > AC_MSG_ERROR(Some system headers not found... please ensure their presence > on your platform.))**** > > **** > > -dnl check for posix_fadvise**** > > -AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(posix_fadvise)])**** > > -**** > > -dnl check for sync_file_range**** > > -AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(sync_file_range)])**** > > -**** > > -# Checks for typedefs, structures, and compiler characteristics.**** > > -AC_C_CONST**** > > -**** > > -# Checks for library functions.**** > > -AC_CHECK_FUNCS([memset])**** > > -**** > > -# Check for nonstandard STRERROR_R**** > > -AC_FUNC_STRERROR_R**** > > -**** > > AC_CONFIG_FILES([Makefile])**** > > AC_OUTPUT**** > > **** > > **** > > Hope this helps**** > > **** > > **** > > **** > > **** > > *From:* Jun Li [mailto:[email protected]] > *Sent:* Friday, June 21, 2013 9:44 AM > *To:* [email protected] > *Subject:* Re: how to turn on NativeIO.posixFadviseIfPossible**** > > **** > > Hi Leo,**** > > **** > > Following your instruction, the following is what I got: > > [junli@mercoop-26 Linux-amd64-64]$ objdump -Tt libhadoop.so | grep -I > fadvise > 0000000000004d70 g F .text 000000000000006f > Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise > 0000000000004d70 g DF .text 000000000000006f Base > Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise**** > > Apparently the part of "the GLIBC" is not there **** > > The libhadoop.so is the one that I downloaded as part of the tar.gz of > hadoop.1.1.2 from the Hadoop Apache web site. **** > > Could you let me know how I can use the source code directory in the > downloaded Hadoop Apache package, to re-compile the libhadoop.so and to > make sure that the fadvise call is able to get referenced?**** > > Regards, > > Jun**** > > **** > > On Fri, Jun 21, 2013 at 9:19 AM, Leo Leung <[email protected]> wrote:**** > > This looks like a compilation problem on the native hadoop libraries. > > Please locate the libhadoop.so library on your system and run > [shell] objdump -Tt libhadoop.so | grep -I fadvise > > If you don't see something like the following *the GLIBC* part (that > means the system where the share lib was compiled did not have it) > > 00000000000056a0 g F .text 00000000000000a3 > Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise > 0000000000000000 F *UND* 0000000000000000 > posix_fadvise@@GLIBC_2.2.5 > 0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 posix_fadvise > 00000000000056a0 g DF .text 00000000000000a3 Base > Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise > > Note: objdump is from binutils- rpm (you can use yum install to install > it if you don't have it)**** > > > > -----Original Message----- > From: Jun Li [mailto:[email protected]] > Sent: Friday, June 21, 2013 1:35 AM > To: [email protected] > Subject: how to turn on NativeIO.posixFadviseIfPossible > > Hi, > > I downloaded the current stable version from the Apache Hadoop web site, > hadoop-1.1.2. My machine is an AMD-based machine and Redhat Enterprise 6.1. > The detailed Linux kernel version is: > 2.6.32-131.0.15.el6.x86_64 > > I ran the TestNativeIO.java under the distribution directory of > "test/org/apache/hadoop/io/nativeio/TestNativeIO.java" and tried to > understand how NativeIO.posixFadviseIfPossible behaves, in particular, to > check whether "posix_fadvise" is turned on or not. I am interested in this > call as it is used in Read-Ahead-Pool to cache data to the OS's buffer > cache. The following is the test case that I ran: > > @Test > public void testPosixFadvise() throws Exception { > FileInputStream fis = new FileInputStream("/dev/zero"); > try { > NativeIO.posixFadviseIfPossible(fis.getFD(), 0, 0, > NativeIO.POSIX_FADV_SEQUENTIAL); > } catch (NativeIOException noe) { > // we should just skip the unit test on machines where we don't > // have fadvise support > assumeTrue(false); > } finally { > fis.close(); > } > > However, when I stepped into the code and reached "NativeIO.java" > under the package of "org.apache.hadoop.io.nativeio", in the particular > call below: > > public static void posixFadviseIfPossible( > FileDescriptor fd, long offset, long len, int flags) > throws NativeIOException { > > if (nativeLoaded && fadvisePossible) { > try { > posix_fadvise(fd, offset, len, flags); > } catch (UnsupportedOperationException uoe) { > fadvisePossible = false; > } catch (UnsatisfiedLinkError ule) { > fadvisePossible = false; > } > } > } > > The call to "posix_fadvise" threw the "UnsupportedOperationException" > exception. > > I further traced to the native library, and in the code "NativeIO.c", I > found > > JNIEXPORT void JNICALL > Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise( > JNIEnv *env, jclass clazz, > jobject fd_object, jlong offset, jlong len, jint flags) { #ifndef > HAVE_POSIX_FADVISE > THROW(env, "java/lang/UnsupportedOperationException", > "fadvise support not available"); #else > > ... > } > > I believe that the problem of throwing the exception is because > "HAVE_POSIX_FADVISE" is not defined. I made sure that the native IO > library is loaded properly in the Java code, as I can successfully run the > other test cases in "TestNativeIO.java". > > So my question is: should I re-compile the "libhadoop" in order to get the > version of the shared library that can have "HAVE_POSIX_FADVISE" > turned on? Or by default, FADVISE is turned on already? > > Thank you!**** > > **** > > ** ** >
