Hi there, Thanks everybody for your valuable help, with your suggestions I finally got it working in my cluster with Cloudera 6 distribution(Hadoop 3) with Java, I would like to list here a few things I had to do to make it work since I used Cloudera to compile:
* In Cloudera, Hadoop libraries are under $CLOUDERA_PATH/lib/hadoop so this is the path I use for ./configure in --with-hdfs * Then ./configure is looking for a file include/hdfs.h , this directory in Cloudera exists in different path (outside Hadoop) $CLOUDERA_PATH/include, so I had to copy it under Hadoop path. * Same for libhdfs.so , make expects to find it in lib/native/libhdfs.so, for Cloudera it is in $CLOUDERA_PATH/lib64 , so I just copied it to the expected location. I tested using command line and also my Java application and /vsihdfs/ works as expected. Thanks!!! Zazhil-ha ________________________________ From: gdal-dev <gdal-dev-boun...@lists.osgeo.org> on behalf of James McClain <james.mccl...@gmail.com> Sent: Friday, November 23, 2018 3:18 PM To: gdal-dev@lists.osgeo.org Subject: Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL Hello, It may not be finding the native HDFS libraries. Please see the pull request https://github.com/OSGeo/gdal/pull/714 for build instructions (in particular, you may need to augment the LD_LIBRARY_PATH environment variable). If trouble persists, I would suggest building against Apache Hadoop 2.7.6 or 2.7.7 (both of those are know to work) as an experiment. Sincerely, James McClian On Fri, Nov 23, 2018 at 2:15 PM ZAZHIL-HA HERENA <zazhil...@hotmail.com<mailto:zazhil...@hotmail.com>> wrote: Thank you so much!, now I am working on 2.4 source code but I am getting an error when trying to configure using: ./configure --prefix=/scratch/zherena/gdal/build/gdal-master/gdal/outputb/ --with-complete=yes --with-java=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/ --with-swig-java=yes --with-hdfs=/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/ --with-curl=/usr/bin/curl-config The error I get is: checking for HDFS in /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/... checking for hdfsConnect in -lhdfs... no checking for /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678//include/hdfs.h... yes configure: error: HDFS support not enabled. Is there any configuration in my environment that I should consider? or maybe another distribution of Hadoop? ________________________________ From: Even Rouault <even.roua...@spatialys.com<mailto:even.roua...@spatialys.com>> Sent: Friday, November 23, 2018 11:52 AM To: gdal-dev@lists.osgeo.org<mailto:gdal-dev@lists.osgeo.org> Cc: ZAZHIL-HA HERENA; James McClain; n...@nikosalexandris.net<mailto:n...@nikosalexandris.net> Subject: Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL > Version says 2.3.2 but libraries say: libgdal.so.20.4.2 . Libtool number (.so.20.4.2) has nothing to do with user-friendly version number (2.3.2) > I am not sure if I > got the latest code, this is the first time I compile it myself, I used > this link to download source code: > http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz This is the latest release, but /vsihdfs/ is in the development version, not yet released, so download https://github.com/OSGeo/gdal/archive/master.zip -- Spatialys - Geospatial professional services http://www.spatialys.com -- "I prayed for freedom for twenty years, but received no answer until I prayed with my legs." -- Frederick Douglass
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev