Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL
Hello, It may not be finding the native HDFS libraries. Please see the pull request https://github.com/OSGeo/gdal/pull/714 for build instructions (in particular, you may need to augment the LD_LIBRARY_PATH environment variable). If trouble persists, I would suggest building against Apache Hadoop 2.7.6 or 2.7.7 (both of those are know to work) as an experiment. Sincerely, James McClian On Fri, Nov 23, 2018 at 2:15 PM ZAZHIL-HA HERENA wrote: > Thank you so much!, now I am working on 2.4 source code but I am getting > an error when trying to configure using: > > *./configure > --prefix=/scratch/zherena/gdal/build/gdal-master/gdal/outputb/ > --with-complete=yes > --with-java=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/ > --with-swig-java=yes > --with-hdfs=/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/ > --with-curl=/usr/bin/curl-config* > > The error I get is: > > > * checking for HDFS in > /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/... checking for > hdfsConnect in -lhdfs... no * > > * checking for > /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678//include/hdfs.h... yes > * > * configure: error: HDFS support not enabled.* > > > Is there any configuration in my environment that I should consider? or > maybe another distribution of Hadoop? > > > -- > *From:* Even Rouault > *Sent:* Friday, November 23, 2018 11:52 AM > *To:* gdal-dev@lists.osgeo.org > *Cc:* ZAZHIL-HA HERENA; James McClain; n...@nikosalexandris.net > *Subject:* Re: [gdal-dev] Question on how to open a raster in HDFS using > GDAL > > > Version says 2.3.2 but libraries say: libgdal.so.20.4.2 . > > Libtool number (.so.20.4.2) has nothing to do with user-friendly version > number (2.3.2) > > > I am not sure if I > > got the latest code, this is the first time I compile it myself, I used > > this link to download source code: > > http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz > > This is the latest release, but /vsihdfs/ is in the development version, > not > yet released, so download > > https://github.com/OSGeo/gdal/archive/master.zip > > -- > Spatialys - Geospatial professional services > http://www.spatialys.com > -- "I prayed for freedom for twenty years, but received no answer until I prayed with my legs." -- Frederick Douglass ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL
Thank you so much!, now I am working on 2.4 source code but I am getting an error when trying to configure using: ./configure --prefix=/scratch/zherena/gdal/build/gdal-master/gdal/outputb/ --with-complete=yes --with-java=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/ --with-swig-java=yes --with-hdfs=/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/ --with-curl=/usr/bin/curl-config The error I get is: checking for HDFS in /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/... checking for hdfsConnect in -lhdfs... no checking for /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678//include/hdfs.h... yes configure: error: HDFS support not enabled. Is there any configuration in my environment that I should consider? or maybe another distribution of Hadoop? From: Even Rouault Sent: Friday, November 23, 2018 11:52 AM To: gdal-dev@lists.osgeo.org Cc: ZAZHIL-HA HERENA; James McClain; n...@nikosalexandris.net Subject: Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL > Version says 2.3.2 but libraries say: libgdal.so.20.4.2 . Libtool number (.so.20.4.2) has nothing to do with user-friendly version number (2.3.2) > I am not sure if I > got the latest code, this is the first time I compile it myself, I used > this link to download source code: > http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz This is the latest release, but /vsihdfs/ is in the development version, not yet released, so download https://github.com/OSGeo/gdal/archive/master.zip -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL
> Version says 2.3.2 but libraries say: libgdal.so.20.4.2 . Libtool number (.so.20.4.2) has nothing to do with user-friendly version number (2.3.2) > I am not sure if I > got the latest code, this is the first time I compile it myself, I used > this link to download source code: > http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz This is the latest release, but /vsihdfs/ is in the development version, not yet released, so download https://github.com/OSGeo/gdal/archive/master.zip -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL
Thank you for the reply.
Nikos,
The path is correct, the space was an error in my redaction.
James,
I tried to run a GDALINFO in my Linux command line using the GDAL I configured
and installed and I am not able to open the raster, even trying to read it from
local:
-bash-4.2$ gdalinfo
/vsihdfs/file:/scratch/zherena/test/write/data/rasters/hawaii.tif
ERROR 4:
/vsihdfs/file:/scratch/zherena/test/write/data/rasters/hawaii.tif: No such file
or directory
gdalinfo failed - unable to open
'/vsihdfs/file:/scratch/zherena/test/write/data/rasters/hawaii.tif'.
If I try the same commad withouth /vsihdfs/file:/... it works fine.
I also tried:
-bash-4.2$ gdalinfo --version
GDAL 2.3.2, released 2018/09/21
Version says 2.3.2 but libraries say: libgdal.so.20.4.2 . I am not sure if I
got the latest code, this is the first time I compile it myself, I used this
link to download source code:
http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz
And I have a doubt, when configuring I used: --with-hdfs=yes, made another
using: --with-hdfs = /scratch/zherena/gdal/hadoop/lib/native. Both tries did
not work. I am not really sure what should I pass as a parameter to --with-hdfs?
This is the list of steps I used to build, I am using Java in my application
where I plan to use gdal:
* make clean distclean(per your suggestion)
*./configure --prefix=/scratch/zherena/gdal/build/gdal-2.3.2/outputb/
--with-complete=yes --with-java=yes --with-swig-java=yes
--with-hdfs=/scratch/zherena/gdal/hadoop/lib/native
--with-curl=/usr/bin/curl-config
* make
* make install
* cd /swig/java
* make veryclean
* make
* cp *.so ../../outputb/lib/
* cp *.jar ../../outputb/lib/
Maybe I am missing something during compile?
Thank you!!
Zazhil-ha
From: gdal-dev on behalf of James McClain
Sent: Thursday, November 22, 2018 8:27 PM
Cc: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL
Hello,
I am the author of the vsihdfs code, I am ready and willing to help.
I just rebuilt it from current master and was able to successfully open an
dataset via an HDFS URI with the GDAL Python bindings. I have a few
suggestions.
First, please try putting the file into a local directory and try something
like `gdalinfo /vsihdfs/file:/tmp/kahoolawe.tif` to establish a baseline.
Second, if you are using the Python bindings, please make sure that they have
been built and installed (and that you are using the ones that you built rather
than other ones that exist on your system). Instructions for building the
Python bindings can be found here:
https://trac.osgeo.org/gdal/wiki/BuildingOnUnix .
In my case, after building and installing the library and bindings, I was able
to successfully open a dataset by starting a python REPL like this:
```bash
export
LD_LIBRARY_PATH=$HOME/local/hadoop-2.7.7/lib/native:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:$HOME/local/gdal-master-vsihdfs/lib
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export CLASSPATH=$($HOME/local/hadoop-2.7.7/bin/hadoop classpath --glob)
PYTHONPATH=$HOME/local/gdal-master-vsihdfs/lib/python2.7/site-packages python
```
then typing this into it:
```python
from osgeo import gdal, gdalconst
ds = gdal.Open('/vsihdfs/file:/tmp/testfile.tif', gdalconst.GA_ReadOnly)
```
(I do not have easy access to and HDFS cluster right at the moment, so I only
tested a local HDFS URI.)
A note: After having done a build without HDFS support in the tree, I had do a
`make clean distclean` before I was able to get a build with working HDFS
support.
Sincerely,
James McClain
On Thu, Nov 22, 2018 at 8:13 PM Nikos Alexandris
mailto:n...@nikosalexandris.net>> wrote:
* ZAZHIL-HA HERENA mailto:zazhil...@hotmail.com>>
[2018-11-22 22:35:32 +]:
>Hello, I am not sure if I should use this mailing list to ask questions but I
>wanted to try, I am a developer trying to use GDAL to open rasters in HDFS.
>
>
>I read in GDAL documentation that starting 2.4 it is possible to open a raster
>in HDFS. I downloaded and compiled the latest source code available version
>and the generated libraries show it is 2.4 (libgdal.so.20.4.2). I compiled
>with option "-with-hdfs=yes" and "--with-java=yes".
>
>I am trying to open a raster using:
>
>
>
>Dataset raster = gdal.Open("/vsihdfs/hdfs://node:8020/user/hdfs
> /spatial_raster/input_raster/kahoolawe.tif", gdalconst.GA_ReadOnly);
Is your path correct? There is a space here (in "/hfds /").
Nikos
>
>
>but I am getting the following error: "ERROR 4: No such file or directory"
[rest deleted]
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
"I prayed for freedom for twenty years, but received no answer until I prayed
with my legs."
-- Frederick
Re: [gdal-dev] How to Use Gdal in Microsoft Visual Studio C++
Athin wrote > If possible can you share with some guide how to use the gdal nuget and > how > can i view the map on wpf not on console? Sure. You should right-click on "References" in your open project in VS and choose "Manage nuget packages". Then search for "GDAL.NET" and install it. Necessary references and GdalConfigurator class will automatically be added to your project. If you don't have internet on your PC to download nuget packages directly from VS, you can download GDAL.NET package from nuget: https://www.nuget.org/packages/GDAL.NET/. Then, you should change you nuget search path, here's good answer on stackoverflow how to do it: https://stackoverflow.com/questions/10240029/how-do-i-install-a-nuget-package-nupkg-file-locally. After you build your project once, compiled gdal bindings will be placed in your dubug/release directory near .exe file. Now you should call "GdalConfigurator.ConfigureGdal()" (or ConfigureOgr(), depends on what you need) method in you program. If it doesn't throw any exceptions, then gdal is found and all EnvironmentVariables set correctly. If it throws exceptions, try changing solution platform on x64 and change the following line in GdalConfiguration class from: var executingAssemblyFile = new Uri(Assembly.GetExecutingAssembly().GetName().CodeBase).LocalPath; to string executingAssemblyFile = Assembly.GetExecutingAssembly().Location; The thing is, that original version works bad, if path contains some special symbols, like "#", etc. After you made your ConfigureGdal() method work correctly, you should add "using OSGeo.*" namespaces to your code files, and now you can use all gdal's c# bindings. Unfortunately, I can't answer to your second question, I haven't done it before. If I'll have time, I'll check it out and write you about results. -- Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] How to Use Gdal in Microsoft Visual Studio C++
Hi Gigas002, If possible can you share with some guide how to use the gdal nuget and how can i view the map on wpf not on console? -- Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
