Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

2018-11-27 Thread James McClain
You are very welcome, thank you for the report!

On Tue, Nov 27, 2018 at 5:44 PM ZAZHIL-HA HERENA 
wrote:

> Hi there,
>
> Thanks everybody for your valuable help, with your suggestions I finally
> got it working in my cluster with Cloudera 6 distribution(Hadoop 3) with
> Java, I would like to list here a few things I had to do to make it work
> since I used Cloudera to compile:
>
>
>- In Cloudera, Hadoop libraries are under $CLOUDERA_PATH/lib/hadoop so
>this is the path I use for ./configure in --with-hdfs
>- Then* ./configure* is looking for  a file *include/hdfs.h* , this
>directory in Cloudera exists in different path (outside Hadoop)
>$CLOUDERA_PATH/include, so I had to copy it under Hadoop path.
>- Same for libhdfs.so , *make* expects to find it in
>*lib/native/libhdfs.so*, for Cloudera it is in $CLOUDERA_PATH/lib64 ,
>so I just copied it to the expected location.
>
>
> I tested using command line and also my Java application and /vsihdfs/
> works as expected.
>
> Thanks!!!
> Zazhil-ha
> --
> *From:* gdal-dev  on behalf of James
> McClain 
> *Sent:* Friday, November 23, 2018 3:18 PM
> *To:* gdal-dev@lists.osgeo.org
> *Subject:* Re: [gdal-dev] Question on how to open a raster in HDFS using
> GDAL
>
> Hello,
>
> It may not be finding the native HDFS libraries.  Please see the pull
> request https://github.com/OSGeo/gdal/pull/714 for build instructions (in
> particular, you may need to augment the LD_LIBRARY_PATH environment
> variable).
>
> If trouble persists, I would suggest building against Apache Hadoop 2.7.6
> or 2.7.7 (both of those are know to work) as an experiment.
>
> Sincerely,
> James McClian
>
> On Fri, Nov 23, 2018 at 2:15 PM ZAZHIL-HA HERENA 
> wrote:
>
> Thank you so much!, now I am working on 2.4 source code but I am getting
> an error when trying to configure using:
>
> *./configure
> --prefix=/scratch/zherena/gdal/build/gdal-master/gdal/outputb/
> --with-complete=yes
> --with-java=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/
> --with-swig-java=yes
> --with-hdfs=/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/
> --with-curl=/usr/bin/curl-config*
>
> The error I get is:
>
>
> *  checking for HDFS in
> /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/... checking for
> hdfsConnect in -lhdfs... no *
>
> *  checking for
> /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678//include/hdfs.h... yes
> *
> *  configure: error: HDFS support not enabled.*
>
>
> Is there any configuration in my environment that I should consider? or
> maybe another distribution of Hadoop?
>
>
> --
> *From:* Even Rouault 
> *Sent:* Friday, November 23, 2018 11:52 AM
> *To:* gdal-dev@lists.osgeo.org
> *Cc:* ZAZHIL-HA HERENA; James McClain; n...@nikosalexandris.net
> *Subject:* Re: [gdal-dev] Question on how to open a raster in HDFS using
> GDAL
>
> > Version says 2.3.2 but libraries say: libgdal.so.20.4.2 .
>
> Libtool number (.so.20.4.2) has nothing to do with user-friendly version
> number (2.3.2)
>
> > I am not sure if I
> > got the latest code, this is the first time I compile it myself, I used
> > this link to download source code:
> > http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz
>
> This is the latest release, but /vsihdfs/ is in the development version,
> not
> yet released, so download
>
> https://github.com/OSGeo/gdal/archive/master.zip
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
>
>
> --
> "I prayed for freedom for twenty years, but received no answer until I
> prayed with my legs."
>  -- Frederick Douglass
>
>

-- 
"I prayed for freedom for twenty years, but received no answer until I
prayed with my legs."
 -- Frederick Douglass
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

2018-11-27 Thread ZAZHIL-HA HERENA
Hi there,

Thanks everybody for your valuable help, with your suggestions I finally got it 
working in my cluster with Cloudera 6 distribution(Hadoop 3) with Java, I would 
like to list here a few things I had to do to make it work since I used 
Cloudera to compile:


  *   In Cloudera, Hadoop libraries are under $CLOUDERA_PATH/lib/hadoop so this 
is the path I use for ./configure in --with-hdfs
  *   Then ./configure is looking for  a file include/hdfs.h , this directory 
in Cloudera exists in different path (outside Hadoop) $CLOUDERA_PATH/include, 
so I had to copy it under Hadoop path.
  *   Same for libhdfs.so , make expects to find it in lib/native/libhdfs.so, 
for Cloudera it is in $CLOUDERA_PATH/lib64 , so I just copied it to the 
expected location.

I tested using command line and also my Java application and /vsihdfs/ works as 
expected.

Thanks!!!
Zazhil-ha

From: gdal-dev  on behalf of James McClain 

Sent: Friday, November 23, 2018 3:18 PM
To: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

Hello,

It may not be finding the native HDFS libraries.  Please see the pull request 
https://github.com/OSGeo/gdal/pull/714 for build instructions (in particular, 
you may need to augment the LD_LIBRARY_PATH environment variable).

If trouble persists, I would suggest building against Apache Hadoop 2.7.6 or 
2.7.7 (both of those are know to work) as an experiment.

Sincerely,
James McClian

On Fri, Nov 23, 2018 at 2:15 PM ZAZHIL-HA HERENA 
mailto:zazhil...@hotmail.com>> wrote:
Thank you so much!, now I am working on 2.4 source code but I am getting an 
error when trying to configure using:

./configure --prefix=/scratch/zherena/gdal/build/gdal-master/gdal/outputb/ 
--with-complete=yes 
--with-java=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/ 
--with-swig-java=yes 
--with-hdfs=/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/ 
--with-curl=/usr/bin/curl-config

The error I get is:

  checking for HDFS in 
/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/... checking for 
hdfsConnect in -lhdfs... no
  checking for 
/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678//include/hdfs.h... yes
  configure: error: HDFS support not enabled.


Is there any configuration in my environment that I should consider? or maybe 
another distribution of Hadoop?



From: Even Rouault 
mailto:even.roua...@spatialys.com>>
Sent: Friday, November 23, 2018 11:52 AM
To: gdal-dev@lists.osgeo.org
Cc: ZAZHIL-HA HERENA; James McClain; 
n...@nikosalexandris.net
Subject: Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

> Version says 2.3.2 but libraries say: libgdal.so.20.4.2 .

Libtool number (.so.20.4.2) has nothing to do with user-friendly version
number (2.3.2)

> I am not sure if I
> got the latest code, this is the first time I compile it myself, I used
> this link to download source code:
> http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz

This is the latest release, but /vsihdfs/ is in the development version, not
yet released, so download

https://github.com/OSGeo/gdal/archive/master.zip

--
Spatialys - Geospatial professional services
http://www.spatialys.com


--
"I prayed for freedom for twenty years, but received no answer until I prayed 
with my legs."
 -- Frederick Douglass

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

2018-11-23 Thread James McClain
Hello,

It may not be finding the native HDFS libraries.  Please see the pull
request https://github.com/OSGeo/gdal/pull/714 for build instructions (in
particular, you may need to augment the LD_LIBRARY_PATH environment
variable).

If trouble persists, I would suggest building against Apache Hadoop 2.7.6
or 2.7.7 (both of those are know to work) as an experiment.

Sincerely,
James McClian

On Fri, Nov 23, 2018 at 2:15 PM ZAZHIL-HA HERENA 
wrote:

> Thank you so much!, now I am working on 2.4 source code but I am getting
> an error when trying to configure using:
>
> *./configure
> --prefix=/scratch/zherena/gdal/build/gdal-master/gdal/outputb/
> --with-complete=yes
> --with-java=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/
> --with-swig-java=yes
> --with-hdfs=/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/
> --with-curl=/usr/bin/curl-config*
>
> The error I get is:
>
>
> *  checking for HDFS in
> /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/... checking for
> hdfsConnect in -lhdfs... no *
>
> *  checking for
> /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678//include/hdfs.h... yes
> *
> *  configure: error: HDFS support not enabled.*
>
>
> Is there any configuration in my environment that I should consider? or
> maybe another distribution of Hadoop?
>
>
> --
> *From:* Even Rouault 
> *Sent:* Friday, November 23, 2018 11:52 AM
> *To:* gdal-dev@lists.osgeo.org
> *Cc:* ZAZHIL-HA HERENA; James McClain; n...@nikosalexandris.net
> *Subject:* Re: [gdal-dev] Question on how to open a raster in HDFS using
> GDAL
>
> > Version says 2.3.2 but libraries say: libgdal.so.20.4.2 .
>
> Libtool number (.so.20.4.2) has nothing to do with user-friendly version
> number (2.3.2)
>
> > I am not sure if I
> > got the latest code, this is the first time I compile it myself, I used
> > this link to download source code:
> > http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz
>
> This is the latest release, but /vsihdfs/ is in the development version,
> not
> yet released, so download
>
> https://github.com/OSGeo/gdal/archive/master.zip
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>


-- 
"I prayed for freedom for twenty years, but received no answer until I
prayed with my legs."
 -- Frederick Douglass
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

2018-11-23 Thread ZAZHIL-HA HERENA
Thank you so much!, now I am working on 2.4 source code but I am getting an 
error when trying to configure using:

./configure --prefix=/scratch/zherena/gdal/build/gdal-master/gdal/outputb/ 
--with-complete=yes 
--with-java=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/ 
--with-swig-java=yes 
--with-hdfs=/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/ 
--with-curl=/usr/bin/curl-config

The error I get is:

  checking for HDFS in 
/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/... checking for 
hdfsConnect in -lhdfs... no
  checking for 
/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678//include/hdfs.h... yes
  configure: error: HDFS support not enabled.


Is there any configuration in my environment that I should consider? or maybe 
another distribution of Hadoop?



From: Even Rouault 
Sent: Friday, November 23, 2018 11:52 AM
To: gdal-dev@lists.osgeo.org
Cc: ZAZHIL-HA HERENA; James McClain; n...@nikosalexandris.net
Subject: Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

> Version says 2.3.2 but libraries say: libgdal.so.20.4.2 .

Libtool number (.so.20.4.2) has nothing to do with user-friendly version
number (2.3.2)

> I am not sure if I
> got the latest code, this is the first time I compile it myself, I used
> this link to download source code:
> http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz

This is the latest release, but /vsihdfs/ is in the development version, not
yet released, so download

https://github.com/OSGeo/gdal/archive/master.zip

--
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

2018-11-23 Thread Even Rouault
> Version says 2.3.2 but libraries say: libgdal.so.20.4.2 .

Libtool number (.so.20.4.2) has nothing to do with user-friendly version 
number (2.3.2)

> I am not sure if I
> got the latest code, this is the first time I compile it myself, I used
> this link to download source code: 
> http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz

This is the latest release, but /vsihdfs/ is in the development version, not 
yet released, so download

https://github.com/OSGeo/gdal/archive/master.zip

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

2018-11-23 Thread ZAZHIL-HA HERENA
Thank you for the reply.

Nikos,

The path is correct, the space was an error in my redaction.

James,

I tried to run a GDALINFO in my Linux command line using the GDAL I configured 
and installed and I am not able to open the raster, even trying to read it from 
local:

  -bash-4.2$ gdalinfo 
/vsihdfs/file:/scratch/zherena/test/write/data/rasters/hawaii.tif
  ERROR 4: 
/vsihdfs/file:/scratch/zherena/test/write/data/rasters/hawaii.tif: No such file 
or directory
  gdalinfo failed - unable to open 
'/vsihdfs/file:/scratch/zherena/test/write/data/rasters/hawaii.tif'.

If I try the same commad withouth /vsihdfs/file:/... it works fine.

I also tried:

  -bash-4.2$ gdalinfo --version
  GDAL 2.3.2, released 2018/09/21

Version says 2.3.2 but libraries say: libgdal.so.20.4.2 . I am not sure if I 
got the latest code, this is the first time I compile it myself, I used this 
link to download source code:  
http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz


And I have a doubt, when configuring I used: --with-hdfs=yes, made another 
using: --with-hdfs = /scratch/zherena/gdal/hadoop/lib/native. Both tries did 
not work. I am not really sure what should I pass as a parameter to --with-hdfs?

This is the list of steps I used to build, I am using Java in my application 
where I plan to use gdal:


  * make clean distclean(per your suggestion)
  *./configure --prefix=/scratch/zherena/gdal/build/gdal-2.3.2/outputb/ 
--with-complete=yes --with-java=yes --with-swig-java=yes 
--with-hdfs=/scratch/zherena/gdal/hadoop/lib/native 
--with-curl=/usr/bin/curl-config
  *   make
  *   make install
  *   cd /swig/java
  *   make veryclean
  *   make
  *   cp *.so ../../outputb/lib/
  *   cp *.jar ../../outputb/lib/


Maybe I am missing something during compile?

Thank you!!
Zazhil-ha

From: gdal-dev  on behalf of James McClain 

Sent: Thursday, November 22, 2018 8:27 PM
Cc: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

Hello,

I am the author of the vsihdfs code, I am ready and willing to help.

I just rebuilt it from current master and was able to successfully open an 
dataset via an HDFS URI with the GDAL Python bindings.  I have a few 
suggestions.

First, please try putting the file into a local directory and try something 
like `gdalinfo /vsihdfs/file:/tmp/kahoolawe.tif` to establish a baseline.

Second, if you are using the Python bindings, please make sure that they have 
been built and installed (and that you are using the ones that you built rather 
than other ones that exist on your system).  Instructions for building the 
Python bindings can be found here: 
https://trac.osgeo.org/gdal/wiki/BuildingOnUnix .

In my case, after building and installing the library and bindings, I was able 
to successfully open a dataset by starting a python REPL like this:

```bash
export 
LD_LIBRARY_PATH=$HOME/local/hadoop-2.7.7/lib/native:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:$HOME/local/gdal-master-vsihdfs/lib
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export CLASSPATH=$($HOME/local/hadoop-2.7.7/bin/hadoop classpath --glob)
PYTHONPATH=$HOME/local/gdal-master-vsihdfs/lib/python2.7/site-packages python
```

then typing this into it:

```python
from osgeo import gdal, gdalconst
ds = gdal.Open('/vsihdfs/file:/tmp/testfile.tif', gdalconst.GA_ReadOnly)
```

(I do not have easy access to and HDFS cluster right at the moment, so I only 
tested a local HDFS URI.)

A note: After having done a build without HDFS support in the tree, I had do a 
`make clean distclean` before I was able to get a build with working HDFS 
support.

Sincerely,
James McClain

On Thu, Nov 22, 2018 at 8:13 PM Nikos Alexandris 
mailto:n...@nikosalexandris.net>> wrote:
* ZAZHIL-HA HERENA mailto:zazhil...@hotmail.com>> 
[2018-11-22 22:35:32 +]:

>Hello, I am not sure if I should use this mailing list to ask questions but I 
>wanted to try, I am a developer trying to use GDAL to open rasters in HDFS.
>
>
>I read in GDAL documentation that starting 2.4 it is possible to open a raster 
>in HDFS. I downloaded and compiled the latest source code available version 
>and the generated libraries show it is 2.4 (libgdal.so.20.4.2). I compiled 
>with option "-with-hdfs=yes" and "--with-java=yes".
>
>I am trying to open a raster using:
>
>
>
>Dataset raster = gdal.Open("/vsihdfs/hdfs://node:8020/user/hdfs 
> /spatial_raster/input_raster/kahoolawe.tif", gdalconst.GA_ReadOnly);

Is your path correct? There is a space here (in "/hfds /").

Nikos

>
>
>but I am getting the following error: "ERROR 4: No such file or directory"

[rest deleted]
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
"I prayed for freedom for twenty years, but received no answer until I prayed 
with my legs."
 -- Frederick 

Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

2018-11-22 Thread James McClain
Hello,

I am the author of the vsihdfs code, I am ready and willing to help.

I just rebuilt it from current master and was able to successfully open an
dataset via an HDFS URI with the GDAL Python bindings.  I have a few
suggestions.

First, please try putting the file into a local directory and try something
like `gdalinfo /vsihdfs/file:/tmp/kahoolawe.tif` to establish a baseline.

Second, if you are using the Python bindings, please make sure that they
have been built and installed (and that you are using the ones that you
built rather than other ones that exist on your system).  Instructions for
building the Python bindings can be found here:
https://trac.osgeo.org/gdal/wiki/BuildingOnUnix .

In my case, after building and installing the library and bindings, I was
able to successfully open a dataset by starting a python REPL like this:

```bash
export
LD_LIBRARY_PATH=$HOME/local/hadoop-2.7.7/lib/native:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:$HOME/local/gdal-master-vsihdfs/lib
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export CLASSPATH=$($HOME/local/hadoop-2.7.7/bin/hadoop classpath --glob)
PYTHONPATH=$HOME/local/gdal-master-vsihdfs/lib/python2.7/site-packages
python
```

then typing this into it:

```python
from osgeo import gdal, gdalconst
ds = gdal.Open('/vsihdfs/file:/tmp/testfile.tif', gdalconst.GA_ReadOnly)
```

(I do not have easy access to and HDFS cluster right at the moment, so I
only tested a local HDFS URI.)

A note: After having done a build without HDFS support in the tree, I had
do a `make clean distclean` before I was able to get a build with working
HDFS support.

Sincerely,
James McClain

On Thu, Nov 22, 2018 at 8:13 PM Nikos Alexandris 
wrote:

> * ZAZHIL-HA HERENA  [2018-11-22 22:35:32 +]:
>
> >Hello, I am not sure if I should use this mailing list to ask questions
> but I wanted to try, I am a developer trying to use GDAL to open rasters in
> HDFS.
> >
> >
> >I read in GDAL documentation that starting 2.4 it is possible to open a
> raster in HDFS. I downloaded and compiled the latest source code available
> version and the generated libraries show it is 2.4 (libgdal.so.20.4.2). I
> compiled with option "-with-hdfs=yes" and "--with-java=yes".
> >
> >I am trying to open a raster using:
> >
> >
> >
> >Dataset raster = gdal.Open("/vsihdfs/hdfs://node:8020/user/hdfs
> /spatial_raster/input_raster/kahoolawe.tif", gdalconst.GA_ReadOnly);
>
> Is your path correct? There is a space here (in "/hfds /").
>
> Nikos
>
> >
> >
> >but I am getting the following error: "ERROR 4: No such file or directory"
>
> [rest deleted]
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev



-- 
"I prayed for freedom for twenty years, but received no answer until I
prayed with my legs."
 -- Frederick Douglass
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

2018-11-22 Thread Nikos Alexandris

* ZAZHIL-HA HERENA  [2018-11-22 22:35:32 +]:


Hello, I am not sure if I should use this mailing list to ask questions but I 
wanted to try, I am a developer trying to use GDAL to open rasters in HDFS.


I read in GDAL documentation that starting 2.4 it is possible to open a raster in HDFS. I 
downloaded and compiled the latest source code available version and the generated libraries show 
it is 2.4 (libgdal.so.20.4.2). I compiled with option "-with-hdfs=yes" and 
"--with-java=yes".

I am trying to open a raster using:



   Dataset raster = gdal.Open("/vsihdfs/hdfs://node:8020/user/hdfs 
/spatial_raster/input_raster/kahoolawe.tif", gdalconst.GA_ReadOnly);


Is your path correct? There is a space here (in "/hfds /").

Nikos




but I am getting the following error: "ERROR 4: No such file or directory"


[rest deleted]
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev