Re: [OSGeo-Discuss] Raster catalog ideas [SEC=UNCLASSIFIED]

2010-05-04 Thread Lucena, Ivan

Hi Mike, Bruce,

We went to that discussion a couple of year ago, and by that time there was some strong opinion 
against storing raster data on databases, but it look like time has changed things a little bit. Am 
I right to say that? I can see basically three reasons for that.


1. Data volume.

Raster images are getting more and more accessible and whoever deals with big volume of those are 
starting to feel the pain of managing it on files. I bet that people have started to think about why 
spending so much money buying and maintaining huge Network Attached Storage (NAS) and end up making 
a person do the job of a database server. They must realize that shared folders accessible by end 
users is a problem not a solution. Yes, the systematical and disciplinary use of file based storage 
is possible too, adding to that some sort of raster catalog, but adding to that also the trouble of 
managing permission access on the NAS, backup procedures, temporary files management, and the result 
is a full time position or two.


2. Performance.

The myth says: the is always a overhead to access your raster data from a database in comparison to 
accessing it directly from the file system. If you are talking about a single computer that might 
be the case since fopen(), read(), write() doesn't need to make connections, check permission, 
parse SQL statement to finally figure out where to read BLOBs with the same C functions. But as the 
complexity of your storage grows the file based options will start to experience overhead too and 
there will be not RDBMS server to do connection pooling or caching for you. Again, you can go to the 
whole trouble of managing overviews and tile cache yourself. That means overhead for a file based 
raster data server and more cost of maintaining it. Are we tied?


3. Examples

Satellite companies, data provider, government agency and consulting firms are giving good testimony 
of their use of raster on database, so it might be true for the FOSS options too. The two examples 
provided by Bruce are pretty good. Rasdaman only works with PostgreSQL while Terralib works with 
directly with PostgreSQL, MySQL and commercial RDBMS systems. Both have some big projects to 
showcase their muscles.


Please keep us posted.

Good luck.

Ivan



Bruce Bannerman wrote:

Hi Mike,


A product for managing (muti-dimensional) raster data within a database that I've been monitoring since around 2002 has just been released under GPL. 


They have also applied for OSGeo Incubation.

This product, Rasdaman, can be used with Postgres as its data store.

Rasdaman has its lineage going back to around 1995 (I think). They are claiming 
the French government as a client using the product to manage TB image mosaics.

I understand that there is a sister product that will soon be merged in with 
Rasdaman that offers good OGC support (WCS 1.1 and 2.0, WMS and WCPS).

We intend investigating this product as a potential tool for managing grid, model and multi/hyperspectral remotely sensed data relating to the Climate domain. 


If you look at the product, let me know. I'll be happy to share experiences.



See [1] for Rasdaman product information and [2] for example implementations.


==

Another option is the Brazillian Terralib Project [3].

They have developed an integrated suite of server and framework tools for developing integrated applications based around the use of imagery and the management of imagery within a database (Postgres from memory). 


I'm not doing them justice with this brief description. They have done some 
very impressive work. I'd recommend Gilberto Camara's intro paper at [4] for an 
overview of Terralib.


===


[1] http://www.rasdaman.org/

[2] www.earthlook.org

[3] http://www.terralib.org/index.php

[4] http://www.terralib.org/docs/papers/TerraLib-OSBook-versionJanuary2008.pdf




Bruce Bannerman




From: discuss-boun...@lists.osgeo.org [discuss-boun...@lists.osgeo.org] On 
Behalf Of Mike Toews [mwto...@gmail.com]
Sent: Tuesday, 4 May 2010 6:11 AM
To: discuss@lists.osgeo.org
Subject: [OSGeo-Discuss] Raster catalog ideas

Hi All,

I'm wondering what existing open source options are available to store rasters 
with attributes.

For example, I have several hundred air photos from northern Canada spanning from the 
1940s to now. They have different projections (UTM zones), cell sizes, etc. This data 
store needs to be accessed from both web and desktop GIS software, but needs to have a 
definition query for the year attribute (so I can take all air photos from 
1962, or between 1973 to 1978, or whatever is required).

Our (my company) present solution is to use ESRI's raster catalog on a 
geodatabase. We've had a mixed range of problems on File/Personal/SDE 
Geodatabases. We've experienced corruption on all levels of storage options, so 
we keep our file path attributes to the original GeoTIFFs so the raster 
catalogs can be restored, if 

RE: [OSGeo-Discuss] Open File FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]

2009-08-22 Thread Lucena, Ivan
That is great. Thanks!

-isl


  ---Original Message---
  From: Michael P. Gerlek m...@lizardtech.com
  Subject: RE: [OSGeo-Discuss] Open File   
 FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]
  Sent: Aug 21 '09 13:59
  
  Yes, JP2 supports signed and unsigned types of up to ~24 bits.  And lots of 
 channels (bands).  And alpha masking.  And arbitrary metadata blobs 
 (geospatial and otherwise).
  
  -mpg
  
  
  -Original Message-
  From: discuss-boun...@lists.osgeo.org 
 [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Lucena, Ivan
  Sent: Friday, August 21, 2009 12:22 PM
  To: OSGeo Discussions; OSGeo Discussions
  Subject: RE: [OSGeo-Discuss] Open File 
 FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]
  
  But you can't compress data types other than byte in JPG. Can you do that in 
 JP2K?
  
  
    ---Original Message---
    From: Landon Blake lbl...@ksninc.com
    Subject: RE: [OSGeo-Discuss] Open File 
 FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]
    Sent: Aug 21 '09 12:42
    
    Paul,
    
    I was wondering the same thing.
    
    It seems a little like choosing to drive a Honda Accord, or a Ferrari.
    The Ferrari is a lot faster and comes with a better looking trophy wife
    (or husband), but the Honda is a lot easier to fix. (Try finding an
    affordable Ferrari mechanic in Stockton, California.)
    
    To tie this back into our original discussion, it seems like the
    government should be choosing to drive a Honda Accord when it can,
    instead of the Ferrari.
    
    I guess you'd really have to crunch the numbers and see if the savings
    in bandwidth/disk space costs were really worth the compression savings
    that result from a proprietary compression scheme (wavelet black
    magic).
    
    The problem with this is a lot of the benefits that come from the Honda
    Accord (open image format + open compression algorithm) aren't easily
    calculated in dollars and cents.
    
    Still, this speaks to an important truth I have discovered in open
    source development: Simple is better, even when it isn't necessarily
    faster and smaller.
    
    I'd rather have code that I can understand, or a file format that a
    programmer in 20 years will understand, than a Ferrari you can't drive
    unless you have a PHD and did a thesis on wavelet compression. :]
    
    Landon
    Office Phone Number: (209) 946-0268
    Cell Phone Number: (209) 992-0658
    
    
    
    -Original Message-
    From: discuss-boun...@lists.osgeo.org
    [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Paul Ramsey
    Sent: Friday, August 21, 2009 10:36 AM
    To: OSGeo Discussions
    Subject: Re: [OSGeo-Discuss] Open File
    FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]
    
    So hung up on wavelets, we are.
    
    Internally tiled TIFF with JPEG compression and similarly formatted
    internal overviews can achieve 10:1 compression rates without
    noticeable image quality reductions, and as an added bonus can be
    decompressed a heck of a lot faster than wavelet-based formats. The
    wavelet stuff is k00l, in that there is no need for an overview
    pyramid (it's implicit in the compression math) and much higher
    compression rates can be achieved. But operationally, you can go a
    long way with the more primitive (open image format + open compression
    algorithm) approach.
    
    P.
    ___
    Discuss mailing list
    disc...@lists.osgeo.org
    http://lists.osgeo.org/mailman/listinfo/discuss
    
    
    Warning:
    Information provided via electronic media is not guaranteed against 
 defects including translation and transmission
  errors. If the reader is not the intended recipient, you are hereby notified 
 that any dissemination, distribution or
  copying of this communication is strictly prohibited. If you have received 
 this information in error, please notify the
  sender immediately.
    ___
    Discuss mailing list
    disc...@lists.osgeo.org
    http://lists.osgeo.org/mailman/listinfo/discuss
    
  ___
  Discuss mailing list
  Discuss@lists.osgeo.org
  http://lists.osgeo.org/mailman/listinfo/discuss
  
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


RE: [OSGeo-Discuss] Open File FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]

2009-08-22 Thread Lucena, Ivan
Hi Landon,

It has been an interesting discussion, algorithms copyrights, most used 
formats, limitations, internal details, etc. I definitely agree that it got out 
of control and we should end some place but I and going to give you a quick 
answer.

  What are the limitations of Geotiff/JPEG compared with the proprietary 
 alternatives?

- It is lossy (not that JPEG itself can't be lossless. you can search for 
libjpeg.doc for more info).

- It can't store NDVI, Temperature or any other calculated or remotely sensored 
data in decimal values.

My sincere best regards,

Ivan
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


RE: [OSGeo-Discuss] Open File FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]

2009-08-21 Thread Lucena, Ivan
Bob,

Sorry to mess with discussion but if someone is suggesting the use of Geotiff 
with JPEG compressed as a Open File Format (copied and pasted from the thread 
title) I would like to remember myself and the audience obout the data type 
limitations. 

There's more in raster data than meet the eye, and that usually doesn't come in 
unsigned integer 8  bits. :)

Regards,

Ivan

  ---Original Message---
  From: Bob Basques bob.basq...@ci.stpaul.mn.us
  Subject: RE: [OSGeo-Discuss] OpenFile
 FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]
  Sent: Aug 21 '09 13:27
  
  All,
  
  
  Can someone remind me again, are we talking about saving space,
  or making it easier to implement something . . .  :c)
  
  
  I personally prefer nice simple internal pyramided tiles with
  indexing, about 10% extra space, and very good performance.
  
  
  Someone earlier in this thread spoke about some of these technologies
  being somewhat obsolete what with the new network and bandwidth speeds
  available for publishing.
  
  
  bobb
  
  
   Lucena, Ivan ivan.luc...@pmldnet.com wrote:
  
  
  But you can't compress data types other than byte in JPG. Can you do
  that in JP2K?
  
  
    ---Original Message---
    From: Landon Blake lbl...@ksninc.com
    Subject: RE: [OSGeo-Discuss] Open
  FileFormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]
    Sent: Aug 21 '09 12:42
   
    Paul,
   
    I was wondering the same thing.
   
    It seems a little like choosing to drive a Honda Accord, or a
  Ferrari.
    The Ferrari is a lot faster and comes with a better looking trophy
  wife
    (or husband), but the Honda is a lot easier to fix.
  (Try finding an
    affordable Ferrari mechanic in Stockton, California.)
   
    To tie this back into our original discussion, it seems like
  the
    government should be choosing to drive a Honda Accord when it
  can,
    instead of the Ferrari.
   
    I guess you'd really have to crunch the numbers and see if the
  savings
    in bandwidth/disk space costs were really worth the compression
  savings
    that result from a proprietary compression scheme (wavelet
  black
    magic).
   
    The problem with this is a lot of the benefits that come from the
  Honda
    Accord (open image format + open compression
  algorithm) aren't easily
    calculated in dollars and cents.
   
    Still, this speaks to an important truth I have discovered in
  open
    source development: Simple is better, even when it isn't
  necessarily
    faster and smaller.
   
    I'd rather have code that I can understand, or a file
  format that a
    programmer in 20 years will understand, than a Ferrari you
  can't drive
    unless you have a PHD and did a thesis on wavelet compression.
  :]
   
    Landon
    Office Phone Number: (209) 946-0268
    Cell Phone Number: (209) 992-0658
   
   
   
    -Original Message-
    From: discuss-boun...@lists.osgeo.org
    [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Paul
  Ramsey
    Sent: Friday, August 21, 2009 10:36 AM
    To: OSGeo Discussions
    Subject: Re: [OSGeo-Discuss] Open File
    FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]
   
    So hung up on wavelets, we are.
   
    Internally tiled TIFF with JPEG compression and similarly
  formatted
    internal overviews can achieve 10:1 compression rates without
    noticeable image quality reductions, and as an added bonus can
  be
    decompressed a heck of a lot faster than wavelet-based formats.
  The
    wavelet stuff is k00l, in that there is no need for an
  overview
    pyramid (it's implicit in the compression math) and
  much higher
    compression rates can be achieved. But operationally, you can
  go a
    long way with the more primitive (open image format + open
  compression
    algorithm) approach.
   
    P.
    ___
    Discuss mailing list
    Discuss@lists.osgeo.org
    [LINK: http://lists.osgeo.org/mailman/listinfo/discuss]
  http://lists.osgeo.org/mailman/listinfo/discuss
   
   
    Warning:
    Information provided via electronic media is not guaranteed
  against defects including translation and transmission
  errors. If the reader is not the intended recipient, you are hereby
  notified that any dissemination, distribution or
  copying of this communication is strictly prohibited. If you have received
  this information in error, please notify the
  sender immediately.
    ___
    Discuss mailing list
    Discuss@lists.osgeo.org
    [LINK: http://lists.osgeo.org/mailman/listinfo/discuss]
  http://lists.osgeo.org/mailman/listinfo/discuss
   
  ___
  Discuss mailing list
  Discuss@lists.osgeo.org
  [LINK: http://lists.osgeo.org/mailman/listinfo/discuss]
  http://lists.osgeo.org/mailman/listinfo/discuss
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org

Re: [OSGeo-Discuss] RE: Open source TIN code?

2009-03-02 Thread Lucena, Ivan
Here goes another one from TerraLib: 
http://www.dpi.inpe.br/terralib/html/v320/html/group___math_const.html

  ---Original Message---
  From: Wolf Bergenheim wolf+gr...@bergenheim.net
  Subject: Re: [OSGeo-Discuss] RE: Open source TIN code?
  Sent: Mar 02 '09 19:46
  
  Again this is GPL, not BSD, but GRASS GIS has TIN support:
  
  http://grass.osgeo.org/grass64/manuals/html64_user/v.delaunay.html
  
  --Wolf
  
  On 02.03.2009 20:34, Traian Stanev wrote:
   Not sure about the BSD licensing part, but here are some cool TIN 
 triangulators which are open source.
  
  
   This one (Streaming Delaunay) has BSD-like license, but may be overkill 
 for what you need:
  
   http://www.cs.unc.edu/~isenburg/sd/
  
  
   This one (FIST) has nebulous licensing (different for commercial and 
 non-commercial uses):
  
   http://www.cosy.sbg.ac.at/~held/projects/triang/triang.html
  
  
   And finally, this one is pretty good, but also has unclear licensing:
  
   http://www.cs.cmu.edu/~quake/triangle.html
  
  
  
   Traian
  
  
  
  
   -Original Message-
   From: discuss-boun...@lists.osgeo.org 
 [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek
   Sent: Monday, March 02, 2009 1:17 PM
   To: 'OSGeo Discussions'
   Subject: [OSGeo-Discuss] Open source TIN code?
  
   The Community has need of BSD-licensed source code for TIN generation (in 
 3-space).  It doesn't have to be 
really good, just good enough for some simple demo apps (for example, full-on 
Delauney support not needed).
  
   I know there are a bunch of TIN algs out there on the net in various 
 places, but I don't have much experience 
with any of them.  If anyone has any pointers, I'd appreciate it.
  
   Thanks --
  
   -mpg
  
   ___
   Discuss mailing list
   Discuss@lists.osgeo.org
   http://lists.osgeo.org/mailman/listinfo/discuss
   ___
   Discuss mailing list
   Discuss@lists.osgeo.org
   http://lists.osgeo.org/mailman/listinfo/discuss
  
  --
  
  :3 ) Wolf Bergenheim ( 8:
  
  ___
  Discuss mailing list
  Discuss@lists.osgeo.org
  http://lists.osgeo.org/mailman/listinfo/discuss
  

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


RE: [OSGeo-Discuss] Re: Raster data on RDBMS

2008-10-31 Thread Lucena, Ivan
SyIvan,

I wrote:

   That is not the answer your are waiting for but...

I let did not mention any benefits.

Sorry if lead you to a misunderstanding but thank you for inputs.

Best regards,

Ivan


  ---Original Message---
  From: Sylvan Ascent Inc. [EMAIL PROTECTED]
  Subject: RE: [OSGeo-Discuss] Re: Raster data on RDBMS
  Sent: Oct 31 '08 10:24
  
  Well, now we get to the crux of the matter, what are the benefits? Let's 
 analyze this a bit more to see if anything seems important. You mention:
  
  1) Spatial Extension - not sure what this is, but maybe you can build image 
 operations into the database.
  2) Schemas - Schemas can be queried, put into views, massaged
  3) Metadata - well that's always handy, and likely much easier to maintain 
 and query in a database.
  4) Georeferences definition - Right, like simple features georeferencing, 
 built into the schema
  5) Spatial Indexation - Could make tiles faster to retrieve, coupled with 
 pyramiding, should be decent performance
  
  I'll add this:
  6) As newer data comes in you can more easily upgrade a raster coverage, as 
 the metadata (like the date) can be queried for the latest and greatest, 
 while retaining the older stuff. This might be trickier in a file based 
 system.
  
  and how about the usual rdbms stuff like
  7) Replication - might be useful once in a while, esp in big systems
  8) Scalability (not sure exactly if this is the word, but db vendors/OS 
 projects have put a lot of effort into scaling over lots of users)
  9) Backup
  
  10) Transactions, possibly, if you are bringing raster data in from a 
 satellite and something goes wrong? Unlikely to be to much of a benefit 
 though.
  11) Potentially more robust than using a file system
  
  More anyone? How about disadvantages, like
  
  1) You have to import the raster data into the database.
  2) Have to decide what format/projection/datum to use to store the data.
  3) Possibly more storage is used, but these days who cares?
  4) Tile edge effects (with most compression schemes, there is often a 
 noticeable joint when joining two tiles)
  5) Partial tiles - when you split up an image, it rarely fits perfectly into 
 your chosen tile size. What do you do with the leftovers?
  
  More?
  
  Another 2 cents,
  
  Roger
  
  
  
  From: [EMAIL PROTECTED] on behalf of Lucena, Ivan
  Sent: Fri 10/31/2008 1:20 AM
  To: OSGeo Discussions
  Subject: Re: [OSGeo-Discuss] Re: Raster data on RDBMS
  
  
  
  Paul,
  
  That is not the answer your are waiting for but...
  
  IMHO, once you overcome the mythical concept that a database server will 
 always perform slower than a direct file access then Spatial is not special 
 anymore! [who said that?] and you can think on the benefits just like a 
 banker or an accounting bureau. Database servers in general are capable of 
 making a good use of the available resources. For raster what is needed is a 
 good BLOB support with cursor preferably. Spatial extension and schemas are 
 indispensable accessories, they should provide metadata, georeferences 
 definition, spatial indexation, etc. but they should not drag down the 
 performance.
  
  Just my two cents.
  
  Ivan
  
    ---Original Message---
    From: Paul Ramsey [EMAIL PROTECTED]
    Subject: Re: [OSGeo-Discuss] Re: Raster data on RDBMS
    Sent: Oct 31 '08 02:11
    
    On Thu, Oct 30, 2008 at 5:25 PM, Gilberto Camara
    [EMAIL PROTECTED] wrote:
     but the benefits of having
     raster data on a DBMS are much more important.
    
    And those benefits are?
    ___
    Discuss mailing list
  [EMAIL PROTECTED]
    http://lists.osgeo.org/mailman/listinfo/discuss
    
  ___
  Discuss mailing list
  Discuss@lists.osgeo.org
  http://lists.osgeo.org/mailman/listinfo/discuss
  
  
  
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] Re: Raster data on RDBMS

2008-10-30 Thread Lucena, Ivan
Paul,

That is not the answer your are waiting for but...

IMHO, once you overcome the mythical concept that a database server will always 
perform slower than a direct file access then Spatial is not special anymore! 
[who said that?] and you can think on the benefits just like a banker or an 
accounting bureau. Database servers in general are capable of making a good use 
of the available resources. For raster what is needed is a good BLOB support 
with cursor preferably. Spatial extension and schemas are indispensable 
accessories, they should provide metadata, georeferences definition, spatial 
indexation, etc. but they should not drag down the performance.

Just my two cents.

Ivan

  ---Original Message---
  From: Paul Ramsey [EMAIL PROTECTED]
  Subject: Re: [OSGeo-Discuss] Re: Raster data on RDBMS
  Sent: Oct 31 '08 02:11
  
  On Thu, Oct 30, 2008 at 5:25 PM, Gilberto Camara
  [EMAIL PROTECTED] wrote:
   but the benefits of having
   raster data on a DBMS are much more important.
  
  And those benefits are?
  ___
  Discuss mailing list
[EMAIL PROTECTED]
  http://lists.osgeo.org/mailman/listinfo/discuss
  
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] Raster data on RDBMS

2008-10-29 Thread Lucena, Ivan
Frank,

  My point is that a tile caching approach is really comparing tile caching
  performance to rendering-on-demand performance while I think the original
  point was that rendering-from-database and rendering-from-filesystem could
  have similar performance for input raster data.

D'accord.

  Your comparison is also of interest but I don't think it is fair to compare
  rendering from Oracle through MapServer (or GeoServer) to satisfying
  map requests directly from a tile cache.

I shouldn't have mentioned MapServer. Web-serving wasn't my point to beginning 
with. I can gdal_translate from 
the georaster driver to vrt and open the result on OpenEV. That gives me a good 
idea of how the driver perform in 
a rendering environment. I would need to change OpenEV do it directly. But by 
doing that I would be testing GDAL 
not Oracle. To see Oracle Georaster in action I can use some freeviewer (free 
as in gratis). That is not my point.

The real point is should we discard RDBMS for Raster storage just because we 
are sure that there will be overhead, 
ours direct fopen(), fread() will always be faster? Myth or fact? Those tests 
have proven otherwise so the question 
is what is going own?

I messed around with some free-open-source RDBMS a long time ago (last century 
actually), checking out how to 
create type extension. But I would not imagine getting into to the core of how 
does things work just for the fun of 
it. So, the only thing I can do is to check the results from the outside and 
Oracle+GDAL/GeoRaster is the 
environment that let me do that because they let you use the software without a 
license, as long it is not on 
production mode. 

I should test mySQL+TerraLib or PostGIS+GDAL/PGCHIP also. Maybe.

Best regards,

Ivan

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


RE: [OSGeo-Discuss] Raster data on RDBMS

2008-10-29 Thread Lucena, Ivan
Thanks Jason,

Those results that I reported attest the performance of GDAL tools. You could 
get completely different results if you use tools from other vendors, like XCI, 
ACXG1S and FM3 [1]. There are some options on how that could be implemented but 
I believe we did some good choices in GDAL. 

Best regards,

Ivan

[1] - These names were mixed on purpose ;)

  ---Original Message---
  From: Jason Birch [EMAIL PROTECTED]
  Subject: RE: [OSGeo-Discuss] Raster data on RDBMS
  Sent: Oct 29 '08 15:09
  
  I find this stuff fascinating, but I believe that the Oracle EULA prohibits 
 users from disclosing the results of benchmark tests.  Be careful how you 
 represent these results.
  
  Jason
  
  -Original Message-
  From: Lucena, Ivan
  Subject: [OSGeo-Discuss] Raster data on RDBMS
  
  I would like to return to a discussion that we had months ago about raster 
 on RDBMS. But this time I would like to present some number.
  
  
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] kicked the ESRI habit, now weening from Bentley MicroStation..looking toward FOSS4G

2008-10-28 Thread Lucena, Ivan
Hi Joe,

  I've been researching OS for about one year now and have seen the
  light. Also with an anticipated move outside of the United States soon
  (Brazil, Italy, Greece or Australia), I think having the skills to

  I think I'll leave my rant at that for now - I am also interested in
  various other things like contributing to the OSGeo community..perhaps with
  documentation creation/editing, etc. as it's quite obvious I'm not
  very well versed in computer science/programming..or getting involved with
  GIS in the countries I mentioned previously as I'm considering a move
  or discussing the various Masters programs in GIS out there - but those are
  for another time ;) CHEERS_ joe

If you choose Brazil I would encourage you to take a look on the Master program 
at www.inpe.br. They develop 
and support OS-GIS and I know a lot people that graduated there and found good 
position with the major players 
on the academic and industrial field all over the Word. If that is what you are 
looking for. And it is free... (gratis)

Ivan

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


[OSGeo-Discuss] Raster data on RDBMS

2008-10-28 Thread Lucena, Ivan
Hi There,

I would like to return to a discussion that we had months ago about raster on 
RDBMS. But this time I would like to present some number.

As long as I could recall there was basically two major arguments contrary to 
storing raster on RDBMS. One very pragmatical: Why waste precious process time 
with the overhead of dealing with queries, tables, client-sever back and forth 
just to get the data from BLOB fields on a database when you can get it 
directly from the file system?. The other argument was semantical: Why store 
raster on RDBMS if in general we are not expecting to have a transactions on 
that data?

I cannot argue against the second one. I basically agreed with that but after 
seeing how fragile and complicated even a well defined structure of folders and 
files could be I would vote in favor of the good and old relational model.

That is my experiment. I downloaded two free data samples from Naveteq website. 
Two geotiff files with the same size and number of bands (14336, 14336,  3):

[EMAIL PROTECTED]:~/Data du -k Barcelona_2007_R2C2.TIF
602828  Barcelona_2007_R2C2.TIF
[EMAIL PROTECTED]:~/Data du -k San_Francisco_2006_R1C2.TIF
602828  San_Francisco_2006_R1C2.TIF

Then I loaded those images to Oracle Spatial GeoRaster using GDAL. The loading 
process is comparable than some commercial ETL products on the market. It took 
about 2 minutes to load each image.

[EMAIL PROTECTED]:~/Data time gdal_translate -of georaster 
Barcelona_2007_R2C2.TIF georaster:scott,tiger,orcl,RDT_2$,2
Input file size is 14336, 14336
0...10...20...30...40...50...60...70...80...90...100 - done.
Ouput dataset: (georaster:scott,tiger,orcl,RDT_2$,2) on GDAL_IMPORT,RASTER
real  1m54.973s
user 0m4.368s
sys   0m1.936s

If you are a Oracle GeoRaster users you might be excited about those number 
already but those are not the numbers I want to show. What I would like to do 
is to compare the time that it takes to extract subset from the original 
geotiff and compare with the time to extract the same subset from the RDBMS. He 
are the numbers:

[EMAIL PROTECTED]:~/Data time gdal_translate 
georaster:scott,tiger,orcl,RDT_2$,2 out.tif -srcwin 0 0 2000 2000
Input file size is 14336, 14336
0...10...20...30...40...50...60...70...80...90...100 - done.
real  0m0.720s
user 0m0.408s
sys   0m0.108s

[EMAIL PROTECTED]:~/Data time gdal_translate Barcelona_2007_R2C2.TIF out2.tif 
-srcwin 0 0 2000 2000
Input file size is 14336, 14336
0...10...20...30...40...50...60...70...80...90...100 - done.
real  0m1.177s
user 0m0.976s
sys   0m0.188s

And I also checked the integrity of the results to see if I get the same result:

[EMAIL PROTECTED]:~/Data gdalinfo -checksum out.tif
...
Band 1 Block=2000x1 Type=Byte, ColorInterp=Red
  Checksum=58248
Band 2 Block=2000x1 Type=Byte, ColorInterp=Green
  Checksum=21226
Band 3 Block=2000x1 Type=Byte, ColorInterp=Blue
  Checksum=8002

[EMAIL PROTECTED]:~/Data gdalinfo -checksum out2.tif
...
Band 1 Block=2000x1 Type=Byte, ColorInterp=Red
  Checksum=58248
Band 2 Block=2000x1 Type=Byte, ColorInterp=Green
  Checksum=21226
Band 3 Block=2000x1 Type=Byte, ColorInterp=Blue
  Checksum=8002

What are others test would be interesting to perform?

Best regards,

Ivan












___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] Raster data on RDBMS

2008-10-28 Thread Lucena, Ivan
Paul,

Good thought. 

Let's see. The default blocking used by the GeoRaster driver is (256, 256, 1). 
That is good because GeoTiffs doesn't tile on band space. So I would imagine 
that if I tiled the GeoTiff this way:

[EMAIL PROTECTED]:~/Data time gdal_translate Barcelona_2007_R2C2.TIF 
Barcelona_2007_R2C2_tiled.TIF -co BLOCKXSIZE=256 -co BLOCKYSIZE=256
Input file size is 14336, 14336
0...10...20...30...40...50...60...70...80...90...100 - done.
real  0m42.991s
user 0m20.289s
sys   0m2.516s

The comparison would be fair:

[EMAIL PROTECTED]:~/Data time gdal_translate Barcelona_2007_R2C2_tiled.TIF 
out2.tif -srcwin 0 0 2000 2000
Input file size is 14336, 14336
0...10...20...30...40...50...60...70...80...90...100 - done.
real  0m1.604s
user 0m1.156s
sys  0m0.444s

What do you think?

I would imagine that if I run gdaladdo to add Pyramids on the GeoRaster one 
application could take advantage of it by telling Oracle to cache the BLOB in 
memory. So the next time a user zoom-in the performance would be even better. 

I am trying to setup a mapserver experiment on that issue but for now I would 
like to keep my analysis on that very simple process of extracting a subset. 

Best regards,

Ivan


  ---Original Message---
  From: Paul Ramsey [EMAIL PROTECTED]
  Subject: Re: [OSGeo-Discuss] Raster data on RDBMS
  Sent: Oct 29 '08 05:00
  
  The data is chunked in Oracle into tiles, so unless you tile the TIFF
  as well you aren't really doing a direct comparison. Even if you end
  up with the same numbers for both processes, I'll still be impressed,
  since I assumed Oracle would have a higher overhead.
  
  P.
  
  On Tue, Oct 28, 2008 at 9:54 PM, Lucena, Ivan [EMAIL PROTECTED] wrote:
   Hi There,
  
   I would like to return to a discussion that we had months ago about raster 
 on RDBMS. But this time I would like to present some number.
  
   As long as I could recall there was basically two major arguments contrary 
 to storing raster on RDBMS. One very pragmatical: Why waste precious process 
 time with the overhead of dealing with queries, tables, client-sever back and 
 forth just to get the data from BLOB fields on a database when you can get it 
 directly from the file system?. The other argument was semantical: Why 
 store raster on RDBMS if in general we are not expecting to have a 
 transactions on that data?
  
   I cannot argue against the second one. I basically agreed with that but 
 after seeing how fragile and complicated even a well defined structure of 
 folders and files could be I would vote in favor of the good and old 
 relational model.
  
   That is my experiment. I downloaded two free data samples from Naveteq 
 website. Two geotiff files with the same size and number of bands (14336, 
 14336,  3):
  
   [EMAIL PROTECTED]:~/Data du -k Barcelona_2007_R2C2.TIF
   602828  Barcelona_2007_R2C2.TIF
   [EMAIL PROTECTED]:~/Data du -k San_Francisco_2006_R1C2.TIF
   602828  San_Francisco_2006_R1C2.TIF
  
   Then I loaded those images to Oracle Spatial GeoRaster using GDAL. The 
 loading process is comparable than some commercial ETL products on the 
 market. It took about 2 minutes to load each image.
  
   [EMAIL PROTECTED]:~/Data time gdal_translate -of georaster 
 Barcelona_2007_R2C2.TIF georaster:scott,tiger,orcl,RDT_2$,2
   Input file size is 14336, 14336
   0...10...20...30...40...50...60...70...80...90...100 - done.
   Ouput dataset: (georaster:scott,tiger,orcl,RDT_2$,2) on GDAL_IMPORT,RASTER
   real  1m54.973s
   user 0m4.368s
   sys   0m1.936s
  
   If you are a Oracle GeoRaster users you might be excited about those 
 number already but those are not the numbers I want to show. What I would 
 like to do is to compare the time that it takes to extract subset from the 
 original geotiff and compare with the time to extract the same subset from 
 the RDBMS. He are the numbers:
  
   [EMAIL PROTECTED]:~/Data time gdal_translate 
 georaster:scott,tiger,orcl,RDT_2$,2 out.tif -srcwin 0 0 2000 2000
   Input file size is 14336, 14336
   0...10...20...30...40...50...60...70...80...90...100 - done.
   real  0m0.720s
   user 0m0.408s
   sys   0m0.108s
  
   [EMAIL PROTECTED]:~/Data time gdal_translate Barcelona_2007_R2C2.TIF 
 out2.tif -srcwin 0 0 2000 2000
   Input file size is 14336, 14336
   0...10...20...30...40...50...60...70...80...90...100 - done.
   real  0m1.177s
   user 0m0.976s
   sys   0m0.188s
  
   And I also checked the integrity of the results to see if I get the same 
 result:
  
   [EMAIL PROTECTED]:~/Data gdalinfo -checksum out.tif
   ...
   Band 1 Block=2000x1 Type=Byte, ColorInterp=Red
    Checksum=58248
   Band 2 Block=2000x1 Type=Byte, ColorInterp=Green
    Checksum=21226
   Band 3 Block=2000x1 Type=Byte, ColorInterp=Blue
    Checksum=8002
  
   [EMAIL PROTECTED]:~/Data gdalinfo -checksum out2.tif
   ...
   Band 1 Block=2000x1 Type=Byte, ColorInterp=Red
    Checksum=58248
   Band 2 Block=2000x1 Type=Byte, ColorInterp=Green
    Checksum=21226

Re: [OSGeo-Discuss] some post-FOSS4G thoughts

2008-10-07 Thread Lucena, Ivan
Last year the organization published a very nice report. That will 
certainly take some time do put all the number together as nice I they 
did before but I believe that it will help someone like me, who did not 
attended, to have a good perspective of what we should expect for Sydney.


Dave Patton wrote:

On 2008/10/07 11:04 AM, Landon Blake wrote:
It would be cool if we could get a point location and radius of 
acceptable travel from each OSGeo member. You could then determine

which host cities for a local or regional conference would impact the
most users.


That would only let you look at the impact on
OSGeo members. Even for a regional conference,
that does not include the complete universe of
potential delegates for the conference. I have
no idea whether it would be a useful predictor
of overall conference attendance.


___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] Brief history of GIS OSS (bit off topic?)

2008-08-20 Thread Lucena, Ivan
George,

  I am using to build this system PostgreSQL + PostGIS, QGis and PHP. In
  advance i would like to thank David Bitner who helped me alot by letting
  me use his geocoding function for PostGIS.

I guess you should also find support from a native FOSS project in your 
country, terralib.org. It supports several RDBMS including PostgreSQL (with and 
without PostGIS) and there is also a TerraPHP API. 

You can also find thesis from students from the department where TerraLib is 
developed at http://www.dpi.inpe.br/geopro/teses.html, including mine :)

List of TerraLib projects: 
http://www.terralib.org/php/about.php?body=ListofProjects

There was a article with the INPE's FOSS history on this same website but I 
think they took it off.

Hope it helps,

Best regards,

Ivan
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] South African safety situation

2008-05-30 Thread Lucena, Ivan

Gavin,

My South Africans friends here in the US and I are deeply concern about 
that situation too. I just wanted to add that thanks to economic growth 
and social reforms, Brazil is now a moving target on what concern wealth 
concentration index. Brazilian business, factories and agriculture are 
investing now in Africa. I hope that Word Cup, tourism, conferences and 
other business could bring more opportunities to South African of all 
ethnicity and that will result in peace and prosperity.


Ivan


Gavin Fleming wrote:

Hi all and thanks for bringing it up Paolo

Yes we are sadly going through a spate of unrest and criminal violence in poverty-stricken neighbourhoods in South Africa, mainly against foreigners from other parts of Africa. 'xenophobic attacks' as they are labelled. Believe me, we have to and will get things right - while FOSS4G is a small community, the whole world is gearing up for the 2010 World Cup in SA. 

It has not affected tourists and visitors although some are obviously scared to come at the moment, and it is abating. Of course this situation and the complexities giving rise to it are experienced in other countries, developed and developing, and are not unique to South Africa. Many of the conference topics indeed attempt to address the root causes of situations like this. 

South Africa recently overtook Brazil as the country with the highest Gini coefficient, which is nothing to be proud of. This disparity in wealth distribution despite high levels of so-called economic 'growth' is one of the root causes of dissatisfaction leading to the current situation here.  We also have by some estimates between 5 and 10 million immigrants, labourers and refugees, mainly from Zimbabwe recently but really from all over Africa on top of a local population of about 45 million.  These things coupled with our apartheid history and recent transformation are what make South Africa a fascinating destination and subject of research for people from all over the world. So we hope people will come for some of these reasons and to have a say in solving some of these challenges. 


As it affects potential FOSS4G 2008 delegates, I suggest be aware of the 
situation but don't let it put you off. Our tourism numbers are increasing at 
about 9% per year and last year was another record in terms of numbers of 
overseas visitors. South Africa is a top international big conference venue. 
Cape Town is a tourist mecca rivalling Paris, Sydney, Vancouver etc. The City 
of Cape Town, the conference centre and environs are safe. Whatever appeals to 
you as a tourist, travelling far and wide in South Africa and even neighbouring 
countries, be it our fantastic wildlife and scenery, backpacking, local 
culture, immersing yourself in township life, shark-cage diving, 
whale-watching, whatever, you are bound to have a great conference and 
memorable travels in South Africa.

Gavin 
FOSS4G 2008 conference chair
 



-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Paolo Cavallini
Sent: 30 May 2008 09:27 AM
To: OSGeo Discussions
Subject: [OSGeo-Discuss] South African safety situation

Hi all.
I'm informed, both from the newspapers and from South African friends, 
that there are serious concerns about safety in South Africa. Of course 
this will not affect FOSS4G directly, but I'm sure many of us are 
planning to have some holidays before or after the conference, so not 
being able to tour around freely will detract much from the appeal of 
the conference. This could result in less participation, thus in turn in 
lower general interest.
Any news or indications from local organizers? I think this is a problem 
that should be dealt with, for the best success of the meeting.

All the best.
pc
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss



___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] AAG Boston OSGeo Social Meetup

2008-04-17 Thread Lucena, Ivan

Sorry. I'm sick. Can't go.

Raj Singh wrote:

I'll be there by 9!
---
Raj


On Apr 15, 2008, at 9:51 PM, Christopher Schmidt wrote:

AAG is happening in Boston this week, and in an effort to make some
things happen, I'm planning to have a social meetup this Thursday with
anyone who is interested in coming.

Time: Thur, Apr 17th, 7:45PM
Place: Grendel's Den, Harvard Square, Cambridge
Who: Anyone interested in Open Source Geo
Why: To get together to chat socially about stuff over some beer
 and bar food.

Directions:

Get to Harvard Square. Head south on JFK street past The Garage on
your left. You will cross Mt. Auburn St. On the Southwest corner is a
small open green space: on the opposite corner from JFK is the entrance
to Grendel's, below  Upstairs on the Square.

I'll be there, hopefully by 7:30, with an eeepc and an OSgeo shirt on.

Map: http://tinyurl.com/uof38

See ya there,
--
Christopher Schmidt
Web Developer
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss



___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] AAG Boston Social Next Week

2008-04-10 Thread Lucena, Ivan

Alex,

I am nearby, Central-Mass. I wasn't planning to go to AAG but if there 
is some OSGeo stuff event that I can attend I might change my plans.


Regards,

Ivan


Alex Mandel wrote:
So any progress planning an OSGeo social next week for the AAG in 
Boston. I know we've got at least 10 people to bring together (at least 
5 from UC Davis). Even just picking a hang out for one night would be 
cool, although you had mentioned MIT before...


What do the locals say?
Alex
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss



___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] OS Spatial environment 'sizing'

2008-02-19 Thread Lucena, Ivan

Hi Randy, Bruce,

That is a nice piece of advise Randy. I am sorry to intrude the 
conversation but I would like to ask how that heavy raster 
manipulation would be treated by PostgreSQL/PostGIS, managed or unmanaged?


Best regards,

Ivan

Randy George wrote:

Hi Bruce,

 

On the “scale relatively quickly” front, you should look 
at Amazon’s EC2/S3 services. I’ve recently worked with it and find it an 
attractive platform for scaling http://www.cadmaps.com/gisblog


 

The stack I like is Ubuntu+Java+ Postgresql/PostGIS + Apache2 mod_jk 
Tomcat + Geoserver + custom SVG or XAML clients run out of Tomcat


 

If you use the larger instances the cost is higher but 
it sounds like you plan on some heavy raster services (WMS,WCS) and lots 
of memory will help.


Small EC2 instance provides $0.10/hr:

1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute 
Unit), 160 GB of instance storage, 32-bit platform


 


Large EC2 instances provide $0.40/hr:

7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 
Compute Units each), 850 GB of instance storage, 64-bit platform


 


Extra large EC2 instances $0.80/hr:

15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute 
Units each), 1690 GB of instance storage, 64-bit platform


 

Note: that the instances do not need to be permanent. Some people 
(WeoGeo) have been using a couple of failover small instances and then 
starting new large instances for specific requirements. The idea is to 
start and stop instances as required rather than having ongoing 
infrastructure costs. It only takes a minute or so to start an ec2 
instance. If you are running a corporate service there may be parts of 
the day with very little use so you just schedule your heavy duty 
instances for peak times. If you can connect your raster to S3 buckets 
rather than instance storage you have built in replicated backup.


 

I know that Java JAI can easily eat up memory and is core to Geoserver 
WMS/WCS so you probably want to look at large memory footprint for any 
platform with lots of raster service. I’m partial to Geoserver because 
of its Java foundation.  I think I would try to keep the Apache2 mod_jk 
Tomcat Geoserver on a separate server instance from PostGIS. This might 
avoid problems for instance startup since your database would need to be 
loaded separately. The instance ami resides in a 10G partition the 
balance of data will probably reside on a /mnt partition separate from 
ec2-run-instances. You may be able to avoid datadir problems by adding 
something like Elastra to the mix. Elastra beta is a wrapper for 
PostgreSql that puts the datadir on S3 rather than local to an instance. 
I suppose they still keep indices(GIST et al) on the local instance.


(I still think it an interesting exercise to see what could be done 
connecting PostGIS to AWS SimpleDB services.)


 


So thinking out loud here is a possible architecture–

Basic permanent setup

put raster in S3 – this may require some customization of Geoserver,

build a datadir in a PostGIS and backup to S3

create a private ami for Postgresql/PostGIS

create a private ami for the load balancer instance

create a private ami with your service stack for both a small and large 
instance for flexibility,


   Startup services

start a balancer instance

point your DNS CNAME to this balancer instance

start a PostGis instance (you could have more than one if necessary but 
it would be easier to just scale to a larger instance type if the load 
demands it)


have a scripted download from an S3 BU to your PostGIS datadir (I’m 
assuming a relatively static data resource)


   Variable services

start service stack instance and connect to PostGIS

update balancer to see new instance – this could be tricky

repeat previous  two steps as needed

at night scale back – cron scaling for a known cycle or use a controller 
like weoceo to detect and respond to load fluctuation


 

By the way the public AWS ami with the best resources that I have found 
is Ubuntu 7.10 Gutsy. The debian dependency tools are much easier to use 
and the resources are plentiful.


 

I’ve been toying with using an AWS stack adapted for serving some larger 
Postgis vector sets such as fully connected census demographic data and 
block polygons here in US. The idea would be to populate the data 
directly from the census SF* and TIGER with a background Java bot. There 
are some potentially novel 3D viewing approaches possible with xaml. 
Anyway lots of fun to have access to virtual systems like this.


 


As you can see I’m excited anyway.

 


randy

 

 

*From:* [EMAIL PROTECTED] 
[mailto:[EMAIL PROTECTED] *On Behalf Of 
[EMAIL PROTECTED]

*Sent:* Monday, February 18, 2008 6:35 PM
*To:* OSGeo Discussions
*Subject:* [OSGeo-Discuss] OS Spatial environment 'sizing'

 



IMO:


Hello everyone,

I'm trying to get a feel for server 'sizing' for a **hypothetical** 
Corporate 

Re: [OSGeo-Discuss] OS Spatial environment 'sizing' + Image Management

2008-02-19 Thread Lucena, Ivan
 as needed
 
  Sorry I haven't had time to try this so it is just theoretical. Of course
  you can go traditional and just keep the coverage imagery files on 
the local

  instance avoiding the S3 proxy idea. The reason I don't like that idea is
  the imagery has to be loaded with every instance creation while an S3
  approach would need only one copy.
 
 
  randy
 
  -Original Message-
  From: Lucena, Ivan [mailto:[EMAIL PROTECTED]
  Sent: Tuesday, February 19, 2008 2:59 PM
  To: [EMAIL PROTECTED]; OSGeo Discussions
  Subject: Re: [OSGeo-Discuss] OS Spatial environment 'sizing'
 
  Hi Randy, Bruce,
 
  That is a nice piece of advise Randy. I am sorry to intrude the
  conversation but I would like to ask how that heavy raster
  manipulation would be treated by PostgreSQL/PostGIS, managed or 
unmanaged?

 
  Best regards,
 
  Ivan
 
  Randy George wrote:
   Hi Bruce,
  

  
   On the scale relatively quickly front, you should 
look
   at Amazon's EC2/S3 services. I've recently worked with it and find 
it an

   attractive platform for scaling http://www.cadmaps.com/gisblog
  

  

   The stack I like is Ubuntu+Java+ Postgresql/PostGIS + Apache2 mod_jk
   Tomcat + Geoserver + custom SVG or XAML clients run out of Tomcat
  

  

   If you use the larger instances the cost is higher but
   it sounds like you plan on some heavy raster services (WMS,WCS) and 
lots

   of memory will help.
  
   Small EC2 instance provides $0.10/hr:
  
   1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 
Compute

   Unit), 160 GB of instance storage, 32-bit platform
  

  

   Large EC2 instances provide $0.40/hr:
  
   7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2
   Compute Units each), 850 GB of instance storage, 64-bit platform
  

  

   Extra large EC2 instances $0.80/hr:
  
   15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 
Compute

   Units each), 1690 GB of instance storage, 64-bit platform
  

  

   Note: that the instances do not need to be permanent. Some people
   (WeoGeo) have been using a couple of failover small instances and then
   starting new large instances for specific requirements. The idea is to
   start and stop instances as required rather than having ongoing
   infrastructure costs. It only takes a minute or so to start an ec2
   instance. If you are running a corporate service there may be parts of
   the day with very little use so you just schedule your heavy duty
   instances for peak times. If you can connect your raster to S3 buckets
   rather than instance storage you have built in replicated backup.
  

  

   I know that Java JAI can easily eat up memory and is core to Geoserver
   WMS/WCS so you probably want to look at large memory footprint for any
   platform with lots of raster service. I'm partial to Geoserver because
   of its Java foundation.  I think I would try to keep the Apache2 
mod_jk
   Tomcat Geoserver on a separate server instance from PostGIS. This 
might
   avoid problems for instance startup since your database would need 
to be

   loaded separately. The instance ami resides in a 10G partition the
   balance of data will probably reside on a /mnt partition separate from
   ec2-run-instances. You may be able to avoid datadir problems by adding
   something like Elastra to the mix. Elastra beta is a wrapper for
   PostgreSql that puts the datadir on S3 rather than local to an 
instance.

   I suppose they still keep indices(GIST et al) on the local instance.
  
   (I still think it an interesting exercise to see what could be done
   connecting PostGIS to AWS SimpleDB services.)
  

  

   So thinking out loud here is a possible architecture-
  
   Basic permanent setup
  
   put raster in S3 - this may require some customization of Geoserver,
  
   build a datadir in a PostGIS and backup to S3
  
   create a private ami for Postgresql/PostGIS
  
   create a private ami for the load balancer instance
  
   create a private ami with your service stack for both a small and 
large

   instance for flexibility,
  
  Startup services
  
   start a balancer instance
  
   point your DNS CNAME to this balancer instance
  
   start a PostGis instance (you could have more than one if necessary 
but

   it would be easier to just scale to a larger instance type if the load
   demands it)
  
   have a scripted download from an S3 BU to your PostGIS datadir (I'm
   assuming a relatively static data resource)
  
  Variable services
  
   start service stack instance and connect to PostGIS
  
   update balancer to see new instance - this could be tricky
  
   repeat previous  two steps as needed
  
   at night scale back - cron scaling for a known cycle or use a 
controller

   like weoceo to detect and respond to load fluctuation
  

  
   By the way the public AWS ami with the best resources that I have 
found
   is Ubuntu 7.10 Gutsy. The debian

Re: [OSGeo-Discuss] Planet OSGeo

2008-02-13 Thread Lucena, Ivan

Me too. Some time I feel like blogging.

Cameron Shorter wrote:

I like the idea and would like to provide occasional content.


___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] Does Open Source need a supervisory government body?

2008-01-25 Thread Lucena, Ivan

Bruce,

That sounds like removing the F from FOSS or should I say, that is a 
bazaar inside a cathedral. :)


Seriously now, IMHO, as an FOSS contributor and a commercial software 
developer that uses FOSS, I believe that there is a complicated process 
of getting to the point to embrace a FOSS initiative and that statement 
does not help it at all. Where are the decisions made, in public 
e-mail-lists or in a cabinet? What about election and cabinet change?


I am not saying that a government agency can't be the incubator of a 
*F*OSS, there are numerous successful example out there, but the 
governance of the project matter a lot. If you love your OSS project 
set it free.


Best regards,

Ivan


[EMAIL PROTECTED] wrote:


IMO:


Sorry for the inflamatory subject heading. I'm hoping to get a few bites 
with my fishing...




I'm currently reviewing a high level government strategy paper (in 
draft) and intend submitting a formal response.


I'd like to see some discussion on the subject by my respected 
colleagues prior to making the submission.



The gist of the comment in the draft strategy is something like:

Open Source approaches to software development will be most effective 
if some form of central authority undertakes the role of verifying 
contributions and providing quality control.





My initial reaction and response to this is something like:

This is a misreading of how Open Source works.

Successful Open Source Projects typically have software of superior 
quality. This is usually due to there being many developers who have 
access to the software for QA purposes.


Any attempt to impose a central authority from outside of Open Source 
projects would be rebuffed vigorously and result in a probably 
irrepairable relationship between that party and the project(s) involved.


The most successful centralised Open Source authority is probably the 
Apache Foundation (http://www.apache.org/) which is behind a wide range 
of projects including the Apache Web Server, probably the most widely 
used Web Server on the Internet. The Foundation pioneered the concept of 
'Meritocracy', where people earn respect and are given greater 
responsibility for projects based on their past contributions and 
'merit'. The Foundation grew from within the Project. It was not imposed 
on the Project. They have developed an enviable reputation for spawning, 
incubating and fostering robust Open Source Projects that routinely 
produce high quality software.


Nearly two years ago, an organisation called the Open Source Geospatial 
Foundation (OSGEO,  http://www.osgeo.org/) was formed based on the 
Apache ethos, to provide similar support for Open Source Spatial 
applications. They currently have a number of prominent spatial projects 
in Incubation with a number of other equally capable projects waiting 
for the next vacancy for incubation.



OK, over to you. I'm interested in all points of view on this issue.


Bruce Bannerman





Notice:
This email and any attachments may contain information that is personal, 
confidential,
legally privileged and/or copyright. No part of it should be reproduced, 
adapted or communicated without the prior written consent of the 
copyright owner.


It is the responsibility of the recipient to check for and remove viruses.

If you have received this email in error, please notify the sender by 
return email, delete it from your system and destroy any copies. You are 
not authorised to use, communicate or rely on the information contained 
in this email.


Please consider the environment before printing this email.

 

 

 





___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] Re: FOSS4GIS business models

2008-01-08 Thread Lucena, Ivan

Paolo,

I just got the latest issue of Dr. Dobbs Magazine with this cover-page 
article South American Software Development:


http://www.ddj.com/architect/205600791;jsessionid=AXESE4MZSIY54QSNDLPCKHSCJUNN2JVN

This article is more focused on Brazil than anywhere else in South 
America. It give a little light on the cultural differences that could 
lead to misunderstanding from outsiders. The article does mention Forum 
participation and language barriers as an issue.


Ciao,

Ivan


Paolo Cavallini wrote:

Lucena, Ivan ha scritto:


I believe that TerraLib would deserve a better technical look than
what I did but my initial impression was very favorable. What impress me
the most was the raster-on-rdbms support.

...

Talking about integration with other OSGeo projects I believe that the
current TerraLib RC uses GDAL.


What I really miss on Terra* is the community: I tried several times to
contact it, especially to help having updated debian packages, but never
get a repy, something unusual for open projects.
All the best.
pc

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] Re: FOSS4GIS business models

2008-01-07 Thread Lucena, Ivan

Hi Bruce,

[EMAIL PROTECTED] wrote:

wrt the Brazillian TerraLib toolkit mentioned in your paper:

- I've had a quick look at the web site. The product appears to be quite 
mature and functional.


- Has anyone from this list had a technical look at the products and 
like to share their observations? Can they be integrated with OSGeo 
apps? Do they support OGC standards etc?


Bruce Bannerman


I believe that TerraLib would deserve a better technical look than 
what I did but my initial impression was very favorable. What impress me 
the most was the raster-on-rdbms support.


I download and installed the TerraView application and imported some 
raster data files to PostgreSQL and it works like a charm, but again, 
that would deserve a performance evaluation.


The source code repository is not as open as GDAL (for example) but I 
believe that out-side contributors should be able to suggest 
modification by sending CVS patches at least.


There are very good (normally expensive) image processing algorithms 
implemented on the library, e.g. segmentation, Wavelets. There is a rich 
set of vectors algorithms too.


Talking about integration with other OSGeo projects I believe that the 
current TerraLib RC uses GDAL.


Best regards,

Ivan




___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] Re: FOSS4GIS business models

2008-01-03 Thread Lucena, Ivan

Hi all,

I am *not* going to disagree with Andrea, Gilberto, Paul, Howard or 
anybody else. I just want to point out a interesting open source 
business model that is making a big impact this days. I am talking about 
Xen [http://en.wikipedia.org/wiki/Xen].


I keep reading news and more news about new commercial products from big 
software companies based on Xen. Is that possible on the GIS world?


Best regards,

Ivan

Gilberto Camara wrote:

Dear OSGEO Discussion List members:

Paul Ramsey´s remarks are right on target.

First, GIS is a large arena and there are
different motivations for developers, that
prevent them from joining a single project such as uDIG.

Second, it is very difficult for a private
company to develop a world-class FOSS4G product
and survive based only on consulting
fees for the commercial sector.

Third, to overcome these limitations there is
a need for governmental intervention, which may
be direct, as in the case of Catalonian government´s
support for gvSIG, or indirect, as in the decision
of Germany to support open source software.

In Brazil, the National Institute for Space Research (INPE)
has been supporting local GIS development for 25 years,
with a lot of success in our national user community.
Without official support, there would be no local FOOS4G
development in Brazil.

In 2003, I did a F00S4G market survey and published the
results as a chapter of a US National Academy of Sciences book:
Open Source GIS Software: Myths and Realities
www.dpi.inpe.br/gilberto/papers/camara_open_source_myths.pdf.

We analysed 70 FOSS4G software projects taken from the
FreeGIS list, and divided them into three categories:
networked products (e.g. GRASS), corporate products (e.g., PostGIS)
and individual products (e.g., CAVOR). From each product,
we assessed its maturity, level of support and functionality.

Our main conclusions at the time were:
(a) Only 6% of the  products were developed by networked teams.
Thus, the “Linux paradigm” is the exception rather than the rule.
(b) Corporations (private or public) are the main developers of
successful open source products. Corporations account for 41% of
all products.
(e) Individual-led software (a small team of 1-3 people) have
less quality and more mortality than the above.

These results show that the impetus behind successful
open source software was not coming from altruistic individuals
working in the midnight hour, but from professional programmers.
I consider that a similar result would be obtained today, should
the assessment be repeated.

This analysis was further elaborated in a JASIST paper:
Information Policies and Open Source Software in Developing Countries
www.dpi.inpe.br/gilberto/papers/camara_fonseca_jasist.pdf.

For the FOSS4G effort to be fruitful and sustainable,
we need a very informed and candid assessment of our
business model. My personal view, based on 25 years of experience,
is that government intervention is essential for the open source
model to survive beyond a handful of examples.

Best regards
Gilberto

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] GIS Data Models

2007-11-30 Thread Lucena, Ivan

Landon,

There it goes:

Landon Blake wrote:
Does anyone know if there has been work on GIS Data Models besides any 
organization other than ESRI? 
(http://www.esri.com/software/arcgis/geodatabase/about/data-models.html)


There it goes:

SPRING: Integrating Remote Sensing and GIS with Object-Oriented Data 
Modelling. G.Câmara, R.Souza, U.Freitas, J.Garrido, F. Ii. *Computers 
and Graphics*, vol.15(6):13-22, 1996.


Note: Ivan Lucena on page 15 is me.

However, this material is definitely written for a user of ESRI 
software. It is possible to extract basic principles from the material 
ESRI produces on data models, although this can be difficult given the 
amount of software specific content.


Sure.

Has there been any effort by the open source community to develop GIS 
Data Models? (By a GIS Data Model I mean a template or set of guidelines 
for one or more thematic layers and the features they contain as these 
layers apply to a particular application. For example: Agriculture)


Just as an example, using that software above mentioned I once developed 
a Data Model for research in Precision Agriculture. Basically what you 
do is given a source of datasets in the real word, like Geological Map 
or Altimetry you take the class that best represent it, like Thematic 
or Numeric and you give it a name and symbology appropriated for your 
application domain. You do that previous to the data acquisition and you 
can use the schema you used in several different projects.


Note: The physical data storage is trick tough, by the concept of 
multi-representation you can have in a database one single Thematic 
layer represented by vector and/or raster. For Numeric layers is even 
tricker, it could have Vector (contour map, triangular grid, 3D points) 
or Raster (regular grid) for the same Layer. How does it sounds? I 
never rear of any other software that does that. Have you?


I am starting work on a data model for Survey Control as part of my 
efforts at the SurveyOS Project and at my day job. As part of this work 
I would like to develop some tutorials and templates for data model 
design that could be used by others in the FOSS GIS arena. These data 
model patterns will focus on “vendor-neutral” GIS design. I hope to work 
on other data models as the years pass, and most of these will be survey 
related.


You can play if that software to get some ideas but it is not exactly 
the neutral solution you want:


http://www.dpi.inpe.br/spring/english/index.html

I am curious if there has been work like this done before. For example, 
I’ll need to define some abstract data types for Feature attributes that 
could be “mapped” to various software platforms and/or programming 
languages.


Again, Not exactly. It answers you first question but not the second one.



 


Any thoughts?


It is certainly a cool topic. :)

Ivan



 


SLB

 

 




*Warning:
*Information provided via electronic media is not guaranteed against 
defects including translation and transmission errors. If the reader is 
not the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited.  If you have received this information in error, please 
notify the sender immediately.





___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] idea for an OSGeo project -- a new, open data format

2007-11-15 Thread Lucena, Ivan

Sampson,

I am not a GML guru and I don't know if a binary version exists already, 
but I would imagine that HDF5 would be a excellent choice by its own 
hierarchical nature. I mean, we can use GML as a schema to store the 
data in binary format in the HDF5 format.


Best regards,

Ivan

Sampson, David wrote:

Alright,

Here are some other thoughts.

First off what about a open office (open base) type approach... This
mimmics the ESRI MSAccess approach and seams to work well for non server
environments. Also open office is a good environment for some basic
applications.

Next, what ever happened to the adoption of GML... Was GML not supposed
to be the NEXT interchange fomrat?  Perhaps this is a good discussion to
include the GML gurus in. The whole discussion of going with a binary
GML format makes sense and GML is already used for many web mapping
(feature) services. It sounds like a duplication of GML to me... Unless
someone can offer a direct compare and contrast between the concept here
and the GML/Binary GML concept.

In either case being able to convert to and from GML would be a necesity
for wide adoption IMHO.

Another thought is to encourage some of the proprietary formats to open
up. What would it take to get ESRI on board to open up the format (open
as in free speech). What about other non-open standards? Once it's open
then we can bring the SHP format to modern day useage. Surely much of
the format could be salvaged.

Besides, if you want wide adoption of an open format then why not go for
those players who hold greatest market share.

Some thoughts.

-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of P Kishor
Sent: Tuesday, November 13, 2007 09:53
To: OSGeo Discussions
Subject: [OSGeo-Discuss] idea for an OSGeo project -- a new, open data
format

So, I am thinking, Shapefile is the de facto data standard for GIS data.
That it is open (albeit not Free) along with the deep and wide presence
of ESRI's products from the beginning of the epoch, it has been widely
adopted. Existence of shapelib, various language bindings, and ready use
by products such as MapServer has continued to cement Shapefile as the
format to use. All this is in spite of Shapefile's inherent drawbacks,
particularly in the area of attribute data management.

What if we came up with a new and improved data format -- call it Open
Shapefile (extension .osh) -- that would be completely Free,
single-file based (instead of the multiple .shp, .dbf, .shx, etc.), and
based on SQLite, giving the .osh format complete relational data
handling capabilities. We would require a new version of Shapelib,
improved language bindings, make it the default and preferred format for
MapServer, and provide seamless and painless import of regular .shp data
into .osh for native rendering. Its adoption would be quick in the open
source community. The non-opensource community would either not give a
rat's behind for it, but it wouldn't affect them...
they would still work with their preferred .shp until they learned
better. By having a completely open and Free single-file based, built on
SQLite, fully relational dbms capable spatial data format, it would be
positioned for continued improvement and development.

Is this too crazy?

--
Puneet Kishor
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss



___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

2007-11-14 Thread Lucena, Ivan

Frank,

I was watching this PyTables video 
[http://www.carabos.com/videos/pytables-1-intro] and one thought came to 
my mind: HDF5 can easily be used to store and retrieve vector, raster 
and attribute tables. We would need to standardize a schema tough.


Best regards,

Ivan

PS. I am not that Ivan on that video.

Frank Warmerdam wrote:

Landon Blake wrote:

P.S. - This is probably a crazy idea, but has anyone ever considered
talking to ESRI about cooperating on an update Shapefile spec?


Landon,

I believe ESRI sees the file based geodatabase as filling roughly the
role that the Shapefile played in the ArcView 3.x days.  Of course it isn't
clear that it will be very open and interoperable with other software 
packages.


I can't imagine a upgrade to shapefiles that would satisfy all our
requirements that has anything but the name in common with the existing
shapefiles.  So why drag ESRI into it?

Best regards,

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss