Re: [Geoserver-users] How to serve large spatio-temporal dataset with GeoServer?

2016-06-21 Thread Andrea Aime 
Hi Peter,
from my experience Oracle is probably the cause, given a spatial and a
regular index
it has sometimes issues deciding which one to use and ends up using the
wrong one.
You should try PostGis, with proper indexes and statistics setup, with
"only" 3 million records
it should respond fast.

Worth investigating at the very least, let us know how it goes.

Cheers
Andrea


On Tue, Jun 21, 2016 at 4:22 PM, Peter Kovac 
wrote:

> Dear GeoServer users,
>
> I have trouble figuring out fast and scalable way to serve lightning
> data via GeoServer.
>
> My dataset consists of several million points spread over central Europe
> spanning several years. I have approximately 3M lightning strikes right
> now and it's just a fraction of what I'll have to handle ultimately. I'm
> using Oracle Locator database with both spatial and normal indexes and
> while it has a few quirks it works reasonably well when the amount of
> lightnings is small (i.e. in thousands or tens of thousands).
>
> While my WMS client will never allow to show more than 2 hours worth of
> data (that's small amount of lightning strikes) there are particular WMS
> requests which take a very long time: when I want to see a "bigger
> picture" of all lightning strikes in central Europe during a specified
> short period of time.
>
> The core of the problem is my data have both spatial and temporal
> dimension and there is no spatio-temporal index in Oracle Locator. So
> even if regular index on the time dimension can limit the number of
> features to a few thousands in the blink of an eye, the spatial index
> over the point geometry column won't help much (since the BBOX in the
> request covers the whole area anyway) and is in fact doing harm. The
> query found in GeoServer logs runs really fast If I omit the spatial
> index clause in such case (just a few hundred ms compared to 6-7 seconds
> for the full query with SDO_FILTER function call). Another bad thing is
> my colleagues predict that the performance will get worse with more data
> in the table once it won't fit into RAM and the database engine will
> have to use hard drives for processing.
>
> The performance improves rapidly as I zoom to larger scales (smaller
> areas), where the spatial index selects just a small subset of data.
> However, I'd like to be able to serve the whole central Europe quickly,
> too.
>
> One possible solution is to add the time dimension to my spatial index
> (so it's 3D instead of 2D), but I'm afraid GeoServer won't be able to
> retrieve data from such index (it won't be EPSG:3857 geometry anymore).
>
> Another solution from an Oracle forum suggests using partitioning over
> time and have separate spatial index for each partition, but that
> requires expensive Oracle Enterprise license (which was not budgeted in
> the project of course) and it's just dividing the problem by a constant
> factor anyway.
>
> So, since I'm out of my own ideas, how would you handle this situation?
> What other tools or formats are useful? Is the Postgres/PostGIS combo
> better at serving large-scale spatio-temporal datasets (with regard to
> GeoServer)?
>
>
> Many thanks for any help!
>
> --
> Peter Kovac
> IMS Programmer
> MicroStep-MIS
> peter.ko...@microstep-mis.com
>
>
>
> --
> Attend Shape: An AT Tech Expo July 15-16. Meet us at AT Park in San
> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
> present their vision of the future. This family event has something for
> everyone, including kids. Get more information and register today.
> http://sdm.link/attshape
> ___
> Geoserver-users mailing list
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>



-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or

Re: [Geoserver-users] How to serve large spatio-temporal dataset with GeoServer?

2016-06-21 Thread Jim Hughes 
Hi Peter,

Are you always querying for a short time period?  If so, you might get 
the most mileage out of a SQL database (Oracle/Postgres) by creating an 
index on time and providing any vendor specific query hints to leverage 
that index.  From a GeoTools/GeoServer perspective, WMS/WFS queries are 
turned into SQL queries; how the database handles those is a database 
admin challenge.  With a time-based index and restricted queries, there 
might be a window where performance can be reasonable.

It may be worth considering a distributed database.  If your 
architecture prevents that, then the options are tuning Oracle or 
PostGIS to the max and/or implementing application-based temporal 
sharding.  The former can require expensive licenses (as you mentioned) 
or some fiddling.  An example of temporal-sharding would be having a 
different table for each month or week.  As you query across time, your 
application would know that it has to ask for different layers in 
GeoServer or tables in the database.

If you can use a distributed database, I'd note that there are a number 
of projects to provide geo extensions to popular options such as 
Accumulo, Cassandra, HBase, ElasticSearch, etc.  A number of those 
projects include a GeoTools datastore implementation or a GeoServer 
plugin which makes them compatible with GeoServer.  I am a GeoMesa 
committer and we've had great success using GeoServer to serve up 
feature data and aggregations like heatmaps over datasets scaling to 
billions of records.  Admittedly, setting up a distributed database is 
non-trivial, but it may allow for more options when working with large 
datasets.

Cheers,

Jim

On 06/21/2016 10:22 AM, Peter Kovac wrote:
> Dear GeoServer users,
>
> I have trouble figuring out fast and scalable way to serve lightning
> data via GeoServer.
>
> My dataset consists of several million points spread over central Europe
> spanning several years. I have approximately 3M lightning strikes right
> now and it's just a fraction of what I'll have to handle ultimately. I'm
> using Oracle Locator database with both spatial and normal indexes and
> while it has a few quirks it works reasonably well when the amount of
> lightnings is small (i.e. in thousands or tens of thousands).
>
> While my WMS client will never allow to show more than 2 hours worth of
> data (that's small amount of lightning strikes) there are particular WMS
> requests which take a very long time: when I want to see a "bigger
> picture" of all lightning strikes in central Europe during a specified
> short period of time.
>
> The core of the problem is my data have both spatial and temporal
> dimension and there is no spatio-temporal index in Oracle Locator. So
> even if regular index on the time dimension can limit the number of
> features to a few thousands in the blink of an eye, the spatial index
> over the point geometry column won't help much (since the BBOX in the
> request covers the whole area anyway) and is in fact doing harm. The
> query found in GeoServer logs runs really fast If I omit the spatial
> index clause in such case (just a few hundred ms compared to 6-7 seconds
> for the full query with SDO_FILTER function call). Another bad thing is
> my colleagues predict that the performance will get worse with more data
> in the table once it won't fit into RAM and the database engine will
> have to use hard drives for processing.
>
> The performance improves rapidly as I zoom to larger scales (smaller
> areas), where the spatial index selects just a small subset of data.
> However, I'd like to be able to serve the whole central Europe quickly, too.
>
> One possible solution is to add the time dimension to my spatial index
> (so it's 3D instead of 2D), but I'm afraid GeoServer won't be able to
> retrieve data from such index (it won't be EPSG:3857 geometry anymore).
>
> Another solution from an Oracle forum suggests using partitioning over
> time and have separate spatial index for each partition, but that
> requires expensive Oracle Enterprise license (which was not budgeted in
> the project of course) and it's just dividing the problem by a constant
> factor anyway.
>
> So, since I'm out of my own ideas, how would you handle this situation?
> What other tools or formats are useful? Is the Postgres/PostGIS combo
> better at serving large-scale spatio-temporal datasets (with regard to
> GeoServer)?
>
>
> Many thanks for any help!
>


--
Attend Shape: An AT Tech Expo July 15-16. Meet us at AT Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
___
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net

[Geoserver-users] Geofence questions

2016-06-21 Thread Anton Lundkvist 
Hi!
Using tomcat7, geoserver 2.8 with community Geofence embedded server.

I'm struggling a bit with the Geofence REST API, see docs at
http://docs.geoserver.org/2.8.x/en/user/community/geofence-server/rest-userrole.html#rest-usergroup-service-servicename-users

I'm able to make successful requests (GET, POST) to read users list and
create a user at the following endpoint:

*rest/usergroup/users/*

However to read, alter and delete information on a specific user, I should
be able to make (GET,POST, DELETE) requests to the following endpoint:

*rest/usergroup/*

My question is what does  denote in the above URI?
I have tried the following URI constructions:

*rest/usergroup/*, userName beeing the user name of the user I
want to read or
*rest/usergroup/*

Neither works. Basic auth in requests are working since the first endpoint
described above works. The error message from tomcat gives me method not
allowed and the allowed header gives me only POST and DELETE which is not
consistent with the docs.

Anybody using the Geofence plugin and ran into the same issue?

regards,
Anton
--
Attend Shape: An AT Tech Expo July 15-16. Meet us at AT Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape___
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Re: [Geoserver-users] IDW Interpolation

2016-06-21 Thread ruz76 
I have the same problem with Barnes surface, but I was successful with
Heatmap. So you can try Heatmap instead of Barnes surface. The function may
work different than interpolation, but may be useful for you purposes (I do
not know them). 

http://www.opengis.net/sld
StyledLayerDescriptor.xsd"
   xmlns="http://www.opengis.net/sld;
   xmlns:ogc="http://www.opengis.net/ogc;
   xmlns:xlink="http://www.w3.org/1999/xlink;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
 
   Heatmap
   
 Heatmap
 A heatmap surface showing population density
 
   
 
   
 data
   
   
 weightAttr
 cat
   
   
 radiusPixels
 
   radius
   100
 
   
   
 pixelsPerCell
 10
   
   
 outputBBOX
 
   wms_bbox
 
   
   
 outputWidth
 
   wms_width
 
   
   
 outputHeight
 
   wms_height
 
   
 
   
  


  
the_geom
  0.6
  





  

   
 
   
 


 



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/IDW-Interpolation-tp5271484p5272624.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

--
Attend Shape: An AT Tech Expo July 15-16. Meet us at AT Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
___
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Re: [Geoserver-users] Possible regression in the gs:Contour rendering transformation

2016-06-21 Thread CGA 
Hi Andrea,

I just created a JIRA ticket with all the required data for this bug :
https://osgeo-org.atlassian.net/browse/GEOS-7596

Cyril



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Possible-regression-in-the-gs-Contour-rendering-transformation-tp5271797p5272603.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

--
Attend Shape: An AT Tech Expo July 15-16. Meet us at AT Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
___
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users