Re: [postgis-users] Geoprocessing & BigData

[David Haynes]

We have done some work, implementing parallel spatial queries using a
spatial declustering algorithm. How large are your datasets?

On Mon, Jan 18, 2016 at 1:51 PM, Rémi Cura  wrote:

> Hey,
> if you have one beefy server you can parallelize throwing several queries
> working on sub set of your data.
> (aka parallel processing trough data partition).
> One conceptual example : you want to process the world, you create 20
> workers, a list of countries, and then make the worker process the list
> country by country.
>
> If you think one postgres server will not be sufficient,
> you could of course shard your data across several servers,
> with options ranging from writting from scratch (you rewrite everything),
> to using existing open source code, to dedicated solution like
>  Postgresql-Xc, greenplum, ...
>
> However, sorry to say this but in your case it looks like your first
> improvement step will not come from massive paralleling but from first
> better understanding the world of geospatial data and postgis.
>
> Cheers,
> Rémi-C
>
> 2016-01-18 19:30 GMT+01:00 Vincent Picavet (ml) :
>
>> Hi Ravi,
>>
>>
>>
>>
>> On 18/01/2016 19:14, Ravi Pavuluri wrote:
>> > Hi All,
>> >
>> > I am checking if there is a way to process quickly large datasets such
>> > as census blocks in PostGIS and also by leveraging big data platform. I
>> > have few questions in this regard.
>> >
>> > 1) When I try intersect for sample census blocks with another polygon
>> > layer, PostGIS 2.2(on Postgres 9.4) takes ~60 minutes (after optimizing
>> > from http://postgis.net/2014/03/14/tip_intersection_faster/ ) while on
>> > ESRI ArcMap takes ~10 minutes. PostGIS layers already have geospatial
>> > indices. Is there anyway to optimize this further?
>>
>> Following the links on your page, here is a good answer from Paul (TL;DR
>> : st_intersection is slow, avoid it) :
>>
>> http://gis.stackexchange.com/questions/31310/acquiring-arcgis-like-speed-in-postgis/31562
>>
>> > 2) What is an equivalent of ESRI Union in PostGIS? I didn't see any out
>> > of the box functions and any tips here are appreciated.
>>
>> If ESRI Union makes a union, maybe st_union ? But I guess there are some
>> semantic issues here.
>>
>> > 3) Is there anyway we can expedite these geoprocessing
>> > tasks(union/intersect etc) using big data platform (Ex: hadoop)? Most
>> > examples talk about analysis (contains etc)  but not about geoprocessing
>> > on geospatial data. Any input is appreciated.
>>
>> Lots of people do geoprocessing too with PostGIS, including long-running
>> jobs on large volumes of data ( worldwide osm data processing namely).
>> "Big data" is a really subjective word. Are your geoprocessing needs
>> really parallelizable ? What kind of volumes are we talking about ? MB,
>> GB, TB ? What kind of hardware do you have at hand ?
>>
>> One way to do some sort of map-reduce with PostGIS is to use a bunch of
>> servers with FDW connections between a source master and these slaves,
>> map the data processing to the slave servers and reduce it on the main
>> server. With a bit of Python as glue code this can be automated and
>> quite efficient, even though this kind of sharding is not automated (
>> yet ?).
>>
>> Vincent
>>
>> >
>> > Thanks,
>> > Ravi.
>> >
>> >
>> > ___
>> > postgis-users mailing list
>> > postgis-users@lists.osgeo.org
>> > http://lists.osgeo.org/mailman/listinfo/postgis-users
>> >
>>
>> ___
>> postgis-users mailing list
>> postgis-users@lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/postgis-users
>
>
>
> ___
> postgis-users mailing list
> postgis-users@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/postgis-users
>
___
postgis-users mailing list
postgis-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/postgis-users

Re: [postgis-users] questions/issues when creating a topology

[Lucas Ferreira Mation]

2) update. Apparently my "manual" brute force insertion does not work.

UPDATE temp_geom_bahia2 SET tg_geom_dump_utm = '(28,1,13314,3)' WHERE
cod_setor = '29039040536'

The desired cell in column tg_geom_dump_utm  of table  temp_geom_bahia2 is
filled. But the topogeom still displays a hole for that area when I see it
in Qgis (even after updating).

Any ideas on how to insert that?


regards
Lucas





2016-01-25 13:26 GMT-02:00 Lucas Ferreira Mation :

> Sandro,
>
> tks very much for the answers. After week to parse and try to implement
> your sugestions I think we made good progress. Of of the 316k polygons, we
> get 308 polygons that were not included.
> Bellow are some follow-up questions (in line bellow).
>
>
>> Message: 1
>> Date: Sun, 17 Jan 2016 14:23:25 +0100
>> From: Sandro Santilli 
>> To: PostGIS Users Discussion 
>> Subject: Re: [postgis-users] questions/issues when creating a topology
>> Message-ID: <20160117132325.GA8245@localhost>
>> Content-Type: text/plain; charset=us-ascii
>>
>> On Thu, Jan 14, 2016 at 09:57:48AM -0200, Lucas Ferreira Mation wrote:
>>
>> > 1) What is the role of ST_Force2D() function in
>> >  toTopoGeom(ST_Force2D(geom),...) ? The code fails if I omit this
>> function.
>> > However my data is already 2D.
>>
>> ST_Force2D should make no difference when the input is already 2D.
>> If it does there's a bug hiding somewhere and you're welcome to
>> file a ticket about it (with a way to reproduce).
>>
>>
> I actually managed to run most test cases without  the ST_Force2D()
> function. The only case it did make a difference was when creating a
> topology with the original SRID (4674 , SIRGAS 2000) and no tolerance
> parameter. With the  ST_Force2D() wrapipng the input polygons, the loop
> runs. Without it, there is a crash in the database (connection lost with
> database).
>
>
>> > 2) In most test I did, I got an error when uploading one or two polygons
>> > (out of a 900 polygons in Acre state). How can I add these afterward?
>> > Assume all the edges that define those error-polygons have already been
>> > added. How do I define a new face  by those edges and attribute that to
>> the
>> > topogeom for these error-polygons?
>> >
>> > Let's say polygon A can be defined by edges 1, 2, 3 and 4, which are
>> > already in the topology when other polygons were inserted. How do I
>> define
>> > face A and attribute that to the topogeom?
>>
>> Faces are automatically defined. You associate attributes with
>> TopoGeometry objects, which you can define in your case by specifying
>> the faces making up the polygon.
>> See http://postgis.net/docs/manual-dev/CreateTopoGeom.html
>>
>>
>  Ok, got it to work. To document for others. Fist I used QGIS to know the
> original polygon - face pairs. This is done manually, by overlaying
> serveral layers (orginal polygons, topgeom, face bounding boxes). For
> example, from the atached image, I learnt that  I should associate the
> polygon cod_setor=29039040536 with face_id=13314 of the corresponding
> topology. This, this fixes the "hole" in the topology for that polygon
>
> UPDATE temp_geom_bahia2 SET tg_geom_dump_utm = '(28,1,13314,3)' WHERE
> cod_setor = '29039040536'
>
> You may also try to re-run toTopoGeom on the previously failed
>> polygons. Sometimes the function works when finding a different
>> starting state.
>>
>
> This worked for 50 aditional cases of of the 358 errors (on the test with
> UTM reprojected polygons and with tolerance parameter of 1m).
>
> I'll now try to increase the tolerance parameter (to say 2m, 5m, 10m) to
> see if the remaining polygons get added to the topology.
>
>
>>
>> > 3) Is there a way to manually edit topologies, in QGIS for instance? I'm
>> > thinking of a workflow for repairing a map. After doing the automatic
>> fixes
>> > in Postgis and creating a topology, someone wants to manually correct
>> some
>> > feature. For instance, move an edge or a point, while maintaining the
>> > topological properties. Is there a QGIS plugin  for that?
>>
>> https://plugins.qgis.org/plugins/pgtopoeditor/
>>
>>
> I see cool options to remove find dangling edges, remove edges and points
> and find Orpaned topogeoms. But supose I wanto to change the shape of an
> edge, by mooving a few vertices among it's path in QGIS, while preserving
> the nodes where it intersects with other edges. Is it possible to do that
> (and preserve the topological properties)? What about moving nodes? Can
> pgtopoeditor manage that?
>
>
>
>> > 4) Is not including a tolerance parameter  in to... equivalent to a zero
>> > tolerance?
>>
>> Yes. This is something that might be good to change in the future
>> but for now it is the same.
>>
>> > 5) Does topology behaves differently depending on whether the SRID is in
>> > decimal degrees or meters? Ideally, I would prefer to have the tolerance
>> > parameter in meters, as it is a more intuitive unit and have a constant
>> 

[postgis-users] [conference] Submit your speaker proposal to FOSS4G-NA in Raleigh this May

[Sarah Cordivano]

Hi PostGIS-ers,

I’m writing to encourage you to consider a proposal submission to the Free
and Open Source for GeoSpatial North America Conference in Raleigh, NC this
May .

My colleague Rob Emanuele (GeoTrellis ) and myself
are very proud to be involved in the organization of this conference. Rob
will be serving as Program Chair again this year and I will be serving as
the Committee Chair. You can meet the rest of the organizing team here
. If
you’ve never been to a FOSS4G  conference,
they are a fun and informative gathering of the smart and dedicated people
that make up the FOSS4G community.  Everyone is welcome!

This is truly a hands-on, collaborative conference and we want to invite
you to share your knowledge and talent by submitting a proposal for the
conference ! We’d like to extend a special
invitation to traditionally underrepresented groups in open source
geospatial to submit a proposal. We are looking to build a great program
that reflects all of the brilliant people and work that make up our
community, but that can only happen if you submit your proposals! The
success of the conference is directly dependent on the quality and
diversity of the submissions, so please help us make FOSS4G NA 2016 the
best possible conference it can be.

In the Call For Proposals , we have laid
out our topics of interest, but if you have something great to talk about
that doesn’t fit to that mold, please still submit!

Conference presenters receive free registration and we also have a scholarship
program .

The deadline for proposals is February 8th.  Hope to see you in Raleigh!
Sarah
-- 
Sarah Cordivano
Project Manager, Data Analytics

Azavea |  340 N 12th St, Ste 402, Philadelphia, PA
scordiv...@azavea.com
Web analytics.azavea.com  |  Blog azavea.co 
m/bl ogs   |
Twitter @azavea 


*-*



*We'll be moving on March 1, 2016. Please update your records with our new
address:990 Spring Garden Street, 5th FloorPhiladelphia, PA 19123*
___
postgis-users mailing list
postgis-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/postgis-users