Hi Pascal, This cries out for a cascaded union, I think. Maybe we should have a 1.0.12 release. Still, the leaks trouble me. Statements like
P = P.union(p) have caused me problems in the past. Probably a matter of held references? Any interest in digging into it? Sean On Apr 3, 2009, at 3:02 AM, Pascal Leroux wrote: > Even with very simple geometries : > > >>> P = Polygon(((0,0),(0,9),(9,9),(9,0))) > >>> p = Polygon(((1,1),(1,5),(5,5),(5,1))) > >>> count = 1 > >>> while 1: > ... if count % 10000 == 0: print count > ... P = P.union(p) > ... count += 1 > > > after 25 minutes : > > > 7020000 > 7030000 > Python(4969) malloc: *** mmap(size=2097152) failed (error code=12) > *** error: can't allocate region > *** set a breakpoint in malloc_error_break to debug > ... > Python(4969) malloc: *** mmap(size=2097152) failed (error code=12) > *** error: can't allocate region > *** set a breakpoint in malloc_error_break to debug > MemoryError > Python(4969) malloc: *** mmap(size=2097152) failed (error code=12) > *** error: can't allocate region > *** set a breakpoint in malloc_error_break to debug > Traceback (most recent call last): > File "<stdin>", line 3, in <module> > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ > python2.5/site-packages/Shapely-1.0.11-py2.5.egg/shapely/ > topology.py", line 34, in __call__ > if not self.context.is_valid: > MemoryError > >>> > > > This code doesn't make sense but it highlights the issue I talked > about last week. > I used a very big polygon in my last email but, here, I cannot use > simpler ones. > > My final purpose is to build a full topology with several shapefiles > located on a 1°x1° area and I often work with more than 200,000 > features at a time. Their geometries, even if they are not as > complicated as the polygon I used last week, are not as simple as > those I used above. > > 7,000,000 calls to union with the two simple polygons crashed the > script because of a lack of memory. That corresponds (in the better > case), with 200,000 features, to an average of 35 calls per initial > feature. Obviously, it's not enough to build a full topology. > > I'm trying to split my area into several parts, to build topology > for each part and then merge the parts. It will be more complicated. > Will it be enough ? Will I have to come back to C/libgeos ? aaargh ! > > Pascal > > > > _______________________________________________ > Community mailing list > [email protected] > http://lists.gispython.org/mailman/listinfo/community _______________________________________________ Community mailing list [email protected] http://lists.gispython.org/mailman/listinfo/community
