On Mon, Feb 9, 2015 at 4:52 PM, Fábio Dias fabio.d...@gmail.com wrote:
I switched to postgis for data storage and the v.generalize time went
down to 130ish minutes, all processes working in parallel.
I'm happy now :) thanks you guys very much.
Thanks for reporting this back. What about a
I switched to postgis for data storage and the v.generalize time went
down to 130ish minutes, all processes working in parallel.
I'm happy now :) thanks you guys very much.
-=--=-=-
Fábio Augusto Salve Dias
ICMC - USP
http://sites.google.com/site/fabiodias/
On Tue, Jan 27, 2015 at 8:50 PM,
On Mon, Jan 26, 2015 at 3:54 PM, Fábio Dias fabio.d...@gmail.com wrote:
Hi,
The machine has 128Gb of ram. Doesn't matter what I do, I can't make a
dent on it. Even with everything cached in ram (shp files, database,
the whole lot), there is still free memory.
OK, it's not RAM.
I'm asking
Hi,
I've kept an iotop, cumulative, running while the generalization run.
No disk IO involved, just a couple of postgre stats. I believe the OS
is keeping everything in RAM cache. I don't believe the disk is a
bottleneck either, it is a 10 disk raid of 15k rpm disks, it's really
fast.
I
Hi,
Running trunk. I was loading the shps into postgis then importing them
to grass. Now I'm importing the shps directly.
The machine has 128Gb of ram. Doesn't matter what I do, I can't make a
dent on it. Even with everything cached in ram (shp files, database,
the whole lot), there is still
On Sun, Jan 25, 2015 at 6:11 PM, Fábio Dias fabio.d...@gmail.com wrote:
Hi,
Running r64249, with a couple of stuff in parallel using . It seems
to be considerably slower.
Very strange. Are you using trunk or GRASS 7.0?
More than 100h, no 1% printed. To be fair,
I'm not entirely sure I'll
On Mon, Jan 26, 2015 at 9:30 AM, Markus Metz
markus.metz.gisw...@gmail.com wrote:
On Sun, Jan 25, 2015 at 6:11 PM, Fábio Dias fabio.d...@gmail.com wrote:
Hi,
Running r64249, with a couple of stuff in parallel using . It seems
to be considerably slower.
Very strange. Are you using trunk or
Hi,
Running r64249, with a couple of stuff in parallel using . It seems
to be considerably slower. More than 100h, no 1% printed. To be fair,
I'm not entirely sure I'll see it when it prints, 10 v.generalize
running (5 for each year) + 1 v.in.ogr for 2012. That v.in.ogr is
running for almost 100h
On Sun, Jan 11, 2015 at 11:32 PM, Markus Metz
markus.metz.gisw...@gmail.com wrote:
On Sat, Jan 10, 2015 at 7:23 PM, Fábio Dias fabio.d...@gmail.com wrote:
I have optimized the GRASS vector library in trunk r64032 and added
another topology check to v.generalize in trunk r64033. The profile of
Hello,
Hopefully my last question regarding v.generalize and speeding up the process.
Context:
I have multiple years of data that need to be generalized. For each
year, I need a number of different generalizations (specific number
TBD).
Question:
What would be the best way to do that in
On Wed, Jan 14, 2015 at 3:54 PM, Fábio Dias fabio.d...@gmail.com wrote:
...
What would be the best way to do that in parallel? One mapset for each
year? Can I run multiple v.generalizes on the same input with
different outputs?
Yes sure.
My first thought was to run completely separated grass
On Sat, Jan 10, 2015 at 7:23 PM, Fábio Dias fabio.d...@gmail.com wrote:
I have optimized the GRASS vector library in trunk r64032 and added
another topology check to v.generalize in trunk r64033. The profile of
v.generalize now shows that it is limited by disk I/O speed (on my
laptop with a
I have optimized the GRASS vector library in trunk r64032 and added
another topology check to v.generalize in trunk r64033. The profile of
v.generalize now shows that it is limited by disk I/O speed (on my
laptop with a standard laptop-like spinning HDD), which means that the
algorithms are,
On Sun, Jan 4, 2015 at 10:45 PM, Fábio Dias fabio.d...@gmail.com wrote:
As promised, profile of v.generalize, as of r63952.
(The data might not be exactly the same, I might have run v.clean somewhere).
Thanks for your thorough code analysis!
My initial guess was wrong,
Another info update (and shameless bump)
Around 18M primitives and 130M vertices.
And I'm left wondering... Has nobody tried to generalize this amount
of data yet? Or I'm going about this on the wrong way?
I even mounted the grassdata dir as a ramdisk to try to speed up the
process, but it is
Another interesting update
I believed that doing a dissolve before generalizing would speed up
the process, because it would remove a lot of edges. The data is very
segmented, stitching would be the right term, I suppose.
Turns out, that belief is really wrong. The really expensive part of
Original:
GRASS 7.1.svn (brasil):~ v.info map=tc10
++
| Name:tc10 |
| Mapset: terraclass|
|
Just for further reference, the v.dissolve takes around 24h in this
dataset. I'll post the v.info of both as soon as it is finished.
Any other ideas? I have a fairly powerful server at my disposal, but
I'm out of ideas...
-=--=-=-
Fábio Augusto Salve Dias
http://sites.google.com/site/fabiodias/
On Wed, Dec 31, 2014 at 5:20 PM, Fábio Dias fabio.d...@gmail.com wrote:
I fussed about the v.generalize code, thinking about pthread
parallelization. The geometry part of the code is *really* fast and
can be easily parallelized so it can run even faster. But, according
to the profile
Attached is pdf generated with google-perf of v.generalize, using
g7b4. I'm running it again for trunk.
-=--=-=-
Fábio Augusto Salve Dias
http://sites.google.com/site/fabiodias/
On Sun, Jan 4, 2015 at 5:54 PM, Markus Metz
markus.metz.gisw...@gmail.com wrote:
On Wed, Dec 31, 2014 at 5:20 PM,
As promised, profile of v.generalize, as of r63952.
(The data might not be exactly the same, I might have run v.clean somewhere).
I still have the raw profiles, if anyone wants them.
F
-=--=-=-
Fábio Augusto Salve Dias
http://sites.google.com/site/fabiodias/
On Sun, Jan 4, 2015 at 6:01 PM,
On Wed, Dec 31, 2014 at 5:20 PM, Fábio Dias fabio.d...@gmail.com wrote:
On Wed, Dec 31, 2014 at 12:23 PM, Markus Neteler nete...@osgeo.org wrote:
On Sun, Dec 28, 2014 at 8:04 PM, Fábio Dias fabio.d...@gmail.com wrote:
Hello all,
Context: I've loaded some shp files into postgis, containing
On Wed, Dec 31, 2014 at 12:23 PM, Markus Neteler nete...@osgeo.org wrote:
On Sun, Dec 28, 2014 at 8:04 PM, Fábio Dias fabio.d...@gmail.com wrote:
Hello all,
Context: I've loaded some shp files into postgis, containing
information over the amazon forest. For reference, the sql script has
23 matches
Mail list logo
