On 22/06/09 07:00, Michael Perdue wrote:
On 19-Jun-09, at 8:54 PM, Hamish wrote:
Michael wrote:
I actually get the same thing.
For the first while the process runs in full use
of one core, but once the process starts writing results to
the db, the whole process bogs down to a
grinding halt. CPU usage for v.lidar.edgedetection
drops down to ~1% (1% of one core) and sqlite usage rises to
~16%. Maybe I'm wrong, but my impression was that the
bottle neck was the modifications to the database.
hmmm. I've seen no problems on Linux64 + DBF backend. 100% core until
completion.
Are you running WinXP + SQlite? Can you try with the DBF driver?
maybe the common problem is in the SQLite driver... ??
I'm running Mac OS X + SQLite with a 64 bit version of grass. I ran a
whole bunch of tests today and it looks like there are at least a couple
of things going on and SQLite is one of the issues. I ran a 500m x500m
tile with three different DB back-ends (SQLite, Postgres and DBF) and
these are the results I obtained;
(SQLite)
time v.lidar.edgedetection input=Cal_QTile output=Cal_QTile_edges
real 38m53.458s
user 1m17.602s
sys 4m6.353s
(Postgres)
time v.lidar.edgedetection input=Cal_QTile output=Cal_QTile_edges
real 6m49.060s
user 0m46.622s
sys 1m20.324s
(DBF)
time v.lidar.edgedetection input=Cal_QTile output=Cal_QTile_edges
real 1m54.065s
user 0m48.530s
sys 1m8.686s
The results with Postgres and SQLite were a real surprise to me.
As Hamish already mentioned, the main bottleneck here is the connection
to the database which is much slower for actual database systems than
for simple file-based DBF. Looking at the code, I see lots of calls to
the Insert(), InsertInterpolation() and UpDate() functions. At each call
you are hit with the overhead of the database connection. I don't know
the code at all, but it might be worth thinking about grouping these
database calls, possibly by putting all the SQL statements into a temp
file and executing that only once at the end. In scripts doing this has
lead to significant speed gains.
Moritz
_______________________________________________
grass-user mailing list
[email protected]
http://lists.osgeo.org/mailman/listinfo/grass-user
