From: Kai Krueger [mailto:kakrue...@gmail.com]
Subject: Re: [OSM-dev] osm2pgsql slow slim import
Anyway, as Frederik suggested, I have changed osm2pgsql to automatically
set the parameter synchronous_commit to off for the import session.
Doing this should be safe. Unlike turning off fsync
2011/12/1 Petr Morávek [Xificurk] xific...@gmail.com:
Actually, I have just found out that the most important is
synchronous_commit = off
If synchronous_commit is helping so much that means that the code is
doing far too many transactions. Obviously there needs to be some
BEGIN/COMMIT added
Can you try if using the switch --number-processes=1? That should disable
any parallelisation and make it more or less the same as before. (Although
there is a slight difference in transaction handling)
It's x2 faster with --number-processes=4 instead of 1 but still far from older
version
sly (sylvain letuffe) napsal(a):
Can you try if using the switch --number-processes=1? That should disable
any parallelisation and make it more or less the same as before. (Although
there is a slight difference in transaction handling)
It's x2 faster with --number-processes=4 instead of 1
Kai has suggested to me changing configuration of postgresql like this:
fsync = off
synchronous_commit = off
And this worked like a charm, now I'm getting significantly faster
imports compared to old config and old osm2pgsql.
Whaoo !
It just work (and very well)
--
sly
qui suis-je :
2011/12/1 sly (sylvain letuffe) li...@letuffe.org:
Kai has suggested to me changing configuration of postgresql like this:
fsync = off
synchronous_commit = off
Cool thing, but I hope you'll switch it back as soon as you get
everything imported. I had my database broken three times (unable to
Hi,
On 12/01/11 11:13, Petr Morávek [Xificurk] wrote:
Kai has suggested to me changing configuration of postgresql like this:
fsync = off
synchronous_commit = off
And this worked like a charm
Yes. I wonder if there's a way for osm2pgsql to query postgres about its
settings and issue a
Frederik Ramm napsal(a):
Hi,
On 12/01/11 11:13, Petr Morávek [Xificurk] wrote:
Kai has suggested to me changing configuration of postgresql like this:
fsync = off
synchronous_commit = off
And this worked like a charm
Yes. I wonder if there's a way for osm2pgsql to query postgres about
Hi,
On 12/01/11 14:23, Petr Morávek [Xificurk] wrote:
Actually, I have just found out that the most important is
synchronous_commit = off
I think that fsync=off includes synchronous_commit=off. I believe the
latter can also be enabled with a SET statement for the current
transaction only,
Thanks for catching this regression. As my test database was always set to
fsync=off, I didn't notice this performance regression during the
development of the parallelisation work.
The problem is that in order to allow multiple threads to work through the
pending ways and thus potentially speed
quot;Petr Morávek [Xificurk]quot; wrote
1) fsync = off, synchronous_commit = off
1990 s = 33 min
2) fsync = on, synchronous_commit = off
3075 s = 51 min
Interestingly, most of the time difference comes from the initial import
of relations.
Yes, I noticed that as well, that the initial
Petr Morávek [Xificurk] napsal(a):
Hello,
I've just upgraded osm2pgsql to latest SVN version (previously I was
running a version from August, not sure which revision exactly) and I'm
experiencing extreme slow-down in the phase Going over pending ways.
The processing is going at rate ~ 0.1k/s.
Hello,
Hi,
sorry, no solutions ;-( but I ran into the exact same trouble
rev26892: import time ~1min
rev26893: import time ~22min (the phase going over pending ways went
at rate 0.11k/s)
Nice catch.
revision 26892 ran that phase for the same extract at 6.36k/s instead of
0.14k/s
I've
sylvain letuffe wrote
Nice catch.
revision 26892 ran that phase for the same extract at 6.36k/s instead of
0.14k/s
6.4k/s is much more what I would expect from such a small extract. So yes
something is wrong there, but I haven't seen that behavior before and I
can't currently reproduce
Hello,
I've just upgraded osm2pgsql to latest SVN version (previously I was
running a version from August, not sure which revision exactly) and I'm
experiencing extreme slow-down in the phase Going over pending ways.
The processing is going at rate ~ 0.1k/s.
With the old version the import of a
15 matches
Mail list logo
