Just killed the bulk_upload.py job, dropped database and recreated it. Used sed to fix the negative numbers.
osmosis took 427263 milliseconds. Yes. I did update the ID sequences in postgres. Things are much happier without all that negativism. It's still very slow in Potlatch. At least part of the problem is the insane complexity of the features (yes, that straight road segment needs 82 nodes!) -Eric -=--=---=----=----=---=--=-=--=---=----=---=--=-=- Eric B. Wolf 720-334-7734 On Tue, Aug 3, 2010 at 2:55 PM, Ian Dees <[email protected]> wrote: > I imagine the bottleneck is the Railsport doing precondition checks for > everything as it's going in. > > I don't think I could give an educated guess for time remaining, but on the > api.osm.org server it usually takes 4+ hours to send in a 50k-change diff > file (around 25MB?). Based on that I'd say you have at least half a day of > waiting left. > > On Tue, Aug 3, 2010 at 3:46 PM, Eric Wolf <[email protected]> wrote: > >> Just how slow is bulk_upload.py? >> >> I am loading a 177MB .osm file into an empty database on a quad 3.6Ghz >> Xeon with 6GB RAM and 700GB of RAID5. The machine is basically idle except >> for this load. >> >> It's already taken almost an hour. >> >> -Eric >> >> -=--=---=----=----=---=--=-=--=---=----=---=--=-=- >> Eric B. Wolf 720-334-7734 >> >> >> >> >> >> On Tue, Aug 3, 2010 at 12:48 PM, andrzej zaborowski <[email protected]>wrote: >> >>> On 3 August 2010 20:28, Eric Wolf <[email protected]> wrote: >>> > This is in reference to the USGS OSMCP project - not the real OSM... >>> > When we imported our chunk of data initially (not me - the guy >>> responsible >>> > is on walkabout in the Rockies), we followed the convention of using >>> > negative IDs in the .OSM file. But osmosis was used to load the data >>> into >>> > the database and now all of our data has negative IDs. This seems to >>> have a >>> > really nasty effect on the API - every time something is edited, a new >>> copy >>> > is created with positive IDs and the old version with the negative IDs >>> > persists. >>> > I assume there is something in the API that says "negative IDs == BAD". >>> I've >>> > been trying to test that theory but keep hitting stumbling blocks. >>> Postgres >>> > doesn't seem to want to let me defer integrity constraints, so my >>> efforts to >>> > change a few IDs to positive values keeps failing. Maybe I've lost my >>> SQL >>> > chops (or maybe I just can't do that as the "openstreetmap" database >>> user). >>> > Am I barking up the right tree? Should I just go ahead and destroy the >>> > database and repopulate it using bulk_upload.py instead of osmosis? >>> >>> If there's no way disable the postgres contraints (I'm sure there is.. >>> but I'm a sql noob), I'd filter your .osm file through sed removing >>> the '-' in 'ref="-' and 'id="-' and reimport with osmosis, or modify >>> your conversion script. Using bulk_upload.py and the API will take >>> ages. >>> >>> Cheers >>> >> >> >> _______________________________________________ >> dev mailing list >> [email protected] >> http://lists.openstreetmap.org/listinfo/dev >> >> >
_______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
