[OSM-dev] How to clean up Negative IDs

2010-08-03 Thread Eric Wolf
This is in reference to the USGS OSMCP project - not the real OSM... When we imported our chunk of data initially (not me - the guy responsible is on walkabout in the Rockies), we followed the convention of using negative IDs in the .OSM file. But osmosis was used to load the data into the

Re: [OSM-dev] How to clean up Negative IDs

2010-08-03 Thread SteveC
On Aug 3, 2010, at 12:28 PM, Eric Wolf wrote: This is in reference to the USGS OSMCP project - not the real OSM... When we imported our chunk of data initially (not me - the guy responsible is on walkabout in the Rockies), we followed the convention of using negative IDs in the .OSM

Re: [OSM-dev] How to clean up Negative IDs

2010-08-03 Thread Matt Amos
On Tue, Aug 3, 2010 at 7:28 PM, Eric Wolf ebw...@gmail.com wrote: I assume there is something in the API that says negative IDs == BAD. I've been trying to test that theory but keep hitting stumbling blocks. Postgres doesn't seem to want to let me defer integrity constraints, so my efforts to

Re: [OSM-dev] How to clean up Negative IDs

2010-08-03 Thread andrzej zaborowski
On 3 August 2010 20:28, Eric Wolf ebw...@gmail.com wrote: This is in reference to the USGS OSMCP project - not the real OSM... When we imported our chunk of data initially (not me - the guy responsible is on walkabout in the Rockies), we followed the convention of using negative IDs in the

Re: [OSM-dev] How to clean up Negative IDs

2010-08-03 Thread Eric Wolf
Just how slow is bulk_upload.py? I am loading a 177MB .osm file into an empty database on a quad 3.6Ghz Xeon with 6GB RAM and 700GB of RAID5. The machine is basically idle except for this load. It's already taken almost an hour. -Eric -=--=---===---=--=-=--=---==---=--=-=- Eric B.

Re: [OSM-dev] How to clean up Negative IDs

2010-08-03 Thread Ian Dees
I imagine the bottleneck is the Railsport doing precondition checks for everything as it's going in. I don't think I could give an educated guess for time remaining, but on the api.osm.org server it usually takes 4+ hours to send in a 50k-change diff file (around 25MB?). Based on that I'd say you

Re: [OSM-dev] How to clean up Negative IDs

2010-08-03 Thread Eric Wolf
Just killed the bulk_upload.py job, dropped database and recreated it. Used sed to fix the negative numbers. osmosis took 427263 milliseconds. Yes. I did update the ID sequences in postgres. Things are much happier without all that negativism. It's still very slow in Potlatch. At least part of

Re: [OSM-dev] How to clean up Negative IDs

2010-08-03 Thread Brett Henderson
On that note, I think there's a lot of scope for improving import speed in Osmosis. Currently it does it all with multi-line SQL inserts. PostgreSQL JDBC drivers now have COPY support and I have tried it out in the --fast-write-pgsql task. It works well and is much faster. There's a bit of