Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-05-16 Thread Erik Johansson
On Fri, May 15, 2009 at 10:58 AM, Tomas Kolda ko...@web2net.cz wrote: Did you try my version of import? I have no responses, so maybe not :). I did not tried complete planet, but indexes was created quite fast. 30seconds on 150MB osm file. I think that you should make a chart (xml size-time

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-05-15 Thread Tomas Kolda
Did you try my version of import? I have no responses, so maybe not :). I did not tried complete planet, but indexes was created quite fast. 30seconds on 150MB osm file. I think that you should make a chart (xml size-time to create index) and than you can see complexity of creating indexes in

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-29 Thread Tomas Kolda
Hi, I will post something, but now I'm not at home. So maybe today night. But Dane wrote me about sqlite support in mapnik, so you can try that: Are you aware we have a postgres - sqlite converter in mapnik?

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-29 Thread marcus.wolschon
On Tue, 28 Apr 2009 14:30:33 +, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: From what I can see at http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/Embedded-Server.html#Embedded-Server the embedded-mode is still a server, just for embedded systems. I was talking about having

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-29 Thread Ævar Arnfjörð Bjarmason
On Wed, Apr 29, 2009 at 8:41 AM, marcus.wolsc...@googlemail.com wrote: On Tue, 28 Apr 2009 14:30:33 +, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: From what I can see at http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/Embedded-Server.html#Embedded-Server the embedded-mode

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-29 Thread marcus.wolschon
On Wed, 29 Apr 2009 10:18:55 +, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: You'd build postrgresql with the compilation flags you'd use for your program: Problem solved. My program is not compiled. It's java. Any embedded database like sqlite would have to talk to libc too, and so

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-28 Thread Shaun McDonald
On 28 Apr 2009, at 00:16, Kelly Jones wrote: I've seen many posts saying that SQLite2 can't handle OpenStreetMap's large planet.osm data file: http://planet.openstreetmap.org/planet-090421.osm.bz2 which is 5.4G bzip2 compressed, about 150G uncompressed. Can SQLite3 handle this? Has

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-28 Thread marcus.wolschon
On Tue, 28 Apr 2009 08:02:30 +0100, Shaun McDonald SQLite isn't designed for huge databases like OpenStreetMap. You could get away with a city or small region, but more than that, you will get the slowness that you are seeing. http://www.sqlite.org/faq.html#q19 and some other FAQs on that

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-28 Thread Stefan de Konink
marcus.wolsc...@googlemail.com wrote: On Tue, 28 Apr 2009 08:02:30 +0100, Shaun McDonald SQLite isn't designed for huge databases like OpenStreetMap. You could get away with a city or small region, but more than that, you will get the slowness that you are seeing.

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-28 Thread Tomas Kolda
There is no reason to be sqlite slow. How much are you are using transactions on your local db? What are you using for import? I'm importing Czech republic (50Mb bziped) in one or two minutes. So for 5.4GB it takes 108 minutes (two hours). But I'm importig to modified osm schema (something

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-28 Thread Matt Amos
On Tue, Apr 28, 2009 at 8:26 AM, Stefan de Konink ste...@konink.de wrote: marcus.wolsc...@googlemail.com wrote: On Tue, 28 Apr 2009 08:02:30 +0100, Shaun McDonald SQLite isn't designed for huge databases like OpenStreetMap. You could get away with a city or small region, but more than that,

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-28 Thread marcus.wolschon
On Tue, 28 Apr 2009 09:26:21 +0200, Stefan de Konink ste...@konink.de wrote: marcus.wolsc...@googlemail.com wrote: On Tue, 28 Apr 2009 08:02:30 +0100, Shaun McDonald SQLite isn't designed for huge databases like OpenStreetMap. You could get away with a city or small region, but more than

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-28 Thread Ævar Arnfjörð Bjarmason
On Tue, Apr 28, 2009 at 12:29 PM, marcus.wolsc...@googlemail.com wrote: On Tue, 28 Apr 2009 09:26:21 +0200, Stefan de Konink ste...@konink.de wrote: marcus.wolsc...@googlemail.com wrote: On Tue, 28 Apr 2009 08:02:30 +0100, Shaun McDonald SQLite isn't designed for huge databases like

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-28 Thread Christoph Eckert
Hi, There is no reason to be sqlite slow. How much are you are using transactions on your local db? What are you using for import? I'm importing Czech republic (50Mb bziped) in one or two minutes. So for 5.4GB it takes 108 minutes (two hours). But I'm importig to modified osm schema

Re: [OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-28 Thread Robert Joop
On 09-04-28 09:14:31 CEST, marcus.wolsc...@googlemail.com wrote: Is there ANY embedded database out there that can handle the planet? Anything that does not require the (unskilled) user of a program to install a local database-server first? has anybody tried embedded innodb? i saw mention of

[OSM-dev] Can SQLite3 handle OSM 150G data file?

2009-04-27 Thread Kelly Jones
I've seen many posts saying that SQLite2 can't handle OpenStreetMap's large planet.osm data file: http://planet.openstreetmap.org/planet-090421.osm.bz2 which is 5.4G bzip2 compressed, about 150G uncompressed. Can SQLite3 handle this? Has anyone tried? I tried to do this myself, but I'm on a