Even Rouault wrote: 23. heinäkuuta 2012 13:27 > Le lundi 23 juillet 2012 00:09:05, Jukka Rahkonen a écrit : >> Even Rouault <even.rouault <at> mines-paris.org> writes: >> > > > However, select with SQL feels sub-optimal. >> > > >> > > Yes, when you use ogr2ogr with explicit layer names, there are >> > > optimizations. For example, when you only specify the layer 'points', >> > > the OSM driver will not even try to index the nodes into the temporary >> > > database because it is not needed. However, as you noticed, there is >> > > not yet any optimization when a SQL request is specified. >> > >> > Optimization for SQL request added in r24690 > >> I had a try with r24696 today, downloaded from >> http://gisinternals.com/sdk/Download.aspx?file=release-1500-gdal-mapserver. >> zip >> >> Filtered commands give me errors. An example: >> ogr2ogr -f "ESRI Shapefile" test germany.osm.pbf multipolygons -gt 20000 >> -progress --config OGR_SQLITE_SYNCHRONOUS OFF -where "natural='forest'" >> >> ERROR 1: Failed inserting node 420797898: database schema has changed
> I could also reproduce and I suppose there was at the begnning of the sequence > of errors : "ERROR 1: Failed inserting node XXXX: I/O error" Most probably yes but I saw only the last lines of the first thousand which were printed on the screen. > It turned out that the mechanism to transfer the temporary in-memory DB file > to > disk when it is too big had been broken by a previous commit, so it stayed on > RAM and at some point, it couldn't fit in RAM, hence the error. Should be > fixed > now with r24699. > I'm experimenting with removing the internal use of SQLite for the temporary > database and replacing it with something custom. Actually, it won't replace it > completely in all cases, but it could definitely be used in well-behaved cases > where the elements in the .osm/.pbf are listed in increasing id order, which > is the case of the data in geofabrik files for example. The first results seem > to show increased performance. > Note: in your above example, you don't need to specify -gt and --config > OGR_SQLITE_SYNCHRONOUS OFF when the output format is not sqlite/spatialite. > And the internal use of SQLite in the OSM driver already sets the > corresponding parameters to the values that give the best performance. Sure true with extra parametes, I just forgot to remove them when doing a quick verification for seeing if the error occurs similarly for spatialite and shapefile outputs. -Jukka- _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev