Hi all, Upgrading to Fiona 1.8.0 won't help the code below because every outfile.write() call entails a transaction of size 1. The user needs to pass a sequence or iterator of many records to outfile.writerecords() if they want to benefit from larger transactions.
On Thu, Nov 1, 2018 at 3:46 AM Even Rouault <[email protected]> wrote: > You are perhaps using a too old version of Fiona. Recent versions insert > features within OGR transactions. Should be fine with latest Fiona 1.8.0 > See https://github.com/Toblerity/Fiona/issues/476 > > > Hi, following is the code I used: > > > > Import os, osmium, fiona > > > > fi = 'europe-latest.osm.pbf' > > fo = 'europe-latest.dpkg' > > > > drv = 'DPKG' > > > > crs = {'no_defs': True, 'ellps': 'WGS84', 'datum': 'WGS84', 'proj': > > 'longlat'} > > > > schema = {'geometry': 'LineString', > > 'properties': {'id': 'float', 'name' : 'str', 'kind' : > > 'str'}} > > > > outfile = fiona.open(fo, 'w', driver=drv, crs=crs, schema=schema) > > > > geomfab = osmium.geom.GeoJSONFactory() > > > > class ShapeConverter(osmium.SimpleHandler): > > def way(self, w): > > if 'place' in w.tags: > > rec = {'geometry' : eval(geomfab.create_linestring(w)), > > 'properties' : {'id' : float(w.id), > > 'name' : w.tags.get('name'), > > 'kind' : w.tags['place']}} > > outfile.write(rec) > > > > ShapeConverter().apply_file(fi, locations=True) > > > > On Thu, Nov 1, 2018 at 4:10 PM jratike80 < > > > > [email protected]> wrote: > > > koji higuchi wrote > > > > > > > Hi > > > > I am extracting .osm.pbf file into .gpkg; but the writing rate is > very > > > > slow. > > > > When I write into .shp, rate is faster but its limit is 4gb. > > > > So, what other format is better for larger than 4gb requirement? > > > > Thanks for your ideas. > > > > Koji > > > > > > Hi, > > > > > > Show the exact command that you are using so we can see if it could be > > > made > > > faster. For example if you happen to use -skipfailures for handling the > > > invalid geometries of the OSM data then you will make GDAL to do a new > > > transaction for each row and that certainly is slow. But let's have a > look > > > at your command first. > > > > > > -Jukka Rahkonen- > > > > > > > > > > > > -- > > > Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html > > > _______________________________________________ > > > gdal-dev mailing list > > > [email protected] > > > https://lists.osgeo.org/mailman/listinfo/gdal-dev > > > -- > Spatialys - Geospatial professional services > http://www.spatialys.com > _______________________________________________ > gdal-dev mailing list > [email protected] > https://lists.osgeo.org/mailman/listinfo/gdal-dev -- Sean Gillies
_______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev
