On Thu, Nov 29, 2018 at 3:43 PM Moritz Lennert <[email protected]> wrote: > > Hi, > > I have large amount of vector maps (~ 400) which each has hundreds of > thousands of areas and over 150 attribute columns. I'm looking for the > most efficient way to export these to a format I can share with > non-GRASS GIS users. My favourite would probably be Geopackage, amongst > others because the attribute column names are sometimes fairly long, > thus making export to a DBF based format not easily feasible. > > I've seen several sources of information about possible performance > increases, either in the SQLite world (as a Geopackage is based on > SQLite) or in the OGR SQLite driver page. > > For OGR [1], there is the -gt option to ogr2ogr which allows increasing > the number of features per transaction (or even running the entire > conversion in one single transaction). There is also mention of the > OGR_SQLITE_CACHE and OGR_SQLITE_SYNCHRONOUS configuration options. > However, I don't seem to understand if, and if yes, how I can set any of > these when using v.out.ogr. In r73117, MarkusM added config and dataset > open options to r.in.gdal and v.in.ogr. Would we need something like > this to be able to set the above config options ? How about the > transaction option ?
For GDAL/OGR, you can set configuration options in various ways [0], e.g. OGR_SQLITE_CACHE=1024 OGR_SQLITE_SYNCHRONOUS=OFF v.out.ogr ... I'm afraid the transaction option (equivalent to ogr2ogr -gt) is out of control of v.out.ogr because it is format-specific and v.out.ogr does not use the output format drivers directly, instead it uses the generic OGR interface. However, we could try GDALDatasetStartTransaction()/GDALDatasetCommitTransaction() in v.out.ogr. Markus M [0] https://trac.osgeo.org/gdal/wiki/ConfigOptions > > On the SQLite side, there are some pragma statements that allow > increasing performance (generally at the cost of less data security, but > in some cases this might be acceptable), such as journal_mode [2] or > synchronous [3] (which I suppose is what OGR_SQLITE_SYNCHRONOUS sets as > well), or possibly locking_mode (I don't know how this is handled > internally in OGR). > > Again, my question would be how these could be set when exporting data > with v.out.ogr. > > Other interesting options (which would also influence performance) > might be a flag in v.out.ogr allowing exporting without attribute data > (although a simple v.db.connect -d before export would probably be > enough) and allowing selecting a subset of columns to export. But that's > something else. > > Moritz > > > > > [1] https://gdal.org/drv_sqlite.html > [2] https://sqlite.org/pragma.html#pragma_journal_mode > [3] https://sqlite.org/pragma.html#pragma_synchronous > _______________________________________________ > grass-dev mailing list > [email protected] > https://lists.osgeo.org/mailman/listinfo/grass-dev
_______________________________________________ grass-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/grass-dev
