Hi, I made a simple test with ogr2ogr and geopackage to geopackage on Windows. With GDAL 3.8.0dev-3e4dc710a2 (no arrow, old R-Tree) the timing was 36 minutes, with GDAL 3.8.0dev-6bbd2c080a the same conversion took 21 minutes. The gpkg file is 75 GB, with 120 layers.
Could it be that the new R-Tree implementation does not print anything into debug? With older GDAL I see GPKG: Creating insert/delete feature_count triggers GPKG: 194505 rows inserted into rtree_virtausnuoli_geom but with the brand new just GPKG: Creating insert/delete feature_count triggers GPKG: Creating insert/delete feature_count triggers -Jukka Rahkonen- -----Alkuperäinen viesti----- Lähettäjä: gdal-dev <gdal-dev-boun...@lists.osgeo.org> Puolesta Even Rouault via gdal-dev Lähetetty: tiistai 31. lokakuuta 2023 16.33 Vastaanottaja: gdal-dev@lists.osgeo.org Aihe: [gdal-dev] GDAL 3.8.0beta1 available for testing Hi, I've prepared a beta1 of GDAL 3.8.0 to get feedback from earlier testers. Sorry no updated NEWS.md file yet, but I'd in particular be interesting by testing of ogr2ogr workflows, since they have underwent significant changes in the underlying implementation: - when the source layer is a layer implementing the ArrowStream API (that is GeoPackage, FlatGeoBuf, Arrow or Parquet), and when no ogr2ogr options than -of, -where, -spat, -lco, -dsco, -gt, -append/-overwrite/-update are used (and -sql as well for GeoPackage). When enabling CPL_DEBUG, you'll see a "OGR2OGR: Using WriteArrowBatch()" trace when that new code path is taken. If specifying other options, the feature-iteration-based traditional implementation is used - and/or when the output layer is GeoPackage (new layer), due to the revamped much faster spatial index creation. This enhanced spatial index creation is not ogr2ogr specific and is actually available more generally for CreateLayer() + CreateFeature() or CreateLayer() + WriteArrowBatch() scenarios. Point of attention would be when in situations with large files and/or with low RAM. The ghcr.io/osgeo/gdal:ubuntu-small-latest, ghcr.io/osgeo/gdal:ubuntu-full-latest, ghcr.io/osgeo/gdal:alpine-normal-latest Docker images have been refreshed with 3.8.0beta1 (ghcr.io/osgeo/gdal:alpine-small-latest still building at time of writing). (Note: the GDAL master conda builds mentioned at https://gdal.org/download.html#gdal-master-conda-builds have been broken for a couple weeks and are thus not usable to test beta1 currently. I'm investigating) Source snapshots at: - https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0beta1.tar.gz - https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0beta1.tar.xz - https://download.osgeo.org/gdal/3.8.0/gdal380beta1.zip Autotest snapshots: - https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0beta1.tar.gz - https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0beta1.zip Even -- http://www.spatialys.com/ My software is free, but my time generally not. _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
