Hi Martin,
I just fiddled with QgsProviderRegistry and noticed, that
"querySublayers" returns a bunch of GPKG-internal-views and -indices,
that are processed in your script, although not containing any clippable
vector data.
Workaround 1: Use osgeo.ogr.Open() to get a list of real
vector-sub-layers and reduce the number of layer-creations,
clip_layer-calls and open files
Workaround 2: Use osge.ogr to do the layer-creation and clip
(sadly beyond my knowledge...)
"""
gpkg_path = "/home/martin/geodata/ZABAGED20251006.gpkg"
# variant with QgsProviderRegistry.instance().querySublayers(...)
gpkg_sub_layer_names = [gpkg_sub_layer_details.name() for
gpkg_sub_layer_details in
QgsProviderRegistry.instance().querySublayers(gpkg_path)]
print(len(gpkg_sub_layer_names))
print(gpkg_sub_layer_names)
# variant with osgeo.ogr.Open(...)
from osgeo import ogr
gpkg_real_sub_layer_names = [gpkg_sub_layer.GetName() for gpkg_sub_layer
in ogr.Open(gpkg_path)]
print(len(gpkg_real_sub_layer_names))
print(gpkg_real_sub_layer_names)
"""
hth
Ludwig
Am 26.12.25 um 12:24 schrieb Martin Landa via QGIS-User:
Dear all,
I have a standalone pyqgis script which processes multiple layers
stored in a single GPKG. The first part of the script loops over
layers, performs clipping by extent and stores the result in a
memory layer. The problem which I am facing is the increasing number
of open files which easily reach the OS limit. The sample script:
"""
import sys
import psutil, os
import gc
from qgis.core import QgsProviderRegistry, QgsVectorLayer,
QgsRectangle, QgsFeatureRequest, QgsWkbTypes, QgsFeature, QgsGeometry
def open_files_count():
return len(process.open_files())
def clip_layer(layer: QgsVectorLayer, extent: QgsRectangle,
layer_name: str) -> QgsVectorLayer:
extent_geom = QgsGeometry.fromRect(extent)
clipped_layer = QgsVectorLayer(
f"{QgsWkbTypes.displayString(layer.wkbType())}?crs={layer.crs().authid()}",
layer_name,
"memory"
)
clipped_layer.dataProvider().addAttributes(layer.fields())
clipped_layer.updateFields()
for feature in
layer.getFeatures(QgsFeatureRequest().setFilterRect(extent)):
geom = feature.geometry()
if geom.intersects(extent_geom):
clipped_feature = QgsFeature()
clipped_feature.setGeometry(geom.intersection(extent_geom))
clipped_feature.setAttributes(feature.attributes())
clipped_layer.dataProvider().addFeature(clipped_feature)
clipped_feature = None
clipped_layer.commitChanges()
return clipped_layer
process = psutil.Process(os.getpid())
gpkg_path = "/home/martin/geodata/ZABAGED20251006.gpkg"
layers = QgsProviderRegistry.instance().querySublayers(gpkg_path)
extent = QgsRectangle(-653547, -1065864, -653225, -1065600)
clipped_layers = []
for sublayer in layers:
name = sublayer.name <http://sublayer.name>()
layer = QgsVectorLayer(sublayer.uri(), 'ogr')
clipped_layer = clip_layer(layer, extent, layer.name
<http://layer.name>())
del layer
print(name, open_files_count())
if clipped_layer.featureCount() == 0:
del clipped_layer
continue
clipped_layers.append(clipped_layer)
"""
When launching the script number of open file is increasing regardless
any attempts to release data sources (del layer, layer = None,
gc.collect(), ...):
ZeleznicniStaniceZastavka 3
ZeleznicniPrejezd_b 4
ZdrojPodzemnichVod 5
...
BazinaMocal 172
ArealZeleznicniStaniceZastavky 173
ArealUceloveZastavby 174
Please how to release data sources correctly to avoid increasing the
number of open files?
Thanks in advance, Martin
--
Martin Landa
https://geomatics.fsv.cvut.cz/en/employees/martin-landa/
https://gismentors.cz/mentors/landa
_______________________________________________
QGIS-User mailing list
[email protected]
List info:https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-user
_______________________________________________
QGIS-User mailing list
[email protected]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
