Le mardi 13 septembre 2016 21:22:20, James Ramm a écrit : > I think you can call SWIG with the -threads argument on the command line so > it will always release the GIL. Could be an easy option if it works
That's mostly what I've done. See my other message : https://lists.osgeo.org/pipermail/gdal-dev/2016-September/045155.html > > On Tuesday, 13 September 2016, Even Rouault <even.roua...@spatialys.com> > > wrote: > > Le mardi 13 septembre 2016 11:07:39, Rutger a écrit : > > > I overlooked the fact that it still moves through Python, is that the > > > 'only' hurdle preventing parallel IO? > > > > Not sure to understand your question. But if you have several sources, > > you could potentially do parallelized reading of them from the Python > > code by using Python threads and GDAL Python API. But looking in the > > SWIG generated code, it doesn't seem that SWIG releases the GIL > > automatically before calling > > native code. Hum... So that should probably added manually, at least > > around GDALRasterIO() calls, otherwise you'll get zero perf > > improvements. > > > > > Since gdalwarp for example has the > > > -multi flag, it seems as if GDAL is capable of it, or is that a > > > specific/specialized implementation? > > > > Parallelized I/O doesn't mean much by itself without more context. You > > may want to parallelize reading of different regions of the same > > dataset, or parallelize reading of different datasets. Due to GDAL > > objects not being thread-safe, the first case (reading of different > > regions of the same dataset) > > can be solved with the second one by opening several datasets for the > > same filename. > > > > Regarding gdalwarp -multi, here's how that works. When you warp a > > dataset, there's a list of all chunks (windows) to be processed that is > > generated. gdalwarp -multi does the following > > > > Thread I/O Thread > > computation Read data for chunk 1 > > Read data for chunk 2 Do calculations for chunk 1 > > Write output of chunk 1 Do calculations for chunk 2 > > Read data for chunk 3 > > Write output of chunk 2 Do calculations for chunk 3 > > > > > Numba has several options which might eliminate using Python during > > > execution. There are c-callbacks: > > > http://numba.pydata.org/numba-doc/dev/user/cfunc.html > > > > You can also use @jit(nopython=True, nogil=True) and your Python method > > will > > end up being pure native code (provided that you don't use too high level > > stuff > > otherwise the jit'ification will fail with an exception). > > > > And for code that is not inlined in the VRT, you can also add cache=True > > so that the jit'ification can be reused. > > > > With all that the cost of the Python layer becomes neglectable (except > > loading > > the Python environment the first time, if not already loaded, but for a > > computation that will be longer than a few seconds, that's not really a > > big deal) > > > > -- > > Spatialys - Geospatial professional services > > http://www.spatialys.com > > _______________________________________________ > > gdal-dev mailing list > > gdal-dev@lists.osgeo.org <javascript:;> > > http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Spatialys - Geospatial professional services http://www.spatialys.com _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev