I overlooked the fact that it still moves through Python, is that the 'only' hurdle preventing parallel IO? Since gdalwarp for example has the -multi flag, it seems as if GDAL is capable of it, or is that a specific/specialized implementation?
Numba has several options which might eliminate using Python during execution. There are c-callbacks: http://numba.pydata.org/numba-doc/dev/user/cfunc.html It probably works by using the Numpy C-API (which i have zero experience with). I don't know if its possible that other programs, like GDAL, can also use those compiled functions without moving to Python first. There is also ahead-of-time compilation (AOT): http://numba.pydata.org/numba-doc/dev/user/pycc.html AOT has the benefit that users only need Numpy as a dependency, and don't need Numba/llvm. There some drawbacks as well, like no longer being able to compile optimizations for the hardware its running on. Regards, Rutger Even Rouault-2 wrote > Le mardi 13 septembre 2016 09:02:09, Rutger a écrit : > > I've only scratched up the surface of Numba (didn't know it 2 days ago). I > guess this might work (might only be interested if the > VRTDerivedRasterBand::IRasterIO() is called with a big enough region. > Which > depends on the pixel access pattern of upper layers). The VRT drivers just > calls a Python function that takes numpy arrays and a few extra args. > >> And would that give parallel >> processing for both IO and calculations? > > Only calculations. I/O is done before going to Python and after returning > from > it. > > Actually... if you specify zero source in the VRT, then it is up to you to > do > the read operations the way you like in Python, so you could possibly > parallelize them there. > > > Even > > -- > Spatialys - Geospatial professional services > http://www.spatialys.com > _______________________________________________ > gdal-dev mailing list > gdal-dev@.osgeo > http://lists.osgeo.org/mailman/listinfo/gdal-dev -- View this message in context: http://osgeo-org.1560.x6.nabble.com/gdal-dev-VRT-derived-band-pixel-functions-written-in-Python-tp5285323p5285482.html Sent from the GDAL - Dev mailing list archive at Nabble.com. _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev