Re: [gdal-dev] Logging network requests/replies?
Hi all, On Sat, 9 Feb 2019 at 00:59, Even Rouault wrote: > I had several thoughts about this > > 1) yes a dedicated callback mechanism could work of course > > 2) As Sean mentionned, and provided we redirect curl logging to our own > callbacks instead of stderr, we could potentially just use the CPLDebug() > infrastructure with the the callback being installed with > CPLSetErrorHandler() > + CPLSetCurrentErrorHandlerCatchDebug(true). We probably wouldn't want so > much > debugging information to be emitted by default (both because of verbosity > and > performance), so this should still be conditionalized by the > CPL_CURL_VERBOSE=YES config option/env. variable > I'd suggest this is the best approach. Alternatively a TRACE loglevel below DEBUG like some of the Java logging frameworks have. > > 3) An alternative would be to have a dedicated generic logging API > Not sure this adds much over our existing logging API which already includes callbacks, and has Python integration/etc? The only thing it's really missing are logging at different levels via topic/categories, and even those are partially there atm (and you can make it work via a C++/Python logging library integration). Rob :) ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Logging network requests/replies?
Hi Even, On Fri, Feb 8, 2019 at 4:59 AM Even Rouault wrote: > > Actually, what I had in mind (thinking out loud) would sit solely > > within GDAL, e.g. a method to globally specify a callback function to > > be called whenever GDAL creates a network request, e.g. within > > CPLHTTPFetchEx (and similarly for replies). > > CPLHTTPFetchEx() is just one of the site that does network access. > cpl_vsil_curl.cpp, cpl_vsil_s3.cpp, cpl_vsil_az.cpp, cpl_vsil_swift.cpp, > cpl_vsil_webhdfs.cpp also do network access > > But at the end of all them must go through CPLHTTPSetOptions() in > cpl_http.cpp > > > (With clients being > > responsible for ensuring that their callback functions are > > appropriately thread safe). > > > > Does this sound desirable? Any concerns/blockers before I start hacking > on > > it? > > I had several thoughts about this > > 1) yes a dedicated callback mechanism could work of course > > 2) As Sean mentionned, and provided we redirect curl logging to our own > callbacks instead of stderr, we could potentially just use the CPLDebug() > infrastructure with the the callback being installed with > CPLSetErrorHandler() > + CPLSetCurrentErrorHandlerCatchDebug(true). We probably wouldn't want so > much > debugging information to be emitted by default (both because of verbosity > and > performance), so this should still be conditionalized by the > CPL_CURL_VERBOSE=YES config option/env. variable > > 3) An alternative would be to have a dedicated generic logging API > > void CPLLogEvent(const char* topic, const char* fmt, ...); > > int CPLHasLoggerFor(const char* topic); > > typedef void (*CPLLoggerCallback)(const char* topic, > >const char* msg, void* user_data); > > void* CPLInstallLogger(const char* topic, > CPLLoggerCallback > cbk, void* user_data); > void CPLUninstallLogger(void*); > > but I'm wondering if there wouldn't be too much confusion/intersection > with > the CPLDebug() mechanism. > > > My current slight preference would go to 2) to reuse existing > infrastructure. > Maybe others, including Robert CC'ed who has hacked in the error/debug > area > recently, have thoughts on this. > > > Even > I'd prefer to see the existing infrastructure reused because then some logging of network activity would automatically appear for users of Rasterio (which already subscribes to CPLDebug messages). I think the right amount of default logging would be to call CPLDebug for new connections and report the URL, but not necessarily call CPLDebug for every request made using that connection. -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Logging network requests/replies?
> Actually, what I had in mind (thinking out loud) would sit solely > within GDAL, e.g. a method to globally specify a callback function to > be called whenever GDAL creates a network request, e.g. within > CPLHTTPFetchEx (and similarly for replies). CPLHTTPFetchEx() is just one of the site that does network access. cpl_vsil_curl.cpp, cpl_vsil_s3.cpp, cpl_vsil_az.cpp, cpl_vsil_swift.cpp, cpl_vsil_webhdfs.cpp also do network access But at the end of all them must go through CPLHTTPSetOptions() in cpl_http.cpp > (With clients being > responsible for ensuring that their callback functions are > appropriately thread safe). > > Does this sound desirable? Any concerns/blockers before I start hacking on > it? I had several thoughts about this 1) yes a dedicated callback mechanism could work of course 2) As Sean mentionned, and provided we redirect curl logging to our own callbacks instead of stderr, we could potentially just use the CPLDebug() infrastructure with the the callback being installed with CPLSetErrorHandler() + CPLSetCurrentErrorHandlerCatchDebug(true). We probably wouldn't want so much debugging information to be emitted by default (both because of verbosity and performance), so this should still be conditionalized by the CPL_CURL_VERBOSE=YES config option/env. variable 3) An alternative would be to have a dedicated generic logging API void CPLLogEvent(const char* topic, const char* fmt, ...); int CPLHasLoggerFor(const char* topic); typedef void (*CPLLoggerCallback)(const char* topic, const char* msg, void* user_data); void* CPLInstallLogger(const char* topic, CPLLoggerCallback cbk, void* user_data); void CPLUninstallLogger(void*); but I'm wondering if there wouldn't be too much confusion/intersection with the CPLDebug() mechanism. My current slight preference would go to 2) to reuse existing infrastructure. Maybe others, including Robert CC'ed who has hacked in the error/debug area recently, have thoughts on this. Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Logging network requests/replies?
On Thu, 7 Feb 2019 at 19:12, Even Rouault wrote: > > On mercredi 6 février 2019 19:54:29 CET Sean Gillies wrote: > > Hi Nyall, > > > > Even will have the final word on this, but as far as I know, the > > information printed when you turn on the following option > > > > https://trac.osgeo.org/gdal/wiki/ConfigOptions#CPL_CURL_VERBOSE > > > > is not capturable using the error handler from the GDAL API > > Yes CPL_CURL_VERBOSE=YES output goes to stderr by default, since this is curl > which handles this. > But I see in > https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html > that we could possibly specify a callback instead of stderr so that it goes > through GDAL logging mechanism. > Actually, what I had in mind (thinking out loud) would sit solely within GDAL, e.g. a method to globally specify a callback function to be called whenever GDAL creates a network request, e.g. within CPLHTTPFetchEx (and similarly for replies). (With clients being responsible for ensuring that their callback functions are appropriately thread safe). Does this sound desirable? Any concerns/blockers before I start hacking on it? Nyall > Even > > -- > Spatialys - Geospatial professional services > http://www.spatialys.com > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Logging network requests/replies?
On mercredi 6 février 2019 19:54:29 CET Sean Gillies wrote: > Hi Nyall, > > Even will have the final word on this, but as far as I know, the > information printed when you turn on the following option > > https://trac.osgeo.org/gdal/wiki/ConfigOptions#CPL_CURL_VERBOSE > > is not capturable using the error handler from the GDAL API Yes CPL_CURL_VERBOSE=YES output goes to stderr by default, since this is curl which handles this. But I see in https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html that we could possibly specify a callback instead of stderr so that it goes through GDAL logging mechanism. Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Logging network requests/replies?
Hi Nyall, Even will have the final word on this, but as far as I know, the information printed when you turn on the following option https://trac.osgeo.org/gdal/wiki/ConfigOptions#CPL_CURL_VERBOSE is not capturable using the error handler from the GDAL API (see https://www.gdal.org/cpl__error_8h.html#a74d0e649d58180e621540bf73b58e4a2). On Wed, Feb 6, 2019 at 7:14 PM Nyall Dawson wrote: > Hi list, > > I'm guessing this is a long-shot, but is there any API within GDAL > which allows clients to be notified about outgoing network requests > created by the different GDAL drivers? > > I'm looking to get some useful information regarding these for > end-user debugging purposes... > > Nyall > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
