Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
2014-09-06 12:50 GMT+02:00 Martin Dobias wonder...@gmail.com: Hi Alessandro Finally I had some time to look at your proposal (your mail and the proposed QEP [1]). I am not that much involved in the QGIS server development, but the Python word in the thread subject always draws my attention :-) Hello Martin, thank you for your comments! Answers follow. On Thu, Aug 28, 2014 at 7:49 PM, Alessandro Pasotti apaso...@gmail.com wrote: The rationale behind server plugins is doublefold: first they could provide additional services without the need to touch the C++ codebase, second they allow for GUI-based configuration since the server plugins are not separated from the desktop plugins (of course the environment and permissions should be carefully configured to allow information sharing from the desktop user to the webserver user ). I understand the first reason (use python instead of c++ to extend the server), but I am not sure I see the second reason (allow GUI-based configuration). Actually I find that mixing of desktop and server plugin functionality is not making things any better than having two separate plugins, one for desktop and one for server. They are meant to be running in two completely different environments, it may be a good idea to treat them in such way. Still I guess having some kind of plugin functionality also in server may be useful. Configuration of QGIS Server from the Desktop is already implemented: you can configure QGIS OWS Server from the the desktop today and everybody finds it useful. Also, I've been really astonished by some of my customers when I knew that they are currently running QGIS Desktop on their (virtualized) server machines in order to manage their QGIS projects and configure the OWS QGIS server. The conclusion is that I'm not sure anymore about the religious separation of server and client machines (I would still keep them separate but many others don't). But I insist that having the possibility to manage a single plugin for both the server and the desktop side is an opportunity, a possibility that will not enforce authors to use it. If you (as a plugin author) will want to split the server and desktop side of a plugin, you will be perfectly allowed to do it: you will just choose a different name for the server and desktop side of your plugin. Another good reason to have the same plugins machinery for the server and the client is that all the services to manage plugins are already in place: plugins website, plugin installer etc., even configuration testing from QGIS desktop would be easier to implement when it's all in one place. At the same time I think in many cases it would be actually better to have the functionality in python plugin (e.g. WPS server based on processing framework) completely separated from QGIS server and run it through an ordinary interface for Python (WSGI). One could even employ some Apache rewriting rules to make the impression that the service is still handled by QGIS server. My proposal is not about any particular service and WPS was just an example, again, this is about the possibility/opportunity to expand QGIS server functionalities through plugins written in Python. It's not me (and probably not you) who knows how plugin authors will exploit this opportunity, maybe they will write 1 million LOC programs, maybe they will write tiny glue scripts to other programs, who knows? What I like of the idea of having server plugins managed through the desktop is that the users will immediately see the available server plugins and their updates and they will be able to install/upgrade them with a click and they will be able to configure them with a nice GUI. This is not something for me, for you or for the majority of the readers of this mailing list: we like vim and editing rc files from the shell. This is something for the average end users and believe me: they are not always system administrators anymore. You mention that this is just an early prototype to see where things can go, so I understand some things are quite hacky / experimental. If the team is happy to proceed with the concept of plugins for QGIS server, here are some things I think need changing compared to the current proposal/implementation: Absolutely agreed. What we need now is a committing step that gives a green light to Python server plugins. With that green light we can start to discuss implementation details. And of course we will need another green light when the details will be fully defined. - we need proper object-oriented approach for the plugins. Like QGIS desktop provides QgisInterface instance for plugins, QGIS server should also provide some kind of QgsServerInterface which would be used by server plugins. Such interface would give the plugin access to all data structures it needs (e.g. parsed project). Yes, I thought about that but it was beyond the scope of this proof-of-concept implementation. I guess that
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
Hi Alessandro Finally I had some time to look at your proposal (your mail and the proposed QEP [1]). I am not that much involved in the QGIS server development, but the Python word in the thread subject always draws my attention :-) On Thu, Aug 28, 2014 at 7:49 PM, Alessandro Pasotti apaso...@gmail.com wrote: The rationale behind server plugins is doublefold: first they could provide additional services without the need to touch the C++ codebase, second they allow for GUI-based configuration since the server plugins are not separated from the desktop plugins (of course the environment and permissions should be carefully configured to allow information sharing from the desktop user to the webserver user ). I understand the first reason (use python instead of c++ to extend the server), but I am not sure I see the second reason (allow GUI-based configuration). Actually I find that mixing of desktop and server plugin functionality is not making things any better than having two separate plugins, one for desktop and one for server. They are meant to be running in two completely different environments, it may be a good idea to treat them in such way. Still I guess having some kind of plugin functionality also in server may be useful. At the same time I think in many cases it would be actually better to have the functionality in python plugin (e.g. WPS server based on processing framework) completely separated from QGIS server and run it through an ordinary interface for Python (WSGI). One could even employ some Apache rewriting rules to make the impression that the service is still handled by QGIS server. You mention that this is just an early prototype to see where things can go, so I understand some things are quite hacky / experimental. If the team is happy to proceed with the concept of plugins for QGIS server, here are some things I think need changing compared to the current proposal/implementation: - we need proper object-oriented approach for the plugins. Like QGIS desktop provides QgisInterface instance for plugins, QGIS server should also provide some kind of QgsServerInterface which would be used by server plugins. Such interface would give the plugin access to all data structures it needs (e.g. parsed project). - in the server there is already QgsOWSServer interface which existing services derive from. Plugins should be also base on that interface (may need some further work) - the output from plugin should be abstracted - in the server there is already QgsRequestHandler interface for such things (may need some further work) - loading of plugins should be done in a very similar way to desktop plugins - similar to classFactory(iface) entry point, there could be serverClassFactory(iface) entry point that returns instance of running server plugin - I don't like the fact that plugins declare their service and methods in metadata.txt. I think plugins should register their QgsOWSServer implementation to server interface when they are loaded. This is more flexible - a plugin may register more services or provide some extra info during the registration Cheers Martin [1] https://github.com/qgis/QGIS-Enhancement-Proposals/pull/3 ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
2014-08-29 18:05 GMT+02:00 René-Luc Dhont rldh...@gmail.com: Le 29/08/2014 16:54, Alessandro Pasotti a écrit : 2014-08-29 16:39 GMT+02:00 René-Luc Dhont rldh...@gmail.com: Hi Alessandro Thanks for this interesting input. I'm happy to test it and to develope some use case. I thought it's a good feature and can help to enhance QGIS-Server easily : WPS, WMTS, Table Join Service, WFS2 or other that OGC does not already know. Regards, René-Luc Hello René-Luc, I'm happy to hear that you are going to test it, please let me know how it goes! Keep in mind that it's just a proof of concept, and yes, we need a RFC or QEP: the current implementation is the simplest, less obtrusive and quickest that I could think about. For example, I would consider adding real signals attached to plugins slots and cached parsed projects passed to plugin instances... I'm currently working on a WPS server plugin that bridges Processing and PyWPS (version 3.2.2 since the latest version 4 is tightly bound to werkzeug), now fighting with parameters but it basically works. Can I help ? I already done some works on it ! Regards Hello, I've got a first highly experimental Processing-PyWPS bridging plugin: https://github.com/elpaso/qgis-wps * I've only tested grass convex hull with a small shapefile * GetCapabilties and other requests work with GET calls ony * it needs Processing patches otherwise it hangs when reading pipes from shell process (see [1]) * Processing needs other patches as well to support concurrent requests (for example the grass temporary script storage should be done in tempfile instead of the current processing folder) I don't know Processing enough to say if there aren't any other big showstoppers for this plugin. Still a lot of work to do :) [1] https://github.com/elpaso/QGIS/commit/afbfb7f12b8f62731d40745035d919532ae5da1f -- Alessandro Pasotti w3: www.itopen.it ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
Another question could be the timeout. Infact. If a processing take too much time to end. It should go in timeout. So not only concurrency need, but also a timeout system to avoid the lock never ending of resources. A. Il 30/08/2014 13:21, Alessandro Pasotti ha scritto: 2014-08-29 18:05 GMT+02:00 René-Luc Dhont rldh...@gmail.com: Le 29/08/2014 16:54, Alessandro Pasotti a écrit : 2014-08-29 16:39 GMT+02:00 René-Luc Dhont rldh...@gmail.com: Hi Alessandro Thanks for this interesting input. I'm happy to test it and to develope some use case. I thought it's a good feature and can help to enhance QGIS-Server easily : WPS, WMTS, Table Join Service, WFS2 or other that OGC does not already know. Regards, René-Luc Hello René-Luc, I'm happy to hear that you are going to test it, please let me know how it goes! Keep in mind that it's just a proof of concept, and yes, we need a RFC or QEP: the current implementation is the simplest, less obtrusive and quickest that I could think about. For example, I would consider adding real signals attached to plugins slots and cached parsed projects passed to plugin instances... I'm currently working on a WPS server plugin that bridges Processing and PyWPS (version 3.2.2 since the latest version 4 is tightly bound to werkzeug), now fighting with parameters but it basically works. Can I help ? I already done some works on it ! Regards Hello, I've got a first highly experimental Processing-PyWPS bridging plugin: https://github.com/elpaso/qgis-wps * I've only tested grass convex hull with a small shapefile * GetCapabilties and other requests work with GET calls ony * it needs Processing patches otherwise it hangs when reading pipes from shell process (see [1]) * Processing needs other patches as well to support concurrent requests (for example the grass temporary script storage should be done in tempfile instead of the current processing folder) I don't know Processing enough to say if there aren't any other big showstoppers for this plugin. Still a lot of work to do :) [1] https://github.com/elpaso/QGIS/commit/afbfb7f12b8f62731d40745035d919532ae5da1f ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
2014-08-30 13:28 GMT+02:00 aperi2007 aperi2...@gmail.com: Another question could be the timeout. Infact. If a processing take too much time to end. It should go in timeout. So not only concurrency need, but also a timeout system to avoid the lock never ending of resources. This is normally done at the webserver level: http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidbusytimeout ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
2014-08-29 10:24 GMT+02:00 Pirmin Kalberer pi...@sourcepole.com: Hi Alessandro, Am Donnerstag, 28. August 2014, 14.49:47 schrieb Alessandro Pasotti: as anticipated to some of you during the last HF, I've started to add Python plugins support for the server side. The rationale behind server plugins is doublefold: first they could provide additional services without the need to touch the C++ codebase, second they allow for GUI-based configuration since the server plugins are not separated from the desktop plugins (of course the environment and permissions should be carefully configured to allow information sharing from the desktop user to the webserver user ). This looks like the first real case for an RFC! I'm quite sceptical adding Python to QGIS server. In my opinion this should be discussed thoroughly first. Hello Pirmin, thanks for the feedback!!! At least this is the start of a discussion :) I knew from previous conversations with some of you that adding python plugins to the server was not at the top of the priority list and I added QEP (RFC) in the original subject because I know this topic must be discussed: I'm not going to do a PR anytime soon, and my implementation is more like an hack to see how far can I go with this idea, but, so far, it seems quite promising. Everybody knows how has been important to have Python plugins on the desktop side, coding in Python is 5 to 10 times faster than building C++ code and moreover the lower barrier to start being productive will bring a lot of new functionalities from a larger pool of developers. Last but not least, python plugins do not need to pass the PR/pull/build/relase cycle that affect the C++ core, which adds a considerable delay to bug fixes and new features. That said, I would really like to hear more from you all about the reasons to be against this idea. -- Alessandro Pasotti w3: www.itopen.it ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
I guess the main goal is to avoid that the adopt of python on server could slow it. Infact , the main difference from a desktop and a server is that the first is serving only one perosn and this mean that the user could wait also for a work for himself. Instead a server is used from many users and every of them dont like to wait too much. So the preferrable is if the python server solution sill be optional. So who don't need it could to not compile it. Sincerelly I don t see any advantage in the python server side inside the qgis. The only scenario I could admit is for a rapid prototype of some solution, but afer, this for an usage in a production environment the python solution should be transform in a C/C++ solution to have the sufficient execution rapidity. A. 2014-08-29 10:42 GMT+02:00 Alessandro Pasotti apaso...@gmail.com: 2014-08-29 10:24 GMT+02:00 Pirmin Kalberer pi...@sourcepole.com: Hi Alessandro, Am Donnerstag, 28. August 2014, 14.49:47 schrieb Alessandro Pasotti: as anticipated to some of you during the last HF, I've started to add Python plugins support for the server side. The rationale behind server plugins is doublefold: first they could provide additional services without the need to touch the C++ codebase, second they allow for GUI-based configuration since the server plugins are not separated from the desktop plugins (of course the environment and permissions should be carefully configured to allow information sharing from the desktop user to the webserver user ). This looks like the first real case for an RFC! I'm quite sceptical adding Python to QGIS server. In my opinion this should be discussed thoroughly first. Hello Pirmin, thanks for the feedback!!! At least this is the start of a discussion :) I knew from previous conversations with some of you that adding python plugins to the server was not at the top of the priority list and I added QEP (RFC) in the original subject because I know this topic must be discussed: I'm not going to do a PR anytime soon, and my implementation is more like an hack to see how far can I go with this idea, but, so far, it seems quite promising. Everybody knows how has been important to have Python plugins on the desktop side, coding in Python is 5 to 10 times faster than building C++ code and moreover the lower barrier to start being productive will bring a lot of new functionalities from a larger pool of developers. Last but not least, python plugins do not need to pass the PR/pull/build/relase cycle that affect the C++ core, which adds a considerable delay to bug fixes and new features. That said, I would really like to hear more from you all about the reasons to be against this idea. -- Alessandro Pasotti w3: www.itopen.it ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- - Andrea Peri . . . . . . . . . qwerty àèìòù - ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
Il 29/08/2014 11:41, Andrea Peri ha scritto: I guess the main goal is to avoid that the adopt of python on server could slow it. Hi Andrea, In the first case I imagine (executing Processing modules or models as WPS services), the overhead will be minimal, as all the heavy computation is run by the appropriate backend. All the best. -- Paolo Cavallini - www.faunalia.eu Corsi QGIS e PostGIS: http://www.faunalia.eu/training.html ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
2014-08-29 11:56 GMT+02:00 Marco Hugentobler marco.hugentob...@sourcepole.ch: Hi Alessandro Thanks for this interesting input. I'm happy to look at it in detail, but need a bit of time to review it and think about it thoroughly. Also I'd like to do some testing (e.g. run some of my internal server benchmarks here). As a fery first 2c: Python on the server can open some interesting options (expression functions in python, custom renderers, ...). Of course it needs the possibility to turn python off for server. Regards, Marco Thank you Marco, I will be happy to hear about your testing results and I highly appreciate all advice that you can give me! I'm sure that there are many improvements that can be made to my hack-ish proof-of-concept implementation. A quick question: python plugins only add an #include and two lines of code in qgis_map_serv.cpp, a #ifdef SERVERPLUGINS block should be enough to disable them completely, where should I add that option to let cmake know about it? -- Alessandro Pasotti w3: www.itopen.it ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
Hi Pirmin, On Fri, 29. Aug 2014 at 10:24:00 +0200, Pirmin Kalberer wrote: This looks like the first real case for an RFC! I'm quite sceptical adding Python to QGIS server. In my opinion this should be discussed thoroughly first. I recently spend some time exploring the server side expecially QWC - or other options. If I didn't know before, I'd now know why it's called web. Frameworks over frameworks in countless languages.Some frameworks are are quite similar, some are clones or heavily inspired by others, just implementend in a different languages. You can't explore them all, but you can choose without exploring either. Really frustrating. News to me was that javascript is apparently also heavily used on the server side based on node.js. So more clones and alternatives on the server side to choose from. And we have javascript in Qt. There's even a generator that builds Qt bindings for javascript. If that could also be done for QGIS bindings, we could have plugins in javascript. And those could work on desktop, server and android. But that's just an idea that striked me - exploring it would have sidetracked me even more, so I didn't. Anyway - maybe good enough to share it. No links. I just tried to lookup qt bindings generator I mentioned and found a qt binding generator for node.js instead - example on the frontpage a message box - message box on the server side? No, I'll stop here, no more exploration for now. ;) Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de signature.asc Description: Digital signature ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
Hi Poalo, How you can say that the overhead should be minimal. ? :)) However surely best ifthe python on server is optional in compiling. So if an user dont want it should remove it. A. 2014-08-29 11:45 GMT+02:00 Paolo Cavallini cavall...@faunalia.it: Il 29/08/2014 11:41, Andrea Peri ha scritto: I guess the main goal is to avoid that the adopt of python on server could slow it. Hi Andrea, In the first case I imagine (executing Processing modules or models as WPS services), the overhead will be minimal, as all the heavy computation is run by the appropriate backend. All the best. -- Paolo Cavallini - www.faunalia.eu Corsi QGIS e PostGIS: http://www.faunalia.eu/training.html ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- - Andrea Peri . . . . . . . . . qwerty àèìòù - ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
Il 29/08/2014 12:09, Andrea Peri ha scritto: Hi Poalo, How you can say that the overhead should be minimal. Because in that case Python does almost nothing, just passing the command to GDAL, SAGA, GRASS, or whatever. All the best. -- Paolo Cavallini - www.faunalia.eu Corsi QGIS e PostGIS: http://www.faunalia.eu/training.html ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
The startuptime increase if iT should lo ad the python environment. All the best. Il 29/ago/2014 12:13 Paolo Cavallini cavall...@faunalia.it ha scritto: Il 29/08/2014 12:09, Andrea Peri ha scritto: Hi Poalo, How you can say that the overhead should be minimal. Because in that case Python does almost nothing, just passing the command to GDAL, SAGA, GRASS, or whatever. All the best. -- Paolo Cavallini - www.faunalia.eu Corsi QGIS e PostGIS: http://www.faunalia.eu/training.html ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
2014-08-29 12:23 GMT+02:00 Andrea Peri aperi2...@gmail.com: The startuptime increase if iT should lo ad the python environment. All the best. The startup will happen only the first time, unless you use CGI and not FCGI. Anyway, I proposed it a couple of years ago, but many told it was useless for various reasons. I think it could be useful if the plugin could affect WMS/WFS request/response. As an independent service I would rather use standalone python web code (actually I do it this way, loading, parsing and caching the qgs project and, in case, using headless QGIS APIs). giovanni Il 29/ago/2014 12:13 Paolo Cavallini cavall...@faunalia.it ha scritto: Il 29/08/2014 12:09, Andrea Peri ha scritto: Hi Poalo, How you can say that the overhead should be minimal. Because in that case Python does almost nothing, just passing the command to GDAL, SAGA, GRASS, or whatever. All the best. -- Paolo Cavallini - www.faunalia.eu Corsi QGIS e PostGIS: http://www.faunalia.eu/training.html ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Giovanni Allegri http://about.me/giovanniallegri Twitter: https://twitter.com/_giohappy_ blog: http://blog.spaziogis.it GEO+ geomatica in Italia http://bit.ly/GEOplus ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
OS the python usable in a fcgi environment ? This mean to load and lock an instante of python forma every fcgi session (almost 40-60 afaik) ? A. Il 29/ago/2014 12:47 G. Allegri gioha...@gmail.com ha scritto: 2014-08-29 12:23 GMT+02:00 Andrea Peri aperi2...@gmail.com: The startuptime increase if iT should lo ad the python environment. All the best. The startup will happen only the first time, unless you use CGI and not FCGI. Anyway, I proposed it a couple of years ago, but many told it was useless for various reasons. I think it could be useful if the plugin could affect WMS/WFS request/response. As an independent service I would rather use standalone python web code (actually I do it this way, loading, parsing and caching the qgs project and, in case, using headless QGIS APIs). giovanni Il 29/ago/2014 12:13 Paolo Cavallini cavall...@faunalia.it ha scritto: Il 29/08/2014 12:09, Andrea Peri ha scritto: Hi Poalo, How you can say that the overhead should be minimal. Because in that case Python does almost nothing, just passing the command to GDAL, SAGA, GRASS, or whatever. All the best. -- Paolo Cavallini - www.faunalia.eu Corsi QGIS e PostGIS: http://www.faunalia.eu/training.html ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Giovanni Allegri http://about.me/giovanniallegri Twitter: https://twitter.com/_giohappy_ blog: http://blog.spaziogis.it GEO+ geomatica in Italia http://bit.ly/GEOplus ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
A quick question: python plugins only add an #include and two lines of code in qgis_map_serv.cpp, a #ifdef SERVERPLUGINS block should be enough to disable them completely, where should I add that option to let cmake know about it? You can make an entries in CMakeLists.txt and in cmake_templates/qgsconfig.h.in. As an example, look at WITH_ORACLE and HAVE_ORACLE Regards, Marco On 29.08.2014 12:06, Alessandro Pasotti wrote: 2014-08-29 11:56 GMT+02:00 Marco Hugentobler marco.hugentob...@sourcepole.ch: Hi Alessandro Thanks for this interesting input. I'm happy to look at it in detail, but need a bit of time to review it and think about it thoroughly. Also I'd like to do some testing (e.g. run some of my internal server benchmarks here). As a fery first 2c: Python on the server can open some interesting options (expression functions in python, custom renderers, ...). Of course it needs the possibility to turn python off for server. Regards, Marco Thank you Marco, I will be happy to hear about your testing results and I highly appreciate all advice that you can give me! I'm sure that there are many improvements that can be made to my hack-ish proof-of-concept implementation. A quick question: python plugins only add an #include and two lines of code in qgis_map_serv.cpp, a #ifdef SERVERPLUGINS block should be enough to disable them completely, where should I add that option to let cmake know about it? -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Weberstrasse 5, CH-8004 Zürich, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
2014-08-29 13:34 GMT+02:00 Marco Hugentobler marco.hugentob...@sourcepole.ch: A quick question: python plugins only add an #include and two lines of code in qgis_map_serv.cpp, a #ifdef SERVERPLUGINS block should be enough to disable them completely, where should I add that option to let cmake know about it? You can make an entries in CMakeLists.txt and in cmake_templates/qgsconfig.h.in. As an example, look at WITH_ORACLE and HAVE_ORACLE Regards, Marco Thanks Marco, just done: https://github.com/elpaso/QGIS/commit/28374bffaad43513812523a2823c740b91a8343c Regards. -- Alessandro Pasotti w3: www.itopen.it ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
Another question. Is about the security of the environment. I don't know too well the possibility of python in a qgis project. But I fear that a qgis project could have some python code to start ad startup of the qgis project. This mean that the same code should be run in the server when it start a qgis-server instance in fcgi ? However the more important option is to be able to choose to fully disable this option. Thx, A. 2014-08-29 13:18 GMT+02:00 Andrea Peri aperi2...@gmail.com: OS the python usable in a fcgi environment ? This mean to load and lock an instante of python forma every fcgi session (almost 40-60 afaik) ? A. Il 29/ago/2014 12:47 G. Allegri gioha...@gmail.com ha scritto: 2014-08-29 12:23 GMT+02:00 Andrea Peri aperi2...@gmail.com: The startuptime increase if iT should lo ad the python environment. All the best. The startup will happen only the first time, unless you use CGI and not FCGI. Anyway, I proposed it a couple of years ago, but many told it was useless for various reasons. I think it could be useful if the plugin could affect WMS/WFS request/response. As an independent service I would rather use standalone python web code (actually I do it this way, loading, parsing and caching the qgs project and, in case, using headless QGIS APIs). giovanni Il 29/ago/2014 12:13 Paolo Cavallini cavall...@faunalia.it ha scritto: Il 29/08/2014 12:09, Andrea Peri ha scritto: Hi Poalo, How you can say that the overhead should be minimal. Because in that case Python does almost nothing, just passing the command to GDAL, SAGA, GRASS, or whatever. All the best. -- Paolo Cavallini - www.faunalia.eu Corsi QGIS e PostGIS: http://www.faunalia.eu/training.html ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Giovanni Allegri http://about.me/giovanniallegri Twitter: https://twitter.com/_giohappy_ blog: http://blog.spaziogis.it GEO+ geomatica in Italia http://bit.ly/GEOplus -- - Andrea Peri . . . . . . . . . qwerty àèìòù - ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
2014-08-29 15:13 GMT+02:00 Andrea Peri aperi2...@gmail.com: Another question. Is about the security of the environment. I don't know too well the possibility of python in a qgis project. But I fear that a qgis project could have some python code to start ad startup of the qgis project. This mean that the same code should be run in the server when it start a qgis-server instance in fcgi ? No. Plugin in QGIS desktop application are not bound to projects but to the user preferences (i.e. are installed in user directory and every single user can enable/disable/install/remove plugins independently. The same is for the server plugins, but because the server user doesn't normally have a user directory, the path in which plugins are stored is configurable by some environmen variables that points the plugins path to a directory of your choice, this can be one of the (trusted) user's directories or whatever else you choose. This means that you could have different FGCI server instances (apache virtual servers, for example) that use different sets of environment options and point to different sets of installed plugins. As always, it's administrator's choice how to configure the environment and secure it (choose proper permissions etc). Up to this point, there is nothing new or special for QGIS. For python startup code, this is simply ignored by my implementation but it would be possible to write a server plugin that parses the project file and executes it (not that I see anything useful in doing that). However the more important option is to be able to choose to fully disable this option. Just done. -- Alessandro Pasotti w3: www.itopen.it ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
Hi Alessandro Thanks for this interesting input. I'm happy to test it and to develope some use case. I thought it's a good feature and can help to enhance QGIS-Server easily : WPS, WMTS, Table Join Service, WFS2 or other that OGC does not already know. Regards, René-Luc Le 28/08/2014 14:49, Alessandro Pasotti a écrit : Hello, as anticipated to some of you during the last HF, I've started to add Python plugins support for the server side. The rationale behind server plugins is doublefold: first they could provide additional services without the need to touch the C++ codebase, second they allow for GUI-based configuration since the server plugins are not separated from the desktop plugins (of course the environment and permissions should be carefully configured to allow information sharing from the desktop user to the webserver user ). I've already coded a first working implementation and a few proof-of-concept plugins. The current implementation is based on the following ideas: QGIS Python plugins can have additional metatags to specify what SERVICE and REQUEST they expose, for example: service=HELLO methods=GetCapabilities,GetOutput,RemoteConsole This plugin will respond to calls like /cgi-bin/qgis_mapserv.fcgi?SERVICE=HELLOREQUEST=RemoteConsole The current implementation is pretty simple and CGI-style, whatever the plugin prints to sdtout is captured and sent back to the client, the return value from the method is the (optional) content type. I would like to discuss this ideas during the next HF (or sooner), in particular: * I'm not happy with the current request handling and I think we need a mehod to pass cached parsed project instances to the plugins when needed and a better handling of HTTP headers * I 'd like to see signals/slots implementation instead of static function calls, but maybe it's just syntactic sugar * from the python side, we don't have an iface to pass to the plugins, so I'm currently stuck to @staticmethod The code is in a branch: https://github.com/elpaso/QGIS/tree/serverplugins HelloServer demo plugin: https://github.com/elpaso/qgis-helloserver It is a test plugin and it provides a remote console (web shell), insanely insecure DO NOT USE IN PRODUCTION!! and arbitrary python code execution configurable through the QGIS desktop plugin GUI. Here is a short demo of the HelloServer plugin: http://youtu.be/ys9VVxQ8uac Have fun !! ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
2014-08-29 16:39 GMT+02:00 René-Luc Dhont rldh...@gmail.com: Hi Alessandro Thanks for this interesting input. I'm happy to test it and to develope some use case. I thought it's a good feature and can help to enhance QGIS-Server easily : WPS, WMTS, Table Join Service, WFS2 or other that OGC does not already know. Regards, René-Luc Hello René-Luc, I'm happy to hear that you are going to test it, please let me know how it goes! Keep in mind that it's just a proof of concept, and yes, we need a RFC or QEP: the current implementation is the simplest, less obtrusive and quickest that I could think about. For example, I would consider adding real signals attached to plugins slots and cached parsed projects passed to plugin instances... I'm currently working on a WPS server plugin that bridges Processing and PyWPS (version 3.2.2 since the latest version 4 is tightly bound to werkzeug), now fighting with parameters but it basically works. Regards -- Alessandro Pasotti w3: www.itopen.it ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
Le 29/08/2014 16:54, Alessandro Pasotti a écrit : 2014-08-29 16:39 GMT+02:00 René-Luc Dhont rldh...@gmail.com: Hi Alessandro Thanks for this interesting input. I'm happy to test it and to develope some use case. I thought it's a good feature and can help to enhance QGIS-Server easily : WPS, WMTS, Table Join Service, WFS2 or other that OGC does not already know. Regards, René-Luc Hello René-Luc, I'm happy to hear that you are going to test it, please let me know how it goes! Keep in mind that it's just a proof of concept, and yes, we need a RFC or QEP: the current implementation is the simplest, less obtrusive and quickest that I could think about. For example, I would consider adding real signals attached to plugins slots and cached parsed projects passed to plugin instances... I'm currently working on a WPS server plugin that bridges Processing and PyWPS (version 3.2.2 since the latest version 4 is tightly bound to werkzeug), now fighting with parameters but it basically works. Can I help ? I already done some works on it ! Regards ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
2014-08-29 18:05 GMT+02:00 René-Luc Dhont rldh...@gmail.com: Le 29/08/2014 16:54, Alessandro Pasotti a écrit : 2014-08-29 16:39 GMT+02:00 René-Luc Dhont rldh...@gmail.com: Hi Alessandro Thanks for this interesting input. I'm happy to test it and to develope some use case. I thought it's a good feature and can help to enhance QGIS-Server easily : WPS, WMTS, Table Join Service, WFS2 or other that OGC does not already know. Regards, René-Luc Hello René-Luc, I'm happy to hear that you are going to test it, please let me know how it goes! Keep in mind that it's just a proof of concept, and yes, we need a RFC or QEP: the current implementation is the simplest, less obtrusive and quickest that I could think about. For example, I would consider adding real signals attached to plugins slots and cached parsed projects passed to plugin instances... I'm currently working on a WPS server plugin that bridges Processing and PyWPS (version 3.2.2 since the latest version 4 is tightly bound to werkzeug), now fighting with parameters but it basically works. Can I help ? I already done some works on it ! Sure! Better have an IRC/gmail talk. I'll try to polish a bit the code and upload what I've done so far. Currently the biggest problem seems that process.Popen on grass commands hangs :( -- Alessandro Pasotti w3: www.itopen.it ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QEP: QGIS Mapserver Python Plugins
Hi Jürgen, On Fri, Aug 29, 2014 at 4:08 AM, Jürgen E. j...@norbit.de wrote: Hi Pirmin, On Fri, 29. Aug 2014 at 10:24:00 +0200, Pirmin Kalberer wrote: This looks like the first real case for an RFC! I'm quite sceptical adding Python to QGIS server. In my opinion this should be discussed thoroughly first. I recently spend some time exploring the server side expecially QWC - or other options. If I didn't know before, I'd now know why it's called web. Frameworks over frameworks in countless languages.Some frameworks are are quite similar, some are clones or heavily inspired by others, just implementend in a different languages. You can't explore them all, but you can choose without exploring either. Really frustrating. News to me was that javascript is apparently also heavily used on the server side based on node.js. So more clones and alternatives on the server side to choose from. And we have javascript in Qt. There's even a generator that builds Qt bindings for javascript. If that could also be done for QGIS bindings, we could have plugins in javascript. And those could work on desktop, server and android. But that's just an idea that striked me - exploring it would have sidetracked me even more, so I didn't. Anyway - maybe good enough to share it. No links. I just tried to lookup qt bindings generator I mentioned and found a qt binding generator for node.js instead - example on the frontpage a message box - message box on the server side? No, I'll stop here, no more exploration for now. ;) But wait, there's more! :-) (Sorry, off-topic.) Yesterday, I came across the Duktape [0] embeddable JavaScript engine, recommended by the osgEarth project as a new dependency [1]. It has a very small footprint and simple implementation. Not sure how that would work with Qt, or even if it is needed beyond Qt's JavaScript generator, engine, etc. Just found it interesting. Could be used to directly integrate things like d3.js [2] or node.js without the extra interface through Qt's QScript or QWebPage? (Just guessing there.) If/when QGIS's code base has been updated to Qt5, JavaScript will become readily available as a scripting solution as well. [0] http://duktape.org/ [1] http://docs.osgearth.org/en/latest/startup.html#get-the-dependencies [2] http://d3js.org/ , http://bost.ocks.org/mike/map/ Regards, Larry Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iD8DBQFUAFEOc/GhTF5ESHURAgk0AJ94Z4mAz3NGtmwY1Hphe413VfxCSQCgxXPc MDoNlMIc6/N1kkTUwJsnpDs= =cCLx -END PGP SIGNATURE- ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
[Qgis-developer] QEP: QGIS Mapserver Python Plugins
Hello, as anticipated to some of you during the last HF, I've started to add Python plugins support for the server side. The rationale behind server plugins is doublefold: first they could provide additional services without the need to touch the C++ codebase, second they allow for GUI-based configuration since the server plugins are not separated from the desktop plugins (of course the environment and permissions should be carefully configured to allow information sharing from the desktop user to the webserver user ). I've already coded a first working implementation and a few proof-of-concept plugins. The current implementation is based on the following ideas: QGIS Python plugins can have additional metatags to specify what SERVICE and REQUEST they expose, for example: service=HELLO methods=GetCapabilities,GetOutput,RemoteConsole This plugin will respond to calls like /cgi-bin/qgis_mapserv.fcgi?SERVICE=HELLOREQUEST=RemoteConsole The current implementation is pretty simple and CGI-style, whatever the plugin prints to sdtout is captured and sent back to the client, the return value from the method is the (optional) content type. I would like to discuss this ideas during the next HF (or sooner), in particular: * I'm not happy with the current request handling and I think we need a mehod to pass cached parsed project instances to the plugins when needed and a better handling of HTTP headers * I 'd like to see signals/slots implementation instead of static function calls, but maybe it's just syntactic sugar * from the python side, we don't have an iface to pass to the plugins, so I'm currently stuck to @staticmethod The code is in a branch: https://github.com/elpaso/QGIS/tree/serverplugins HelloServer demo plugin: https://github.com/elpaso/qgis-helloserver It is a test plugin and it provides a remote console (web shell), insanely insecure DO NOT USE IN PRODUCTION!! and arbitrary python code execution configurable through the QGIS desktop plugin GUI. Here is a short demo of the HelloServer plugin: http://youtu.be/ys9VVxQ8uac Have fun !! -- Alessandro Pasotti w3: www.itopen.it ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
