Re: [Qgis-developer] QGIS Processing Framework
Hi Camilo On Tue, Apr 26, 2011 at 9:46 PM, Camilo Polymeris cpolyme...@gmail.com wrote: Hello everyone, I was accepted as a student in GSoC to implement a SAGA interface for QGIS. First of all congratulations to you and to other accepted students for the Summer of Code programme! There are also proyects that try to interface QGIS with GRASS and the Orfeo Toolbox (OTB). Paolo has informed me that the creation of a common framework for those systems was discussed and approved during HF. A very interesting idea, IMO, that completely changes the way I would have to work on my project. Great that you like the idea of generic framework for processing tools in QGIS. Helping us to bring this framework from ideas into code would be another great benefit of your project. While I didn't participate in those discussions and don't know what conclusions were drawn, as a brainstorming exercise I am writing a short description of what a very simple API to that framework could look like: https://github.com/polymeris/qgis/wiki/QGIS-Processing-Framework Would that more or less fit what you have in mind? Please comment/criticize. The discussion was not going into implementation details, we were mostly collecting the ideas how the things should work. Framework implementation: as you write in the proposal, ideally it should be implemented in C++ with Python bindings. I would suggest you to start the implementation in Python and skip C++ coding at least at the beginning. My experience is that Python is great for this kind of prototyping (no compiling, shorter code etc). If you will end up with sufficient amount of time at the end of the summer, you can move that into C++ and add Python bindings. The idea is not to waste time with writing and rewriting various c++ classes when the API is not clear yet. Consider this as a suggestion, not a command how to do it :-) Forms/Parameters: I would start with with generic input fields (line edit / spin box) for integers, floats, strings that can be constrained with a range (numbers) or regular expressions (strings) - easily achieved with QValidator classes. Then some custom widgets can be added for better handling of some types: layer selector, enumeration etc. The framework may be extensible so that if a library often uses a particular custom type, the plugin integrating the library could add a custom widget for it. Advanced parameters: some modules may allow users set lots of parameters. Although this is handy, many times the default values work well and only few basic parameters are changed regularly. It should be possible to mark some parameters as advanced and they would be hidden by default, they would show up after clicking a checkbox or a button. For example, when creating a vector buffer, the distance would be a basic parameter while the number of segments per quadrant would be advanced parameter. Categorization: currently we are starting to face some trouble with organization of plugins and the modules they provide. We thought about creating fixed categories as in GRASS (e.g. Raster, Vector, Database) where the modules would be placed. We also considered custom categories and subcategories. The general problem with categories is that many modules do not suit them well if they are not coming from a well-organized library such as grass, saga or otb. The conclusion that I recall from the discussion at hackfest is that tagging is probably the only way to go: each module could have several tags assigned to it, then the list of modules would be a tree with top level items containing tags and modules would be shown inside these tags. Bigger libraries requiring deeper hierarchy might come with hierarchical tags like SAGA/Grid - Tools. The names of tags will be left for developers' creativity. This list of processing modules should later get some advanced functionality like searching, favourite modules or even attaching most used modules on a toolbar, but that is something for the future, we are not expecting you to do that during the summer :-) A simple tree listing tags and modules assigned to them will suffice. Formats and format conversion: for import and export of map layers the framework should support any layer loaded (or loadable) by QGIS. In case the library uses a different format for input/output, it should take care of import/export. Running modules from scripts: you probably have that in mind but let's keep it explicit: the framework should provide API that would allow user to run modules from python console or from a plugin, QGIS would basically act just as a default GUI using that API. This will finally allow users to run the processing modules in batches - poor man's pipelining :-) Feedback: many modules are able to return some feedback - either in the form of a progress to indicate how long the process will take - or to return some messages that will be shown to user. Support for cancelling a running module would be helpful too, but not
Re: [Qgis-developer] QGIS Processing Framework
Il giorno mer, 27/04/2011 alle 11.20 +0200, Martin Dobias ha scritto: into C++ and add Python bindings. The idea is not to waste time with writing and rewriting various c++ classes when the API is not clear yet. Consider this as a suggestion, not a command how to do it :-) Agreed; it will also be much easier for interested people to follow the work. Advanced parameters: some modules may allow users set lots of parameters. Although this is handy, many times the default values work well and only few basic parameters are changed regularly. It should be possible to mark some parameters as advanced and they would be hidden by default, they would show up after clicking a checkbox or a button. Right, this works very well with GRASS modules. Categorization: currently we are starting to face some trouble with ... This list of processing modules should later get some advanced functionality like searching, favourite modules or even attaching most used modules on a toolbar, but that is something for the future, we are not expecting you to do that during the summer :-) A simple tree listing tags and modules assigned to them will suffice. I personally like and use the filtering option of GRASS plugin a lot, and find this the best option for all users except possibly for the very first-timers. Running modules from scripts: you probably have that in mind but let's Right, this is very important, also to be able to add extra options etc. Any news from OTB people? Their input is important. All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Processing Framework
Hello, I'm currently publishing what we have (the specs we made and the code of our new framework), so that we can see what can be merged and how. More extensive inputs and reaction to current ideas this afternoon... Julien Le 27/04/2011 11:30, Paolo Cavallini a écrit : Il giorno mer, 27/04/2011 alle 11.20 +0200, Martin Dobias ha scritto: into C++ and add Python bindings. The idea is not to waste time with writing and rewriting various c++ classes when the API is not clear yet. Consider this as a suggestion, not a command how to do it :-) Agreed; it will also be much easier for interested people to follow the work. Advanced parameters: some modules may allow users set lots of parameters. Although this is handy, many times the default values work well and only few basic parameters are changed regularly. It should be possible to mark some parameters as advanced and they would be hidden by default, they would show up after clicking a checkbox or a button. Right, this works very well with GRASS modules. Categorization: currently we are starting to face some trouble with ... This list of processing modules should later get some advanced functionality like searching, favourite modules or even attaching most used modules on a toolbar, but that is something for the future, we are not expecting you to do that during the summer :-) A simple tree listing tags and modules assigned to them will suffice. I personally like and use the filtering option of GRASS plugin a lot, and find this the best option for all users except possibly for the very first-timers. Running modules from scripts: you probably have that in mind but let's Right, this is very important, also to be able to add extra options etc. Any news from OTB people? Their input is important. All the best. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
[Qgis-developer] Bug or problem under Mac, opening a raster tif
Using 1.6 or 1.7 15814. i have some problem with tif raster georeferenced with qgis extension under mac. Trying similar operations under Windows and Mac I noticed a problem. Project are set to WGS84 and fly reprojection activated (windows and mac) When I m adding a raster under georeferencing extension, Windows ask me for assigning a system projection for this raster , not under Mac. Regards Patrice Vetsel___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Processing Framework
Thanks, Martin, for your well-thought-out response! https://github.com/polymeris/qgis/wiki/QGIS-Processing-Framework Would that more or less fit what you have in mind? Please comment/criticize. The discussion was not going into implementation details, we were mostly collecting the ideas how the things should work. Yes. I wrote that API description not so much thinking it would be the actual interface, but as a way of reflecting on what issues could arise and how to solve them. Framework implementation: as you write in the proposal, ideally it should be implemented in C++ with Python bindings. I would suggest you to start the implementation in Python and skip C++ coding at least at the beginning. Ok. I think that's a good idea. I'll start with Python and later see when (and if) migration to C++ makes sense. Forms/Parameters: I would start with with generic input fields (line edit / spin box) for integers, floats, strings that can be constrained with a range (numbers) or regular expressions (strings) - easily achieved with QValidator classes. Then some custom widgets can be added for better handling of some types: layer selector, enumeration etc. The framework may be extensible so that if a library often uses a particular custom type, the plugin integrating the library could add a custom widget for it. Agreed. In general I think it may be a good idea to keep the GUI stuff in QGIS and have the implementations only care about interface to the various systems. BTW, in the document linked above I use the term library for a collection of modules, as opposed to implementation for a given interface to a system (GRASS, SAGA, OTB). There may be more than one library per implementation. Just to clarify-- terminology can be changed, of course. I don't even know if the title (Processing Framework) is accurate. Perhaps Analysis is a better name. Advanced parameters: some modules may allow users set lots of parameters. Although this is handy, many times the default values work well and only few basic parameters are changed regularly. It should be possible to mark some parameters as advanced and they would be hidden by default, they would show up after clicking a checkbox or a button. For example, when creating a vector buffer, the distance would be a basic parameter while the number of segments per quadrant would be advanced parameter. Good point. SAGA, as far as I know, does not provide information on which parameters are considered basic and which are advanced. Perhaps a list of parameters considered advanced can be included with the plugin. Not very elegant, because it would be out-of-sync with SAGA, but practical. Categorization: currently we are starting to face some trouble with organization of plugins and the modules they provide. We thought about creating fixed categories as in GRASS (e.g. Raster, Vector, Database) where the modules would be placed. We also considered custom categories and subcategories. The general problem with categories is that many modules do not suit them well if they are not coming from a well-organized library such as grass, saga or otb. The conclusion that I recall from the discussion at hackfest is that tagging is probably the only way to go: each module could have several tags assigned to it, then the list of modules would be a tree with top level items containing tags and modules would be shown inside these tags. Bigger libraries requiring deeper hierarchy might come with hierarchical tags like SAGA/Grid - Tools. The names of tags will be left for developers' creativity. This list of processing modules should later get some advanced functionality like searching, favourite modules or even attaching most used modules on a toolbar, but that is something for the future, we are not expecting you to do that during the summer :-) A simple tree listing tags and modules assigned to them will suffice. Again, SAGA provides no tag information. It would have to be generated programatically from library/module name description or provided in a list. Any thoughts on the GUI for that tree? More like SAGA's module tree or more like GIMP's filter menu? I don't think the search/filter/favorites/frequently used part is very hard.. but still, I will keep things simple. Feedback: many modules are able to return some feedback - either in the form of a progress to indicate how long the process will take - or to return some messages that will be shown to user. Support for cancelling a running module would be helpful too, but not really necessary right now. Related topics: * Interactive modules. I had this more or less resolved for SAGA (I think), but designing a more generic solution could prove difficult. * Pushing pipelining to the implementation: Potentially more efficient, but less generic. * Parameter value serialization: load and save presets to file * Undo redo ability. I am eager to see what the OTB team has come up with. I am not very
Re: [Qgis-developer] Bug or problem under Mac, opening a raster tif
Hi, I cannot test on any of these 2 OSes, but just a hint (maybe): In menu Settings/options, CRS tab, there are 3 different options on what to do when a layer has no CRS. I would first check that they are identical on both your platforms. Mayeul Le mercredi 27 avril 2011 à 17:00 +0200, Patrice Vetsel a écrit : Using 1.6 or 1.7 15814. i have some problem with tif raster georeferenced with qgis extension under mac. Trying similar operations under Windows and Mac I noticed a problem. Project are set to WGS84 and fly reprojection activated (windows and mac) When I m adding a raster under georeferencing extension, Windows ask me for assigning a system projection for this raster , not under Mac. Regards Patrice Vetsel___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Processing Framework
Hi Camilo, Hi all, First of all congratulation for your GSoC project ! Some apologies for the delay in my response, we are approaching the next OTB release so I'm busy on fixing little stuff/builds here and there. Please don't expect real time answers in the next week or two. But it's nice to see you left the starting blocks a while ago :) For OTB we are developing a module framework also, with auto-generated interface. As I told people at the HF, we have made a first step with our last 3.8 release, but are not satisfied enough with the GUI part, so we are currently revamping the core to add a lot of functionnalities. Our ideas/wish-list is summed up here : http://wiki.orfeo-toolbox.org/index.php/OTB-Wrapper The current code for what we want to push into QGis is here (please note that it is in development, i've completed the first usefull app just after the HackFest) : http://hg.orfeo-toolbox.org/OTB-Wrapper Don't hesitate to ask me for help to build it if you need. Some general comments : [1] We should be very careful about not being limited by too much genericity. The SAGA and GRASS modules have each their own design/parameter types/inner but also outer logic. On the OTB side, we have some liberty since we are currently working on them, but we have some plans to integrate them in other tools, possibly benefiting from their additional possibilities (the pipeline capabilities of Monteverdi for example). We have some ideas for making the GUI as dynamic as possible (and it is with their Qt/Qgis implementation that we plan to have the most full-featured GUI) but I guess other libs will have capabilities that we don't have. Trying to fit everyone in the same framework will be hard : either each library (SAGA/GRASS/OTB) will only use a subset of the provided functionnalities, either we will end up taking the common factor of all libs and everyone will feel limited. I'm worried than we fall into one of these two pitfalls. Please do not impose to much constraints ! Reactions very welcome on this point ! What seems reasonable to me at first sight : - using similar widgets for similar functionnalities/parameter types, maybe with base classes to reuse (but again make them really high level !). - having a common way to call the modules - all in all, make them all look the same This will give the user the illusion that they all comes from the same library, but will give each implementation the necessary liberty to fine tune their GUIs using the specificities of their library/internal module framework. We have the chance of being in a similar phase of development for both project, so i'm sure we will find the right common ground. By the way, after SAGA, Grass and OTB, maybe someone will consider Ossim : they also have a similar framework for modules with autogenerated interfaces, to do raster processing. Seeing Maxim presenting a Qgis classification plugin based on OpenCV, maybe you will also end up with some OpenCV modules in the future... [2] Chaining modules from different libraries seems a little ambitious to me, but this would be awesome ! As you said Camilo, I think we can leave pipelining out as a first step. We are not really focusing on that part now neither, though the current implementation should allow this. [3] You seem to consider parameter validation only in an independent way From our experience it is insufficient (for example, absolute default values is not enough). How we plan to solve this (trying to keep it simple also...), is to give our modules 3 main entry points : - description of the parameters - a global method to update all the parameters, called as soon as one parameter is changed - an execution method Let's imagine a simple orthorectification application with the following params : - input/output image - the output CRS - output origin, size, and spacing We want the origin, size and spacing to be auto-computed each time the input or CRS changes If CRS changes from UTM to WGS84, we expect the outputorigin units/display to also change from meter to degree, for example. [4] (related to [3]) Our first working-and-usefull application in this new framework we're building shows also dynamic parameters list in the GUI (see the screenshot on the Wiki). It is a smoothing application with 3 modes: Median, Gaussian, Anisotropic Filtering. Each mode has associated params. Median and Gaussian needs a radius, Anisotropic Filtering needs a time step and a number of iterations. So below the combobox, the widget displayed depend on the combobox value. [5] The model we imagined for the parameter list will be a tree more than a flat list. A group of parameters is just one node of the tree, containing a parameters list, where each parameter can also be a group of parameters. Please find also some other more specific comments below : Le 27/04/2011 11:20, Martin Dobias a écrit : Hi Camilo On Tue, Apr 26, 2011 at 9:46 PM,
Re: [Qgis-developer] QGIS Processing Framework
Il giorno mer, 27/04/2011 alle 12.00 -0400, Camilo Polymeris ha scritto: Good point. SAGA, as far as I know, does not provide information on which parameters are considered basic and which are advanced. Perhaps a list of parameters considered advanced can be included with the plugin. Not very elegant, because it would be out-of-sync with SAGA, but practical. It is the same approach used in GRASS: in fact, it is better (even if not elegant, I agree) to be able to decide which options should be exposed as default, and which ones as advanced). In fact it would be also good to add a general option user/power user; in the first case only default options will be exposed, in the second the full options. Again, SAGA provides no tag information. It would have to be generated programatically from library/module name description or provided in a list. Any thoughts on the GUI for that tree? More like SAGA's module tree or more like GIMP's filter menu? I don't think the search/filter/favorites/frequently used part is very hard.. but still, I will keep things simple. I think filtering is often the easiest option. All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Processing Framework
Le mercredi 27 avril 2011 à 20:46 +0200, Paolo Cavallini a écrit : Il giorno mer, 27/04/2011 alle 12.00 -0400, Camilo Polymeris ha scritto: Good point. SAGA, as far as I know, does not provide information on which parameters are considered basic and which are advanced. Perhaps a list of parameters considered advanced can be included with the plugin. Not very elegant, because it would be out-of-sync with SAGA, but practical. It is the same approach used in GRASS: in fact, it is better (even if not elegant, I agree) to be able to decide which options should be exposed as default, and which ones as advanced). In fact it would be also good to add a general option user/power user; in the first case only default options will be exposed, in the second the full options. +1 for the concept of a user/power user checkbox. You could call it Always show advanced options which is more neutral (does not make any judgement on the user) and describes exactly what it does. Would be great to have an Advanced options checkbox on each window anyway, to be able to switch their visibility on a case by case basis. In addition, where relevant (functions that use command-line tools or have command-line tools equivalent) and in advanced mode only, it would be great to have the GUI generate the command line corresponding to the options chosen in the GUI; the GUI would show this command line and let the user edit it (or run it as it is). There are many advantages: - the user knows exactly what is going to be executed; advanced QGIS users will be able to use the GUI but benefit from existing help targeted at the command-line users; debugging, sharing questions and ideas in a forum or saving/automating a frequent task is facilitated - you do not have to put very rare options in the GUI (power users can still add the options here when they need it, which should be rare). The idea is to partly bridge the gap between GUI and command line by having the advantages of both (GUI is often easier, command line is generally more powerful). This concept is used successfully in quite a variety of software; just to give a few names: - pgadmin (SQL preview of any schema modification) - vlc (save, convert broadcast functionalities) - R (a few functions of the default Rgui; the Rcommander GUI package)... Just my 2 cents. All the best for this work! Mayeul Again, SAGA provides no tag information. It would have to be generated programatically from library/module name description or provided in a list. Any thoughts on the GUI for that tree? More like SAGA's module tree or more like GIMP's filter menu? I don't think the search/filter/favorites/frequently used part is very hard.. but still, I will keep things simple. I think filtering is often the easiest option. All the best. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Processing Framework
On Wed, Apr 27, 2011 at 4:58 PM, Mayeul Kauffmann mayeul.kauffm...@free.fr wrote: Le mercredi 27 avril 2011 à 20:46 +0200, Paolo Cavallini a écrit : Il giorno mer, 27/04/2011 alle 12.00 -0400, Camilo Polymeris ha scritto: Good point. SAGA, as far as I know, does not provide information on which parameters are considered basic and which are advanced. Perhaps a list of parameters considered advanced can be included with the plugin. Not very elegant, because it would be out-of-sync with SAGA, but practical. It is the same approach used in GRASS: in fact, it is better (even if not elegant, I agree) to be able to decide which options should be exposed as default, and which ones as advanced). In fact it would be also good to add a general option user/power user; in the first case only default options will be exposed, in the second the full options. +1 for the concept of a user/power user checkbox. You could call it Always show advanced options which is more neutral (does not make any judgement on the user) and describes exactly what it does. Agreed. Would be great to have an Advanced options checkbox on each window anyway, to be able to switch their visibility on a case by case basis. If it is a check-box, tab or toggle button would have to be considered. I am under the impression that the latter is more common, but I don't know the rationale behind. In addition, where relevant (functions that use command-line tools or have command-line tools equivalent) and in advanced mode only, it would be great to have the GUI generate the command line corresponding to the options chosen in the GUI; the GUI would show this command line and let the user edit it (or run it as it is). There are many advantages: - the user knows exactly what is going to be executed; advanced QGIS users will be able to use the GUI but benefit from existing help targeted at the command-line users; debugging, sharing questions and ideas in a forum or saving/automating a frequent task is facilitated Very good idea. It could even work both ways, and use that for saving/loading parameter presets. - you do not have to put very rare options in the GUI (power users can still add the options here when they need it, which should be rare). The GUI would be auto-generated, so adding parameters means no extra programming effort. Not putting things on the GUI would, thus, be a UI design decision, to keep it from being cluttered. Wow. This is getting more and more ambitious. Will have to see what fits in only 1 summer :) Thanks, Camilo ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Processing Framework
W dniu 27.04.2011 23:25, Paolo Cavallini pisze: Il giorno mer, 27/04/2011 alle 17.15 -0400, Camilo Polymeris ha scritto: Wow. This is getting more and more ambitious. Will have to see what fits in only 1 summer :) Well, you're not alone :) Paolo is right! After such great discussion your fan club is rising Camilo ;) I hope to join this threat after 'lost in 1.7 translation'. Robert ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
