Re: [Qgis-developer] Improving ftools geoprocessing tools

[Bernd Vogelgesang]

Am 22.10.2015, 13:30 Uhr, schrieb João Gaspar :


Dear QGIS Developers,

I'm writing on behalf of my employer, a company that has mixed,
proprietary and open source (especially QGIS of course) gis
infrastructure, with qgis being adopted each day a little more.
Actually in our migration path we have found a road blocker: the very
bad performances (and sometimes results) of most of geoprocessing
tools in the vector menu. Yes I know that there are alternatives in
the Processing toolbox, but that is not an option for most of our very
basic users: they need to do simple operations (clips, unions,
intersections, etc.) with usually very large/detailed vectors (usually
shapefiles). In this scenario QGIS usually chokes or is at best very
slow (compared to other gis packages, the ones we are trying to get
rid of).

What can be done to improve this tools? Rewriting the tools to improve
efficiency and accuracy?

We available some funds: this is the first time I convince people
above me to put some money in QGIS development, and a decision would
need to me made until the end of November tops, otherwise money will
be used otherwise.

Please leave advice and feedback.


Regards
João Gaspar





Hi,
I'm not a developer, but a simple user following this list.

My point of view is, that the ftools are just a relict from the past since  
the processing toolbox was introduced.
I'm glad that people step up for funding QGIS, but in this particular  
case, you would better invest your money and time in lifting your "basic"  
users to a little more advanced users and show them how to use the  
processing toolbox properly.


As far as I know, the toolbox is nearly feature complete with the ftools  
functions, and most of them are so much faster and/or versatile, that I  
did not touch the ftools for ages.
Prepare some centrally stored models for reoccuring use cases, and have  
happy users ;)


Just my two cents
Bernd



--
Bernd Vogelgesang
Siedlerstraße 2
91083 Baiersdorf/Igelsdorf
Tel: 09133-825374___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Victor Olaya]

shp is the default format for temp files, and as such it is used in
the modeler for intermediate results. That is, however, very easy to
change, so if we agree on any other format, it can be changed and set
as the default for Processing

2015-10-22 16:08 GMT+02:00 Bernhard Ströbl :
> Hi Andreas,
>
> AFAIK processing uses shp for in between results in a model, too. So if you
> build a model you loose your fields, no matter which output format you
> choose. IMHO it should therefore be considered to change processing's data
> source. Maybe SpatialLite is an option? Users can still export to shp if
> needed.
>
> Bernhard
>
>
>
> Am 22.10.2015 um 15:48 schrieb Andreas Neumann:
>>
>> Bernhard,
>>
>> I agree that these tools should be QGIS algorithms - maybe even ported
>> from another GIS (if feasible).
>>
>> One major annoyance with fTools and some processing algorithms is that
>> reliance on ESRI shapefile as output. That way I always loose my columns
>> - or they are truncated to be meaningless. For that reason I often don't
>> use QGIS, but FME for analysis. Or try to do the processing with SQL in
>> Postgis.
>>
>> If we renovate these algorithms in the vector menu we'd have to make
>> sure that the output could be any chosen vector format, or at least
>> supports geopackage/spatialite/postgres as output.
>>
>> Andreas
>>
>> On 22.10.2015 15:26, Bernhard Ströbl wrote:
>>>
>>> IMHO all ftools should be replaced by native QGIS processing
>>> algorithms. For "simple" users it may be not feasible (or not allowed)
>>> to install additional software. Therefore native QGIS algorithms
>>> should be improved to be fast, robust and offering choices the users
>>> need (see recent discussion on dissolve)
>>>
>>> Bernhard
>>>
>>> Am 22.10.2015 um 15:13 schrieb Andreas Neumann:

 If we'd replace the ftools (vector and raster menu) through processing.
 What would replace them? Would we do research and see which of the
 alternative processing providers provides best speed/reliability - if
 there are multiple versions? It may be, that for Algorithm A, Saga works
 best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
 builtin in mechanism.

 Andreas

 On 22.10.2015 14:59, Victor Olaya wrote:
>>
>>   * Why is the processing toolbox not an option for your users? (Do
>> you
>> require shortcuts (menu entries) to the tools? Is it the parameter
>> interface which is more fine-tuned? Missing functionality?)
>
>
> If what you need is to have the tools in a menu, we have been
> discussing that before, and the idea is to mantain the same menu
> structure if the users wants that, but have the algorithms based on
> Processing to avoid redundant (and more difficult to mantain) code. As
> I said, this has already been discussed, but it is not yet
> implemented. Maybe your funding could help us do it?
>
> And if algorithms are not performant enough, it's clearly better to
> put your effort in the Processing ones (which, in general, are just a
> copy of the ftools ones), and have them exposed through the menus, as
> explained above.
>
> Regards
>
>
>
> __ Information from ESET Mail Security, version of virus signature
> database 12448 (20151022) __
>
> The message was checked by ESET Mail Security.
> http://www.eset.com
>
>
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Victor Olaya]

> The issue here is about the lousy performances (and even wrong
> results) of the native qgis vector geoprocessing tools. I think we all
> agree that qgis should have a few (at least the most commonly used)
> ones, without depending on any 3rd party software.

I fully agree on that

Do we have any benchmark or list of those algorithms with very bad
performance or that give wrong results? I don't think that all of them
behave so badly, so it would be good to have more detail about where
we should be focusing our efforts in this

Cheers
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Bernhard Ströbl]

Hi Andreas,

AFAIK processing uses shp for in between results in a model, too. So if 
you build a model you loose your fields, no matter which output format 
you choose. IMHO it should therefore be considered to change 
processing's data source. Maybe SpatialLite is an option? Users can 
still export to shp if needed.


Bernhard


Am 22.10.2015 um 15:48 schrieb Andreas Neumann:

Bernhard,

I agree that these tools should be QGIS algorithms - maybe even ported
from another GIS (if feasible).

One major annoyance with fTools and some processing algorithms is that
reliance on ESRI shapefile as output. That way I always loose my columns
- or they are truncated to be meaningless. For that reason I often don't
use QGIS, but FME for analysis. Or try to do the processing with SQL in
Postgis.

If we renovate these algorithms in the vector menu we'd have to make
sure that the output could be any chosen vector format, or at least
supports geopackage/spatialite/postgres as output.

Andreas

On 22.10.2015 15:26, Bernhard Ströbl wrote:

IMHO all ftools should be replaced by native QGIS processing
algorithms. For "simple" users it may be not feasible (or not allowed)
to install additional software. Therefore native QGIS algorithms
should be improved to be fast, robust and offering choices the users
need (see recent discussion on dissolve)

Bernhard

Am 22.10.2015 um 15:13 schrieb Andreas Neumann:

If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.

Andreas

On 22.10.2015 14:59, Victor Olaya wrote:

  * Why is the processing toolbox not an option for your users? (Do
you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)


If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?

And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.

Regards



__ Information from ESET Mail Security, version of virus signature 
database 12448 (20151022) __

The message was checked by ESET Mail Security.
http://www.eset.com


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Victor Olaya]

2015-10-22 15:13 GMT+02:00 Andreas Neumann :
> If we'd replace the ftools (vector and raster menu) through processing. What
> would replace them? Would we do research and see which of the alternative
> processing providers provides best speed/reliability - if there are multiple
> versions? It may be, that for Algorithm A, Saga works best, whereas for
> algorithm B GRASS is better and for C GDAL or the QGIS builtin in mechanism.
>

SAGA algorithms are pretty fast, and we wont probably get that
performance unless we move the logic of algorithms to the C++ core (as
Alex suggested). However, I wouldn't put the SAGA algorithm to replace
the ftools menu, for the reason that you mentioned: SAGA only handles
shapefiles. I think that it would be better to get a good performance
(even if it is not that good when compared to GRASS or SAGA), but have
the support for all formats. Processing will give you that support,
converting files as needed, but in the end, it has to pass through a
shapefile, and we have the problem with field names...
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Andreas Neumann]

Bernhard,

I agree that these tools should be QGIS algorithms - maybe even ported 
from another GIS (if feasible).


One major annoyance with fTools and some processing algorithms is that 
reliance on ESRI shapefile as output. That way I always loose my columns 
- or they are truncated to be meaningless. For that reason I often don't 
use QGIS, but FME for analysis. Or try to do the processing with SQL in 
Postgis.


If we renovate these algorithms in the vector menu we'd have to make 
sure that the output could be any chosen vector format, or at least 
supports geopackage/spatialite/postgres as output.


Andreas

On 22.10.2015 15:26, Bernhard Ströbl wrote:
IMHO all ftools should be replaced by native QGIS processing 
algorithms. For "simple" users it may be not feasible (or not allowed) 
to install additional software. Therefore native QGIS algorithms 
should be improved to be fast, robust and offering choices the users 
need (see recent discussion on dissolve)


Bernhard

Am 22.10.2015 um 15:13 schrieb Andreas Neumann:

If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.

Andreas

On 22.10.2015 14:59, Victor Olaya wrote:
  * Why is the processing toolbox not an option for your users? (Do 
you

require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)


If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?

And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.

Regards
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer




__ Information from ESET Mail Security, version of virus 
signature database 12448 (20151022) __


The message was checked by ESET Mail Security.
http://www.eset.com


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Bernhard Ströbl]

What what be a suitable format? It should
1) support all geometry types needed
2) be fast
3) support spatial indices?
4) support long (unlimited) field names
5) support all necessary field types
6) store its data on local hd

anything else?

Bernhard

Am 22.10.2015 um 16:14 schrieb Victor Olaya:

shp is the default format for temp files, and as such it is used in
the modeler for intermediate results. That is, however, very easy to
change, so if we agree on any other format, it can be changed and set
as the default for Processing

2015-10-22 16:08 GMT+02:00 Bernhard Ströbl :

Hi Andreas,

AFAIK processing uses shp for in between results in a model, too. So if you
build a model you loose your fields, no matter which output format you
choose. IMHO it should therefore be considered to change processing's data
source. Maybe SpatialLite is an option? Users can still export to shp if
needed.

Bernhard



Am 22.10.2015 um 15:48 schrieb Andreas Neumann:


Bernhard,

I agree that these tools should be QGIS algorithms - maybe even ported
from another GIS (if feasible).

One major annoyance with fTools and some processing algorithms is that
reliance on ESRI shapefile as output. That way I always loose my columns
- or they are truncated to be meaningless. For that reason I often don't
use QGIS, but FME for analysis. Or try to do the processing with SQL in
Postgis.

If we renovate these algorithms in the vector menu we'd have to make
sure that the output could be any chosen vector format, or at least
supports geopackage/spatialite/postgres as output.

Andreas

On 22.10.2015 15:26, Bernhard Ströbl wrote:


IMHO all ftools should be replaced by native QGIS processing
algorithms. For "simple" users it may be not feasible (or not allowed)
to install additional software. Therefore native QGIS algorithms
should be improved to be fast, robust and offering choices the users
need (see recent discussion on dissolve)

Bernhard

Am 22.10.2015 um 15:13 schrieb Andreas Neumann:


If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.

Andreas

On 22.10.2015 14:59, Victor Olaya wrote:


   * Why is the processing toolbox not an option for your users? (Do
you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)



If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?

And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.

Regards



__ Information from ESET Mail Security, version of virus signature 
database 12448 (20151022) __

The message was checked by ESET Mail Security.
http://www.eset.com


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Giovanni Manghi]

> Icon support is already in Processing. The first versions actually
> used the same ftools icons, but I removed that because there were new
> algorithms being added, and it was quite messy to see some algorithms
> with icons and others with the default icon. Instead, I started using
> the same icon for all algorithms for a given provider, for the sake of
> homogeneity.


the main issue here is not if leave th vector menu or not, i think we
all agree we should get rid of it and have all the tools in
processing.

The issue here is about the lousy performances (and even wrong
results) of the native qgis vector geoprocessing tools. I think we all
agree that qgis should have a few (at least the most commonly used)
ones, without depending on any 3rd party software.

cheers!

-- G --
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Bernd Vogelgesang]

Am 22.10.2015, 16:14 Uhr, schrieb Victor Olaya :


shp is the default format for temp files, and as such it is used in
the modeler for intermediate results. That is, however, very easy to
change, so if we agree on any other format, it can be changed and set
as the default for Processing


+1000 Yes, please, get rid of this pain in the a.. as soon as possible and  
use SpatiaLite e.g..
Creating these files might take considerably longer and they consume more  
space, but compared to the time I spend refactoring column names all the  
time and finding workarounds, this is nothing.

It's also one of the reasons why I do my heavy lifting work in R nowadays.

Cheers
Bernd


2015-10-22 16:08 GMT+02:00 Bernhard Ströbl :

Hi Andreas,

AFAIK processing uses shp for in between results in a model, too. So if  
you

build a model you loose your fields, no matter which output format you
choose. IMHO it should therefore be considered to change processing's  
data

source. Maybe SpatialLite is an option? Users can still export to shp if
needed.

Bernhard



Am 22.10.2015 um 15:48 schrieb Andreas Neumann:


Bernhard,

I agree that these tools should be QGIS algorithms - maybe even ported
from another GIS (if feasible).

One major annoyance with fTools and some processing algorithms is that
reliance on ESRI shapefile as output. That way I always loose my  
columns
- or they are truncated to be meaningless. For that reason I often  
don't

use QGIS, but FME for analysis. Or try to do the processing with SQL in
Postgis.

If we renovate these algorithms in the vector menu we'd have to make
sure that the output could be any chosen vector format, or at least
supports geopackage/spatialite/postgres as output.

Andreas

On 22.10.2015 15:26, Bernhard Ströbl wrote:


IMHO all ftools should be replaced by native QGIS processing
algorithms. For "simple" users it may be not feasible (or not allowed)
to install additional software. Therefore native QGIS algorithms
should be improved to be fast, robust and offering choices the users
need (see recent discussion on dissolve)

Bernhard

Am 22.10.2015 um 15:13 schrieb Andreas Neumann:


If we'd replace the ftools (vector and raster menu) through  
processing.

What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga  
works
best, whereas for algorithm B GRASS is better and for C GDAL or the  
QGIS

builtin in mechanism.

Andreas

On 22.10.2015 14:59, Victor Olaya wrote:


  * Why is the processing toolbox not an option for your users? (Do
you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)



If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code.  
As

I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?

And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just  
a
copy of the ftools ones), and have them exposed through the menus,  
as

explained above.

Regards




__ Information from ESET Mail Security, version of virus  
signature

database 12448 (20151022) __

The message was checked by ESET Mail Security.
http://www.eset.com


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer



--
Bernd Vogelgesang
Siedlerstraße 2
91083 Baiersdorf/Igelsdorf
Tel: 09133-825374
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Andreas Neumann]

If we'd replace the ftools (vector and raster menu) through processing. 
What would replace them? Would we do research and see which of the 
alternative processing providers provides best speed/reliability - if 
there are multiple versions? It may be, that for Algorithm A, Saga works 
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS 
builtin in mechanism.


Andreas

On 22.10.2015 14:59, Victor Olaya wrote:

  * Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)


If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?

And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.

Regards
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Mathieu Pellerin]

QGIS 3.0 seems to be a good moment/opportunity to terminate ftools and
replace the menu items to be shortcuts to processing algorithms.

M
On 22 Oct 2015 19:59, "Victor Olaya"  wrote:

> >
> >  * Why is the processing toolbox not an option for your users? (Do you
> > require shortcuts (menu entries) to the tools? Is it the parameter
> > interface which is more fine-tuned? Missing functionality?)
>
>
> If what you need is to have the tools in a menu, we have been
> discussing that before, and the idea is to mantain the same menu
> structure if the users wants that, but have the algorithms based on
> Processing to avoid redundant (and more difficult to mantain) code. As
> I said, this has already been discussed, but it is not yet
> implemented. Maybe your funding could help us do it?
>
> And if algorithms are not performant enough, it's clearly better to
> put your effort in the Processing ones (which, in general, are just a
> copy of the ftools ones), and have them exposed through the menus, as
> explained above.
>
> Regards
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Matthias Kuhn]

/me adding "icon" to the list of actions

Therefore the plan of action should be:

 * Add icon support to processing
 * Create menu entries for often used (internal) processing algorithms
 * Remove ftools algorithms
 * Identify slow algorithms
 * Improve performance

Cheers
Matthias

On 10/22/2015 03:29 PM, Mathieu Pellerin wrote:
>
> Yep. I think the user-friendliness of ftools are that a/ algorithms
> are offered in a simple and accessible menu items, and b/ the menu
> item icons _greatly_ ease the understanding of what algorithms do.
>
> Both can be added to processing.
>
> On 22 Oct 2015 20:26, "Matthias Kuhn"  > wrote:
>
> Hi Giovanni,
>
>
> On 10/22/2015 02:38 PM, Giovanni Manghi wrote:
> > This users are
> > usually not able to handle (or willingly to) the possible
> problems of
> > using a toolbox based on other software.
>
> There are a number of internal algorithms available in processing
> (i.e.
> no external dependencies).
> Unfortunately they are duplicates of ftools algorithms. But they have
> the advantage that they can be used in models.
> So if something is going to be improved it should be those - or we end
> up with a situation where we have fast algorithms in the menus but the
> same algorithm slower in the toolbox.
>
> Therefore the plan of action should be:
>
>  * Create menu entries for often used (internal) processing algorithms
>  * Remove ftools algorithms
>  * Identify slow algorithms
>  * Improve performance
>
> This will solve João's problems in the most sustainable and effective
> way possible.
>
> Chers
> Matthias
>
>
>
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org 
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Bernhard Ströbl]

IMHO all ftools should be replaced by native QGIS processing algorithms. 
For "simple" users it may be not feasible (or not allowed) to install 
additional software. Therefore native QGIS algorithms should be improved 
to be fast, robust and offering choices the users need (see recent 
discussion on dissolve)


Bernhard

Am 22.10.2015 um 15:13 schrieb Andreas Neumann:

If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.

Andreas

On 22.10.2015 14:59, Victor Olaya wrote:

  * Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)


If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?

And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.

Regards
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer




__ Information from ESET Mail Security, version of virus signature 
database 12448 (20151022) __

The message was checked by ESET Mail Security.
http://www.eset.com


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Giovanni Manghi]

Hi Matthias,

the issues are the following:


there is whole universe of very basic users that need to do classic
geoprocessing operations (clips, unions, etc.) sometimes also with
large/dense geometries, possibly with geometry errors. This users are
usually not able to handle (or willingly to) the possible problems of
using a toolbox based on other software. For them the qgis
geoprocessing tools would be enough (now, as a separet menu as now or
only in processing this is not an issue).

problem is that when using large/dense inputs this qgis tools are
usually very slow, especially when compared to the proprietary
software many are trying to leave. In other worst cases the results
are wrong, but not in a way that is openly wrong, making the usability
of such tools even more difficult.

so said that I understand perfectly what João says, and I would say
that is time to the action, especially considering that there is an
offer on the table.

cheers

-- g --



>
>  * Why is the processing toolbox not an option for your users? (Do you
> require shortcuts (menu entries) to the tools? Is it the parameter
> interface which is more fine-tuned? Missing functionality?)
>  * Are the processing toolbox tools enough performant for your needs? If
> not, which ones require attention? It would be good to have a list.
>
> If the first one is a problem, I think it would be an approach to let
> system administrators define "favorite algorithms" (or similar) which
> would then be offered as a shortcut in the menu.
>
> If the second one applies, then we already have proven the point that
> maintaining the same algorithm twice is silly and the effort should
> rather be put into unifying these tools instead of rewriting duplicate
> code. I will probably soon be working on a test suite for processing
> which should ensure long-term quality assurance for the algorithms
> there. Another advantage of consolidating the two code bases.
>
> I hope that it's possible for your company to consider different
> approaches for this problem to get to a long-term enhancement.
>
> Best regards,
> Matthias
>
>
> -- next part --
> A non-text attachment was scrubbed...
> Name: signature.asc
> Type: application/pgp-signature
> Size: 819 bytes
> Desc: OpenPGP digital signature
> URL: 
> 
>
> --
>
> Message: 5
> Date: Thu, 22 Oct 2015 12:30:26 +
> From: Alexandre Neto 
> To: Lauri Kajan , qgis-developer
> 
> Subject: Re: [Qgis-developer] encoding in custom python functions
> Message-ID:
> 
> Content-Type: text/plain; charset="utf-8"
>
> Ah... I think I now understand what you are saying. You are saying that
> expression files are always saved in your system default encoding system.
>
> Sorry for the noise.
>
> Lauri Kajan  escreveu no dia qui, 22/10/2015 às
> 13:25:
>
>> Hi Alexandre,
>>
>> The u'ä' works if I tell the file encoding using your snippet
>>
>> # -*- coding: utf-8 -*-
>>
>> But this works only in qgis and is not a correct way.
>> I cannot then debug my function in any IDE cause the file is encoded
>> differently than it says.
>>
>> -Lauri
>>
>> On Thu, Oct 22, 2015 at 1:37 PM, Alexandre Neto 
>> wrote:
>>
>>> Have you tried to put the following line in the top of the file?
>>>
>>> # -*- coding: utf-8 -*-
>>>
>>>
>>>
>>> Lauri Kajan  escreveu no dia qui, 22/10/2015 às
>>> 08:42:
>>>
 Hi all,

 I'm trying to figure out some encoding problems when using custom python
 expression functions.

 I'm using python functions for defining some label texts. I need use
 some special characters like ä and ö.

 When a new file is created in function editor the file is encoded in my
 system encoding cp1252. At least the ä is coded to 0xe4 if I check that
 with hex-editor.
 Let the custom function just be
 @qgsfunction(args='auto', group='Custom')
 def encodingtest(feature, parent):
 return 'ä'

 This displays on a map like 'ä'
 It seems that the file is converted to utf-8 before execution and the ä
 is then coded to 0xc3 0xa4. Now QGIS still tries to interpret this as
 cp1252 and then displays this as 'ä'.

 It doesn't help to define the ä as unicode like u'ä' because QGIS still
 sees the ä as 0xc3 0xa4 and think that it is in cp1252.

 The only way to resolve this in the code at least that I have found is
 to use
 return 'ä'.decode('utf-8')
 This works on QGIS but the file can't be executed elsewhere (IDE) since
 the file is truly in cp1252 and 0xe4 can't be decoded using utf-8.


 Does anyone else confirm this and have I understood right the 

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Alexander Bruy]

Some time ago Carson ported at least some fTools
algorithms to QGIS analysis library. Maybe we can
revise this code, update it and then reuse in Processing
and 3rd party plugins?

Core implementation should be faster than Python code,
also maybe we can adopt multithreading approach for
some alforothms.


2015-10-22 14:30 GMT+03:00 João Gaspar :
> Dear QGIS Developers,
>
> I'm writing on behalf of my employer, a company that has mixed,
> proprietary and open source (especially QGIS of course) gis
> infrastructure, with qgis being adopted each day a little more.
> Actually in our migration path we have found a road blocker: the very
> bad performances (and sometimes results) of most of geoprocessing
> tools in the vector menu. Yes I know that there are alternatives in
> the Processing toolbox, but that is not an option for most of our very
> basic users: they need to do simple operations (clips, unions,
> intersections, etc.) with usually very large/detailed vectors (usually
> shapefiles). In this scenario QGIS usually chokes or is at best very
> slow (compared to other gis packages, the ones we are trying to get
> rid of).
>
> What can be done to improve this tools? Rewriting the tools to improve
> efficiency and accuracy?
>
> We available some funds: this is the first time I convince people
> above me to put some money in QGIS development, and a decision would
> need to me made until the end of November tops, otherwise money will
> be used otherwise.
>
> Please leave advice and feedback.
>
>
> Regards
> João Gaspar
>
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer



-- 
Alexander Bruy
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Matthias Kuhn]

Hi João,

Nice that there is interest in improving the analysis part of QGIS.

There was recently a similar discussion with Bernhard Ströbl who also
put some effort into improving the dissolve tool (although it turned out
to be complex and will probably only be merged in 2.14).

In the discussion there, the need for coexistence of these two suits
offering the same tools was questioned. In particular, maintaining twice
the same tool in QGIS is an extra effort in maintenance and there is no
advantage visible.

It would be nice to plan how to resolv this situation, in particular
could you answer the following questions:

 * Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
 * Are the processing toolbox tools enough performant for your needs? If
not, which ones require attention? It would be good to have a list.

If the first one is a problem, I think it would be an approach to let
system administrators define "favorite algorithms" (or similar) which
would then be offered as a shortcut in the menu.

If the second one applies, then we already have proven the point that
maintaining the same algorithm twice is silly and the effort should
rather be put into unifying these tools instead of rewriting duplicate
code. I will probably soon be working on a test suite for processing
which should ensure long-term quality assurance for the algorithms
there. Another advantage of consolidating the two code bases.

I hope that it's possible for your company to consider different
approaches for this problem to get to a long-term enhancement.

Best regards,
Matthias




signature.asc
Description: OpenPGP digital signature
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Victor Olaya]

>
>  * Why is the processing toolbox not an option for your users? (Do you
> require shortcuts (menu entries) to the tools? Is it the parameter
> interface which is more fine-tuned? Missing functionality?)


If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?

And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.

Regards
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Mathieu Pellerin]

Yep. I think the user-friendliness of ftools are that a/ algorithms are
offered in a simple and accessible menu items, and b/ the menu item icons
_greatly_ ease the understanding of what algorithms do.

Both can be added to processing.
On 22 Oct 2015 20:26, "Matthias Kuhn"  wrote:

> Hi Giovanni,
>
>
> On 10/22/2015 02:38 PM, Giovanni Manghi wrote:
> > This users are
> > usually not able to handle (or willingly to) the possible problems of
> > using a toolbox based on other software.
>
> There are a number of internal algorithms available in processing (i.e.
> no external dependencies).
> Unfortunately they are duplicates of ftools algorithms. But they have
> the advantage that they can be used in models.
> So if something is going to be improved it should be those - or we end
> up with a situation where we have fast algorithms in the menus but the
> same algorithm slower in the toolbox.
>
> Therefore the plan of action should be:
>
>  * Create menu entries for often used (internal) processing algorithms
>  * Remove ftools algorithms
>  * Identify slow algorithms
>  * Improve performance
>
> This will solve João's problems in the most sustainable and effective
> way possible.
>
> Chers
> Matthias
>
>
>
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Victor Olaya]

>>
>> Yep. I think the user-friendliness of ftools are that a/ algorithms
>> are offered in a simple and accessible menu items, and b/ the menu
>> item icons _greatly_ ease the understanding of what algorithms do.
>>


Icon support is already in Processing. The first versions actually
used the same ftools icons, but I removed that because there were new
algorithms being added, and it was quite messy to see some algorithms
with icons and others with the default icon. Instead, I started using
the same icon for all algorithms for a given provider, for the sake of
homogeneity.

However, the mechanism that I implemented for putting processing
algorithms in a menu allows to set an icon, so the current ftools menu
can be perfectly replicated, including icons

I copy here an example that  I added in another thread, about how to
replicate the Raster/Conversion  menu from Processing


import os
from GdalTools import GdalTools
from processing.gui.utils import addAlgorithmEntry
from PyQt4.QtGui import *

def addRasterAlgorithm(name, group, iconName):
icon = QIcon(os.path.join(os.path.dirname(GdalTools.__file__),
"icons", iconName + ".png"))
addAlgorithmEntry(name, "Raster", group, icon = icon)

addRasterAlgorithm("gdalogr:rasterize", "Conversion", "rasterize")
addRasterAlgorithm("gdalogr:translate", "Conversion", "translate")
addRasterAlgorithm("gdalogr:pcttorgb", "Conversion", "24-to-8-bits")
addRasterAlgorithm("gdalogr:rgbtopct", "Conversion", "8-to-24-bits")
addRasterAlgorithm("gdalogr:polygonize", "Conversion", "polygonize")

The addAlgorithmEntry is a part of Processing since quite a few
commits ago, and it allows to put an algorithm "outside" of the
toolbox, in a menu entry.

So the code is basically there, it is just a matter of finding out
which algorithms are missing and deciding where to call this methods
from. Then I guess we could use that funding to improve algorithms and
make them more robust and fast


Cheers




>> Both can be added to processing.
>>
>> On 22 Oct 2015 20:26, "Matthias Kuhn" > > wrote:
>>
>> Hi Giovanni,
>>
>>
>> On 10/22/2015 02:38 PM, Giovanni Manghi wrote:
>> > This users are
>> > usually not able to handle (or willingly to) the possible
>> problems of
>> > using a toolbox based on other software.
>>
>> There are a number of internal algorithms available in processing
>> (i.e.
>> no external dependencies).
>> Unfortunately they are duplicates of ftools algorithms. But they have
>> the advantage that they can be used in models.
>> So if something is going to be improved it should be those - or we end
>> up with a situation where we have fast algorithms in the menus but the
>> same algorithm slower in the toolbox.
>>
>> Therefore the plan of action should be:
>>
>>  * Create menu entries for often used (internal) processing algorithms
>>  * Remove ftools algorithms
>>  * Identify slow algorithms
>>  * Improve performance
>>
>> This will solve João's problems in the most sustainable and effective
>> way possible.
>>
>> Chers
>> Matthias
>>
>>
>>
>> ___
>> Qgis-developer mailing list
>> Qgis-developer@lists.osgeo.org 
>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Giovanni Manghi]

On Thu, Oct 22, 2015 at 3:27 PM, Victor Olaya  wrote:
>> The issue here is about the lousy performances (and even wrong
>> results) of the native qgis vector geoprocessing tools. I think we all
>> agree that qgis should have a few (at least the most commonly used)
>> ones, without depending on any 3rd party software.
>
> I fully agree on that
>
> Do we have any benchmark or list of those algorithms with very bad
> performance or that give wrong results? I don't think that all of them
> behave so badly, so it would be good to have more detail about where
> we should be focusing our efforts in this


great, so I would suggest try underestand why the tools here

https://github.com/qgis/QGIS/blob/master/python/plugins/fTools/tools/doGeoprocessing.py

are so bad with such "problematic" datasets (that I can easily
provide). If is possible to do something
then take João company offer use it to improve the situation, then we
can decide with calm were to put the
shortcuts :)

cheers!

-- G --
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Improving ftools geoprocessing tools

[Matthias Kuhn]

Giovanni,

Would (some of) these datasets be suitable to include in unit tests for
processing?
It would be very interesting to have these available for this purpose.

Matthias

On 10/22/2015 05:07 PM, Giovanni Manghi wrote:
> On Thu, Oct 22, 2015 at 3:27 PM, Victor Olaya  wrote:
>>> The issue here is about the lousy performances (and even wrong
>>> results) of the native qgis vector geoprocessing tools. I think we all
>>> agree that qgis should have a few (at least the most commonly used)
>>> ones, without depending on any 3rd party software.
>> I fully agree on that
>>
>> Do we have any benchmark or list of those algorithms with very bad
>> performance or that give wrong results? I don't think that all of them
>> behave so badly, so it would be good to have more detail about where
>> we should be focusing our efforts in this
>
> great, so I would suggest try underestand why the tools here
>
> https://github.com/qgis/QGIS/blob/master/python/plugins/fTools/tools/doGeoprocessing.py
>
> are so bad with such "problematic" datasets (that I can easily
> provide). If is possible to do something
> then take João company offer use it to improve the situation, then we
> can decide with calm were to put the
> shortcuts :)
>
> cheers!
>
> -- G --
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer




signature.asc
Description: OpenPGP digital signature
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer