Re: [GRASS-dev] Adding an expert mode to the parser

[Vaclav Petras]

On Wed, Sep 28, 2016 at 4:35 PM, Markus Metz 
wrote:

> On Tue, Sep 27, 2016 at 5:55 PM, Vaclav Petras 
> wrote:
> >
> > On Mon, Sep 26, 2016 at 5:50 PM, Vaclav Petras 
> wrote:
> >>
> >> If the --help is just for scanning and the issue is that it simply too
> >> long, hiding some parameters is not the only option we have. For many
> (and
> >> more in the future hopefully) parameters we have (short) label and
> (longer)
> >> description. --help prints both (if both are present, that's at least 2
> >> lines per parameter). Additionally, if the option has predefined values
> >> which have descriptions, there is one line for each of those. So, the
> >> question is would it be helpful (at least as a first step) if --help
> would
> >> print less information for each parameter but still provided all
> parameters?
> >
> >
> > Additionally, the label and description is also for the module itself, so
> > when we have label, we don't need to show description, that's -1 line for
> > r.slope.aspect. Then we can remove keywords, that's -3 lines. Then we
> have
> > the standard/long flags. (These might be similar to the general options
> from
> > `ogr2ogr --help-general` which Madi posted.) They are the same all the
> time,
> > except for --o, so that's -4 or -3 for the old ones (depending on module)
> > and -2 for the recently added flags (--qq --ui). They still can be in
> > `Usage:` section or they may not. There is also an empty line at the
> > beginning, we can remove it as well because don't go for beauty and the
> > `Description:` title is also not necessary, so we could remove that as
> well,
> > that's -2 lines. That would be 12 lines total. (I have 55 lines in the
> > terminal windows I'm using right now, so 12 seems to be quite a lot).
>
> +1


Done in r69602. I limited the changes to the usage() function (G_usage()
backend). The behavior is driven by 3 variables which can become function
parameters in the future. For now, full description can be found in the
using man or g.manual commands. Some --full-help can be implemented if
desired. Related to that, I did not preserved the functionality of
--help-text, this can be fixed as well.

Next step would be ensuring that each module has its label (or description)
short enough that it fits into one line (72 characters?). Similarly, each
option and flag should have label (or description) which fits on one line
considering that we need some space for flag, option name and horizontal
spacing.

Anyway, here is the state before:

> r.slope.aspect --help 2>&1 | wc
 48 2552137
> v.in.lidar --help 2>&1 | wc
 72 4623634

And after:

> r.slope.aspect --help 2>&1 | wc
 32 1941620
> v.in.lidar --help 2>&1 | wc
 43 2422002

Please test,
Vaclav


https://trac.osgeo.org/grass/changeset/69602
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Sören Gebbert]

Hi,

[snip]
> >
> > As an example, when aiming at processing all Sentinel-2 tiles
> > globally, we speak about currently 73000 scenes * up-to-16 tiles along
> > with global data, analysis on top of other global data is more complex
> > when doing each job in its own mapset and reintegrate it in a single
> > target mapset as if able to process then in parallel in one mapset by
> > simply specifying the respective region to the command of interest.
> > Yes, different from the current paradigm and not for G7.
>
> from our common experience, I would say that creating separate mapsets
> is a safety feature. If anything goes wrong with that particular
> processing chain, cleaning up is easy, simply delete this particular
> mapset and run the job again, if possible on a different host/node
> (assuming that failed jobs are logged). Anyway, I would be surprised
> if the overhead of opening a separate mapset is measurable when
> processing all Sentinel-2 tiles globally. Reintegration into a single
> target mapset could cause problems with regard to IO saturation, but
> in a HPC environment temporary data always need to be copied to a
> final target location at some stage. The HPC system you are using now
> is most probably quite different from the one we used previously, so
> this is a lot of guessing, particularly about the storage location of
> temporary data (no matter if it is the same mapset or a separate
> mapset).
>

Imagine you have a tool that is able to distribute the processing of a
large time series of satellite images across a cluster. Each node in the
cluster should process a stack of r.mapcalc, r.series or r.neighbors
commands in a local temporary mapset, that gets later merged into a single
one. A single stack of commands may have hundreds of jobs that will run in
a single temporary mapset. In this scenario you need separate region
settings for each command in the stack, because of the different spatial
extents of the satellite images. The size of the stack depends on the size
of the time series (number of maps) and the number of available nodes.

Having region setting options in the parser will make the implementation of
such an approach much easier. Commands like t.rast.algebra and
t.rast.neighbors will directly benefit from a region parser option,
allowing the parallel processing of satellite image time series on a
multi-core machine.

Best regards
Soeren


> To be continued in a GRASS+HPC thread?
>
> Markus M
>
> >
> > But my original comment was targeted at the increasing number of
> > module parameters and how to handle that (with some new HPC related
> > idea as an example).
> >
> > I'm fine to archive this question for now, it will likely come up
> > again in the future.
> >
> > markusN
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Markus Metz]

On Sun, Sep 25, 2016 at 9:49 PM, Markus Neteler  wrote:
> On Fri, Sep 23, 2016 at 11:30 PM, Markus Metz
>  wrote:
>> On Fri, Sep 23, 2016 at 11:22 PM, Markus Neteler  wrote:
>>> On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
>>>  wrote:
 On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler  wrote:
>>> ...
 Your motivation is to provide a specialized CLI interface for HPC
 processing?
>>>
>>> No, not the case.
>>>
 We used GRASS with HPC processing for years and the
 problems we faced were causes by the HPC hardware and software
 infrastructure, not by GRASS. What exactly is the problem with using
 GRASS and HPC processing?
>>>
>>> There is no problem. There is just the issue that with an increasing
>>> amount of additions (e.g. maybe the need to provide region/resolution
>>> to individual modules for independent parallel processing without the
>>> overhead of always opening a new mapset)
>>
>> Getting closer it seems. Can you quantify "the overhead of always
>> opening a new mapset"?
>
> As an example, when aiming at processing all Sentinel-2 tiles
> globally, we speak about currently 73000 scenes * up-to-16 tiles along
> with global data, analysis on top of other global data is more complex
> when doing each job in its own mapset and reintegrate it in a single
> target mapset as if able to process then in parallel in one mapset by
> simply specifying the respective region to the command of interest.
> Yes, different from the current paradigm and not for G7.

from our common experience, I would say that creating separate mapsets
is a safety feature. If anything goes wrong with that particular
processing chain, cleaning up is easy, simply delete this particular
mapset and run the job again, if possible on a different host/node
(assuming that failed jobs are logged). Anyway, I would be surprised
if the overhead of opening a separate mapset is measurable when
processing all Sentinel-2 tiles globally. Reintegration into a single
target mapset could cause problems with regard to IO saturation, but
in a HPC environment temporary data always need to be copied to a
final target location at some stage. The HPC system you are using now
is most probably quite different from the one we used previously, so
this is a lot of guessing, particularly about the storage location of
temporary data (no matter if it is the same mapset or a separate
mapset).

To be continued in a GRASS+HPC thread?

Markus M

>
> But my original comment was targeted at the increasing number of
> module parameters and how to handle that (with some new HPC related
> idea as an example).
>
> I'm fine to archive this question for now, it will likely come up
> again in the future.
>
> markusN
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Markus Metz]

On Tue, Sep 27, 2016 at 5:55 PM, Vaclav Petras  wrote:
>
> On Mon, Sep 26, 2016 at 5:50 PM, Vaclav Petras  wrote:
>>
>> If the --help is just for scanning and the issue is that it simply too
>> long, hiding some parameters is not the only option we have. For many (and
>> more in the future hopefully) parameters we have (short) label and (longer)
>> description. --help prints both (if both are present, that's at least 2
>> lines per parameter). Additionally, if the option has predefined values
>> which have descriptions, there is one line for each of those. So, the
>> question is would it be helpful (at least as a first step) if --help would
>> print less information for each parameter but still provided all parameters?
>
>
> Additionally, the label and description is also for the module itself, so
> when we have label, we don't need to show description, that's -1 line for
> r.slope.aspect. Then we can remove keywords, that's -3 lines. Then we have
> the standard/long flags. (These might be similar to the general options from
> `ogr2ogr --help-general` which Madi posted.) They are the same all the time,
> except for --o, so that's -4 or -3 for the old ones (depending on module)
> and -2 for the recently added flags (--qq --ui). They still can be in
> `Usage:` section or they may not. There is also an empty line at the
> beginning, we can remove it as well because don't go for beauty and the
> `Description:` title is also not necessary, so we could remove that as well,
> that's -2 lines. That would be 12 lines total. (I have 55 lines in the
> terminal windows I'm using right now, so 12 seems to be quite a lot).

+1

>
> In GUI or even in the manual, we could put the standard/long flags to a
> separate section as well.
>
>> r.slope.aspect --help
>
> Description:
>  Generates raster maps of slope, aspect, curvatures and partial derivatives
> from an elevation raster map.
>  Aspect is calculated counterclockwise from east.
>
> Keywords:
>  raster, terrain, aspect, slope, curvature
>
> Usage:
>  r.slope.aspect [-a] elevation=name [slope=name] [aspect=name]
>[format=string] [precision=string] [pcurvature=name] [tcurvature=name]
>[dx=name] [dy=name] [dxx=name] [dyy=name] [dxy=name] [zscale=value]
>[min_slope=value] [--overwrite] [--help] [--verbose] [--quiet] [--ui]
>
> Flags:
>   -a   Do not align the current region to the raster elevation map
>  --o   Allow output files to overwrite existing files
>  --h   Print usage summary
>  --v   Verbose module output
>  --q   Quiet module output
>  --qq  Super quiet module output
>  --ui  Force launching GUI dialog
>
>
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Vaclav Petras]

On Tue, Sep 27, 2016 at 5:38 PM, Sören Gebbert  wrote:

> However, the correct use of GRASS_REGION is still unclear to me.


There is a function in grass.script [1] which can generate the right
string. It calls g.region with -u, so you can modify the region with
g.region, but without actually changing it. Then you just set the
environmental variable for the process. If you anyway control the
environment for the subprocess, I think, there is not much difference
between GRASS_REGION and WIND_OVERRIDE expect for a file and length of
command line.

If the g.region code is moved to library (there might be some modules which
would benefit from it) and parser accepts standard options with values,
then parser could just set this variable in the same way it sets
GRASS_OVERWRITE (to keep parser and rest of lib/gis separate - although
this might not be an option anyway if it needs to use the g.region
functions).

[1] https://grass.osgeo.org/grass70/manuals/libpython/
script.html#script.core.region_env
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Vaclav Petras]

On Tue, Sep 27, 2016 at 5:01 PM, Sören Gebbert  wrote:

> From my understanding, the GRASS_REGION environmental variable is
> supported by G_get_window() [1]. However, most raster modules
> uses  Rast_window_rows() and Rast_window_cols() to get the current
> computational region rows and cols. Hence, setting GRASS_REGION will not
> affect most of the raster modules.
>

This would be a buggy behavior but I think it is not the case. All 2D
raster window functions use Rast__init_window() [2] which calls
G__init_window() [3] which uses G_get_window() [4]. I'm not sure about 3D
rasters but I think WIND_OVERRIDE worked for me. Display modules seems to
accept GRASS_REGION because that's actually how it is used in code. Vector
modules probably just use the G_ function.

Vaclav

[2]
https://trac.osgeo.org/grass/browser/grass/trunk/lib/raster/window.c?rev=69586
[3]
https://trac.osgeo.org/grass/browser/grass/trunk/lib/raster/set_window.c?rev=69586#L24
[4]
https://trac.osgeo.org/grass/browser/grass/trunk/lib/gis/set_window.c?rev=69586#L70



>
> Best regards
> Soeren
>
> [1] https://trac.osgeo.org/grass/browser/grass/trunk/lib/
> gis/get_window.c?rev=69586#L30
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Sören Gebbert]

Hi,

> On Sun, Sep 25, 2016 at 5:40 PM, Sören Gebbert <
>> soerengebb...@googlemail.com> wrote:
>>
>>> >r.mapclac --raster-region= --north= --south= --west= --east= --res=
 >--ewres= --nsres= --vect-region --raster-align= ...

>>>
>> I like this in the sense that if the region setting for the module is
>> something which should be managed by these extra options, then they should
>> be managed in parser, rather than introduced one by one for individual
>> modules.
>>
>> But how is this different from using GRASS_REGION? Convenience of 
>> --raster-region=?
>> Better syntax than environmental variable?
>>
>
> From my understanding, the GRASS_REGION environmental variable is
> supported by G_get_window() [1]. However, most raster modules
> uses  Rast_window_rows() and Rast_window_cols() to get the current
> computational region rows and cols. Hence, setting GRASS_REGION will not
> affect most of the raster modules.
>

While inspecting the raster and libgis code i realized that the code that
initializes the input and output raster window eventually calls
G_get_window(). Hence, the GRASS_REGION environmental variable will have an
effect on  Rast_window_rows() and Rast_window_cols(). However, the correct
use of GRASS_REGION is still unclear to me.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Sören Gebbert]

Hi,

2016-09-26 4:16 GMT+02:00 Vaclav Petras :

>
> On Sun, Sep 25, 2016 at 5:40 PM, Sören Gebbert <
> soerengebb...@googlemail.com> wrote:
>
>> >r.mapclac --raster-region= --north= --south= --west= --east= --res=
>>> >--ewres= --nsres= --vect-region --raster-align= ...
>>>
>>
> I like this in the sense that if the region setting for the module is
> something which should be managed by these extra options, then they should
> be managed in parser, rather than introduced one by one for individual
> modules.
>
> But how is this different from using GRASS_REGION? Convenience of 
> --raster-region=?
> Better syntax than environmental variable?
>

>From my understanding, the GRASS_REGION environmental variable is supported
by G_get_window() [1]. However, most raster modules
uses  Rast_window_rows() and Rast_window_cols() to get the current
computational region rows and cols. Hence, setting GRASS_REGION will not
affect most of the raster modules.

Best regards
Soeren

[1]
https://trac.osgeo.org/grass/browser/grass/trunk/lib/gis/get_window.c?rev=69586#L30
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Vaclav Petras]

On Mon, Sep 26, 2016 at 5:50 PM, Vaclav Petras  wrote:

> If the --help is just for scanning and the issue is that it simply too
> long, hiding some parameters is not the only option we have. For many (and
> more in the future hopefully) parameters we have (short) label and (longer)
> description. --help prints both (if both are present, that's at least 2
> lines per parameter). Additionally, if the option has predefined values
> which have descriptions, there is one line for each of those. So, the
> question is would it be helpful (at least as a first step) if --help would
> print less information for each parameter but still provided all parameters?


Additionally, the label and description is also for the module itself, so
when we have label, we don't need to show description, that's -1 line for
r.slope.aspect. Then we can remove keywords, that's -3 lines. Then we have
the standard/long flags. (These might be similar to the general options
from `ogr2ogr --help-general` which Madi posted.) They are the same all the
time, except for --o, so that's -4 or -3 for the old ones (depending on
module) and -2 for the recently added flags (--qq --ui). They still can be
in `Usage:` section or they may not. There is also an empty line at the
beginning, we can remove it as well because don't go for beauty and the
`Description:` title is also not necessary, so we could remove that as
well, that's -2 lines. That would be 12 lines total. (I have 55 lines in
the terminal windows I'm using right now, so 12 seems to be quite a lot).

In GUI or even in the manual, we could put the standard/long flags to a
separate section as well.

> r.slope.aspect --help

Description:
 Generates raster maps of slope, aspect, curvatures and partial derivatives
from an elevation raster map.
 Aspect is calculated counterclockwise from east.

Keywords:
 raster, terrain, aspect, slope, curvature

Usage:
 r.slope.aspect [-a] elevation=name [slope=name] [aspect=name]
   [format=string] [precision=string] [pcurvature=name] [tcurvature=name]
   [dx=name] [dy=name] [dxx=name] [dyy=name] [dxy=name] [zscale=value]
   [min_slope=value] [--overwrite] [--help] [--verbose] [--quiet] [--ui]

Flags:
  -a   Do not align the current region to the raster elevation map
 --o   Allow output files to overwrite existing files
 --h   Print usage summary
 --v   Verbose module output
 --q   Quiet module output
 --qq  Super quiet module output
 --ui  Force launching GUI dialog
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Margherita Di Leo]

On Tue, Sep 27, 2016 at 9:15 AM, Margherita Di Leo 
wrote:

> Hi,
>
> On Tue, Sep 27, 2016 at 7:13 AM, Moritz Lennert <
> mlenn...@club.worldonline.be> wrote:
>
>> On 26/09/16 23:50, Vaclav Petras wrote:
>>
>>>
>>> On Mon, Sep 26, 2016 at 5:24 PM, Veronica Andreo >> > wrote:
>>>
>>> I'm with MaDi in that if I see a very long list of flags and
>>> parameters in the terminal when using --h, i just go to the
>>> manual... I just use --h in CLI for a quick recalling of
>>> flags/options... A reduced list of most commonly used flags would be
>>> nice, but still keep the possibility to get the advanced
>>> flags/parameters as well, if the user needs more info...
>>>
>>>
>>> If the --help is just for scanning and the issue is that it simply too
>>> long, hiding some parameters is not the only option we have. For many
>>> (and more in the future hopefully) parameters we have (short) label and
>>> (longer) description. --help prints both (if both are present, that's at
>>> least 2 lines per parameter). Additionally, if the option has predefined
>>> values which have descriptions, there is one line for each of those. So,
>>> the question is would it be helpful (at least as a first step) if --help
>>> would print less information for each parameter but still provided all
>>> parameters?
>>>
>>
>> In line with your other mail where you caution about "hiding" useful
>> information, how about not changing the --help output, but rather adding a
>> "--simple-help" or somthing like this which would output a simplified help.
>
>
> +1.
>


For inspiration, looking at gdal: http://www.gdal.org/ogr2ogr.html

$ ogr2ogr --help-general
Generic GDAL/OGR utility command options:
[...]

$ ogr2ogr --help
Usage: ogr2ogr [--help-general] [-skipfailures] [-append] [-update]
   [-select field_list] [-where restricted_where]
   [-progress] [-sql ] [-dialect dialect]
   [-preserve_fid] [-fid FID]
   [-spat xmin ymin xmax ymax] [-geomfield field]
   [-a_srs srs_def] [-t_srs srs_def] [-s_srs srs_def]
   [-f format_name] [-overwrite] [[-dsco NAME=VALUE] ...]
   dst_datasource_name src_datasource_name
   [-lco NAME=VALUE] [-nln name] [-nlt type] [-dim
2|3|layer_dim] [layer [layer ...]]

Advanced options :
[...]
Note: ogr2ogr --long-usage for full help.

$ ogr2ogr --long-usage
Usage: ogr2ogr [--help-general] [-skipfailures] [-append] [-update]
  [...]

Advanced options :
   [...]

 -f format_name: output file format name, possible values are:
  [...]



-- 
Margherita Di Leo
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Newcomb, Doug]

+1

On Mon, Sep 26, 2016 at 6:11 PM, Vaclav Petras  wrote:

>
> On Fri, Sep 23, 2016 at 5:22 PM, Markus Neteler  wrote:
>
>> * various commands: "Use the low-memory version" - also more advanced
>
>
> I guess this is a typical example where we need to be careful. I can
> imagine a situation when a beginner has a lot of data and low-end hardware.
> By hiding an parameter like this, we would make it even more advanced. Now
> you not only need to understand that your hardware is not enough and that
> the algorithm should somehow account for that. You also need to know that
> there might be a hidden option for that and you need to know how to find
> it. Having this in the manual or even a Memory or Speed tab in the GUI,
> gives some hope that even a beginner will be able to use it.
>
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>



-- 
Doug Newcomb
USFWS
Raleigh, NC
919-856-4520 ext. 14 doug_newc...@fws.gov
-
The opinions I express are my own and are not representative of the
official policy of the U.S.Fish and Wildlife Service or Dept. of the
Interior.   Life is too short for undocumented, proprietary data formats.
As a federal employee, my email may be subject to FOIA request.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Rainer M Krug]

Moritz Lennert  writes:

> On 26/09/16 23:50, Vaclav Petras wrote:
>>
>> On Mon, Sep 26, 2016 at 5:24 PM, Veronica Andreo > > wrote:
>>
>> I'm with MaDi in that if I see a very long list of flags and
>> parameters in the terminal when using --h, i just go to the
>> manual... I just use --h in CLI for a quick recalling of
>> flags/options... A reduced list of most commonly used flags would be
>> nice, but still keep the possibility to get the advanced
>> flags/parameters as well, if the user needs more info...
>>
>>
>> If the --help is just for scanning and the issue is that it simply too
>> long, hiding some parameters is not the only option we have. For many
>> (and more in the future hopefully) parameters we have (short) label and
>> (longer) description. --help prints both (if both are present, that's at
>> least 2 lines per parameter). Additionally, if the option has predefined
>> values which have descriptions, there is one line for each of those. So,
>> the question is would it be helpful (at least as a first step) if --help
>> would print less information for each parameter but still provided all
>> parameters?
>
> In line with your other mail where you caution about "hiding" useful
> information, how about not changing the --help output, but rather
> adding a "--simple-help" or somthing like this which would output a
> simplified help. Although:

Everybody is so used to using --help (or -h) that I don't think that an
--simple-help would do any good.

Instead, possibly -hh (as in -vv for more verbose) and mention this in
the first line of the help?

This would open even for -hhh if most advanced options are there.

Concerning --help, this should possibly change to displaying the basic
-h

Rainer

>
>> and manual pages then (a tab or section for advanced
>> flags/parameters)...
>>
>>
>> Considering that we have currently as system of (gui) sections which
>> place the options to individual tabs in GUI, isn't showing the different
>> sections in the manual the right thing to do?
>
> I would prefer this: show everything, but structure it differently,
> i.e. possibly introduce a new parser keyword (advanced: yes) which
> would put the option into a specific section of the manual. IMHO, this
> should be independent of the GUI sections logic as one might to group
> less advanced and advanced options in the same thematic tab...
>
>>  IMHO, a major issue however is the lack of examples for the usage of
>>  more advanced flags or options (or even the required and more common
>>  ones). Take the case of several i.* modules, for example... but maybe
>>  this should go in a separate thread :-)
>>
>>
>> Good point. If you have an explanation and example for a flag, perhaps
>> you can understand it and it is not so advanced at the end.
>
> I think this is actually a major issue: man pages without description,
> notes and example sections are almost useless IMHO. At the foss4g.be
> last week someone presented a simple use of GRASS GIS (to create this
> map [1] for television) and explained how he actually found the GRASS
> GIS manual system extremely well done and useful, because of the
> explanations and examples...
>
> Moritz
>
> [1] http://www.rtbf.be/services/meteo/apere
>
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>

-- 
Rainer M. Krug
email: Rainerkrugsde
PGP: 0x0F52F982


signature.asc
Description: PGP signature
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Margherita Di Leo]

Hi,

On Tue, Sep 27, 2016 at 7:13 AM, Moritz Lennert <
mlenn...@club.worldonline.be> wrote:

> On 26/09/16 23:50, Vaclav Petras wrote:
>
>>
>> On Mon, Sep 26, 2016 at 5:24 PM, Veronica Andreo > > wrote:
>>
>> I'm with MaDi in that if I see a very long list of flags and
>> parameters in the terminal when using --h, i just go to the
>> manual... I just use --h in CLI for a quick recalling of
>> flags/options... A reduced list of most commonly used flags would be
>> nice, but still keep the possibility to get the advanced
>> flags/parameters as well, if the user needs more info...
>>
>>
>> If the --help is just for scanning and the issue is that it simply too
>> long, hiding some parameters is not the only option we have. For many
>> (and more in the future hopefully) parameters we have (short) label and
>> (longer) description. --help prints both (if both are present, that's at
>> least 2 lines per parameter). Additionally, if the option has predefined
>> values which have descriptions, there is one line for each of those. So,
>> the question is would it be helpful (at least as a first step) if --help
>> would print less information for each parameter but still provided all
>> parameters?
>>
>
> In line with your other mail where you caution about "hiding" useful
> information, how about not changing the --help output, but rather adding a
> "--simple-help" or somthing like this which would output a simplified help.


+1.


> Although:
>
> and manual pages then (a tab or section for advanced
>> flags/parameters)...
>>
>>
>> Considering that we have currently as system of (gui) sections which
>> place the options to individual tabs in GUI, isn't showing the different
>> sections in the manual the right thing to do?
>>
>
> I would prefer this: show everything, but structure it differently, i.e.
> possibly introduce a new parser keyword (advanced: yes) which would put the
> option into a specific section of the manual. IMHO, this should be
> independent of the GUI sections logic as one might to group less advanced
> and advanced options in the same thematic tab...


+1 also here :-)

>
>
> IMHO, a major issue however is the lack of examples for the usage
>> of
>> more advanced flags or options (or even the required and more
>> common
>> ones). Take the case of several i.* modules, for example... but
>> maybe
>> this should go in a separate thread :-)
>>
> >
> >
> > Good point. If you have an explanation and example for a flag, perhaps
> > you can understand it and it is not so advanced at the end.
>
> I think this is actually a major issue: man pages without description,
> notes and example sections are almost useless IMHO. At the foss4g.be last
> week someone presented a simple use of GRASS GIS (to create this map [1]
> for television) and explained how he actually found the GRASS GIS manual
> system extremely well done and useful, because of the explanations and
> examples...
>

+1000 :-)
So this afternoon at the GRASS meetup [
https://grasswiki.osgeo.org/wiki/GRASS_GIS_Ispra_meetups_2016] we can give
a meaningful example on how users can contribute to GRASS without coding
:-)

Cheers,
madi

>
> Moritz
>
> [1] http://www.rtbf.be/services/meteo/apere
>
>
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>



-- 
Margherita Di Leo
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Moritz Lennert]

On 26/09/16 23:50, Vaclav Petras wrote:


On Mon, Sep 26, 2016 at 5:24 PM, Veronica Andreo > wrote:

I'm with MaDi in that if I see a very long list of flags and
parameters in the terminal when using --h, i just go to the
manual... I just use --h in CLI for a quick recalling of
flags/options... A reduced list of most commonly used flags would be
nice, but still keep the possibility to get the advanced
flags/parameters as well, if the user needs more info...


If the --help is just for scanning and the issue is that it simply too
long, hiding some parameters is not the only option we have. For many
(and more in the future hopefully) parameters we have (short) label and
(longer) description. --help prints both (if both are present, that's at
least 2 lines per parameter). Additionally, if the option has predefined
values which have descriptions, there is one line for each of those. So,
the question is would it be helpful (at least as a first step) if --help
would print less information for each parameter but still provided all
parameters?


In line with your other mail where you caution about "hiding" useful 
information, how about not changing the --help output, but rather adding 
a "--simple-help" or somthing like this which would output a simplified 
help. Although:



and manual pages then (a tab or section for advanced
flags/parameters)...


Considering that we have currently as system of (gui) sections which
place the options to individual tabs in GUI, isn't showing the different
sections in the manual the right thing to do?


I would prefer this: show everything, but structure it differently, i.e. 
possibly introduce a new parser keyword (advanced: yes) which would put 
the option into a specific section of the manual. IMHO, this should be 
independent of the GUI sections logic as one might to group less 
advanced and advanced options in the same thematic tab...



IMHO, a major issue however is the lack of examples for the usage of
more advanced flags or options (or even the required and more common
ones). Take the case of several i.* modules, for example... but maybe
this should go in a separate thread :-)

>
>
> Good point. If you have an explanation and example for a flag, perhaps
> you can understand it and it is not so advanced at the end.

I think this is actually a major issue: man pages without description, 
notes and example sections are almost useless IMHO. At the foss4g.be 
last week someone presented a simple use of GRASS GIS (to create this 
map [1] for television) and explained how he actually found the GRASS 
GIS manual system extremely well done and useful, because of the 
explanations and examples...


Moritz

[1] http://www.rtbf.be/services/meteo/apere

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Vaclav Petras]

On Fri, Sep 23, 2016 at 5:22 PM, Markus Neteler  wrote:

> * various commands: "Use the low-memory version" - also more advanced


I guess this is a typical example where we need to be careful. I can
imagine a situation when a beginner has a lot of data and low-end hardware.
By hiding an parameter like this, we would make it even more advanced. Now
you not only need to understand that your hardware is not enough and that
the algorithm should somehow account for that. You also need to know that
there might be a hidden option for that and you need to know how to find
it. Having this in the manual or even a Memory or Speed tab in the GUI,
gives some hope that even a beginner will be able to use it.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Vaclav Petras]

On Fri, Sep 23, 2016 at 5:22 PM, Markus Neteler  wrote:

> Maybe not hide then but at least group the parameters and flags rather
> than alpha-order or random mix.
>

This is something which can be done right now. The options and flags are
presented in the order as they are defined in the source code. These
additional rules currently apply in general: (one letter) flags are listed
before (key-value) options. Long (multi-letter standard) flags are listed
in between them. Additionally in GUI, before this sorting, the options and
flags are always sorted to the tabs according to the specified (or
automatically generated) (gui) sections. Typically options and flags
(separately or combined) with (gui) section are specified together in the
source code, consequently they are grouped in --help and manual (but
without visible borders or any other indication).

Of course, this doesn't mean that the (gui) section cannot be contain, for
example 'basic', 'common' and 'advanced' subsections.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Vaclav Petras]

On Mon, Sep 26, 2016 at 5:24 PM, Veronica Andreo 
wrote:

> I'm with MaDi in that if I see a very long list of flags and parameters in
> the terminal when using --h, i just go to the manual... I just use --h in
> CLI for a quick recalling of flags/options... A reduced list of most
> commonly used flags would be nice, but still keep the possibility to get
> the advanced flags/parameters as well, if the user needs more info...
>

If the --help is just for scanning and the issue is that it simply too
long, hiding some parameters is not the only option we have. For many (and
more in the future hopefully) parameters we have (short) label and (longer)
description. --help prints both (if both are present, that's at least 2
lines per parameter). Additionally, if the option has predefined values
which have descriptions, there is one line for each of those. So, the
question is would it be helpful (at least as a first step) if --help would
print less information for each parameter but still provided all parameters?


> and the same in the GUI
>

In the case of GUI, we should first answer the question why are the
sections/tabs not enough? Because they already should separate basic and
advanced while providing actually much finer and partially self-documenting
way (for example, you fill in Required tab, go also to Selection, but leave
aside Transformation and Advanced tabs).


> and manual pages then (a tab or section for advanced flags/parameters)...
>
>
Considering that we have currently as system of (gui) sections which place
the options to individual tabs in GUI, isn't showing the different sections
in the manual the right thing to do? This would not only make the manual
more organized, but it would also link the manual more to the GUI.
Depending how good and smart are we with the styling and JavaScript we can
even hide some by default. But perhaps there is a reason why this is not
enough.


> IMHO, a major issue however is the lack of examples for the usage of more
> advanced flags or options (or even the required and more common ones). Take
> the case of several i.* modules, for example... but maybe this should go in
> a separate thread :-)
>

Good point. If you have an explanation and example for a flag, perhaps you
can understand it and it is not so advanced at the end.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Veronica Andreo]

Hello

I'm with MaDi in that if I see a very long list of flags and parameters in
the terminal when using --h, i just go to the manual... I just use --h in
CLI for a quick recalling of flags/options... A reduced list of most
commonly used flags would be nice, but still keep the possibility to get
the advanced flags/parameters as well, if the user needs more info... and
the same in the GUI and manual pages then (a tab or section for advanced
flags/parameters)...

IMHO, a major issue however is the lack of examples for the usage of more
advanced flags or options (or even the required and more common ones). Take
the case of several i.* modules, for example... but maybe this should go in
a separate thread :-)

Cheers,
Vero

2016-09-26 6:01 GMT-03:00 Margherita Di Leo :

> Hi,
>
> On Fri, Sep 23, 2016 at 12:00 AM, Markus Neteler 
> wrote:
>
>> Hi,
>>
>> in order to not hinder the addition of important new flags/parameters
>> to modules while keeping things optionally easy it would be good to
>> implement some expert mode to the parser.
>>
>> The flags and parameters for advanced users should be hidden by
>> default (maybe by using an extra definition in order to "tag" them in
>> the source code). Then e.g. by setting a variable they would become
>> visible in the help text and GUI.
>> Probably they should be always accepted when being invoked.
>>
>
>  This makes sense to me. If I type name.module --help and the command line
> yield too long a list of parameters, I normally prefer reading it on the
> manual page, where it is more clear, and I find extremely useful the usage
> examples of the flags. I don't find it very informative to list all the
> flags in command line, even the less used. There could be for example a
> --ext-help for extended help display. So for example, typing only --help
> would list only most used options and would end with "type --ext-help for
> extended help" or something like that.
>
> My 2 c
>
> --
> Margherita Di Leo
>
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Margherita Di Leo]

Hi,

On Fri, Sep 23, 2016 at 12:00 AM, Markus Neteler  wrote:

> Hi,
>
> in order to not hinder the addition of important new flags/parameters
> to modules while keeping things optionally easy it would be good to
> implement some expert mode to the parser.
>
> The flags and parameters for advanced users should be hidden by
> default (maybe by using an extra definition in order to "tag" them in
> the source code). Then e.g. by setting a variable they would become
> visible in the help text and GUI.
> Probably they should be always accepted when being invoked.
>

 This makes sense to me. If I type name.module --help and the command line
yield too long a list of parameters, I normally prefer reading it on the
manual page, where it is more clear, and I find extremely useful the usage
examples of the flags. I don't find it very informative to list all the
flags in command line, even the less used. There could be for example a
--ext-help for extended help display. So for example, typing only --help
would list only most used options and would end with "type --ext-help for
extended help" or something like that.

My 2 c

-- 
Margherita Di Leo
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Blumentrath, Stefan]

Hi,

I have been struggling with similar issues and solved them by writing a 
GRASS_BATCH_JOB script that I execute in parallel in different mapsets.
Thus, the approach Markus and Sören sketch looks very useful to me (even for 
people who are not on HPC).

Do you have more options / flags in mind than the region settings? Just asking 
to get a better idea about the amount of how module complexity may increase 
with the changes you plan?

If it is only region settings per module-call that should not be overwhelming 
for users. E.g., the QGIS-GRASS plugin (and I guess Processing too) has a 
button for «use region of this map» for every command with raster input, which 
is pointing in this direction (though not as sophisticated as your solution). 
And QGIS aimed at simplifying the interaction with GRASS analysis ;-).

In order to not blowing up the manual too much, one might write a compact 
standard description that points to g.region or a central, more detailed 
description of the new options, which will be the same for every module that 
uses them, would not they?

After you explained a bit more what you are after I do not really see a problem 
with this. In fact I am very positive!
But I have to admit that I did not understand the difference between 
--raster-region and --vector-region. Are you planning to implement some 
vector-tiles solution?

Cheers
Stefan

From: grass-dev [mailto:grass-dev-boun...@lists.osgeo.org] On Behalf Of Vaclav 
Petras
Sent: 26. september 2016 04:28
To: Markus Neteler <nete...@osgeo.org>
Cc: GRASS developers list <grass-dev@lists.osgeo.org>; Markus Metz 
<markus.metz.gisw...@gmail.com>
Subject: Re: [GRASS-dev] Adding an expert mode to the parser


On Sun, Sep 25, 2016 at 3:49 PM, Markus Neteler 
<nete...@osgeo.org<mailto:nete...@osgeo.org>> wrote:
when doing each job in its own mapset and reintegrate it in a single
target mapset as if able to process then in parallel in one mapset by
simply specifying the respective region to the command of interest.
Yes, different from the current paradigm and not for G7.

I think, we can accommodate this behavior in G7. In fact, each command can run 
with a separate region even now. It can set it on its own, take it from 
GRASS_REGION or WIND_OVERRIDE. But if you are talking about automatic patching, 
that's of course different.


But my original comment was targeted at the increasing number of
module parameters and how to handle that (with some new HPC related
idea as an example).

I think we need to talk about specific use cases and decide based on that. For 
some, Advanced or HPC tab might be enough. Some others might be better 
addressed by a specific global options like what Soeren just suggested (in the 
style of --overwrite).
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Vaclav Petras]

On Sun, Sep 25, 2016 at 3:49 PM, Markus Neteler  wrote:

> when doing each job in its own mapset and reintegrate it in a single
> target mapset as if able to process then in parallel in one mapset by
> simply specifying the respective region to the command of interest.
> Yes, different from the current paradigm and not for G7.
>

I think, we can accommodate this behavior in G7. In fact, each command can
run with a separate region even now. It can set it on its own, take it from
GRASS_REGION or WIND_OVERRIDE. But if you are talking about automatic
patching, that's of course different.


>
> But my original comment was targeted at the increasing number of
> module parameters and how to handle that (with some new HPC related
> idea as an example).
>

I think we need to talk about specific use cases and decide based on that.
For some, Advanced or HPC tab might be enough. Some others might be better
addressed by a specific global options like what Soeren just suggested (in
the style of --overwrite).
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Vaclav Petras]

On Sun, Sep 25, 2016 at 5:40 PM, Sören Gebbert  wrote:

> >r.mapclac --raster-region= --north= --south= --west= --east= --res=
>> >--ewres= --nsres= --vect-region --raster-align= ...
>>
>
I like this in the sense that if the region setting for the module is
something which should be managed by these extra options, then they should
be managed in parser, rather than introduced one by one for individual
modules.

But how is this different from using GRASS_REGION? Convenience of
--raster-region=?
Better syntax than environmental variable?
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Sören Gebbert]

[snip]

> >r.mapclac --raster-region= --north= --south= --west= --east= --res=
> >--ewres= --nsres= --vect-region --raster-align= ...
>
> Can't you do this at a higher level, i.e using the GRASS Python API's
> use_temp_region ?
>

This is exactly what i'm doing right now[1,2], covering a single module
call in several g.region and g.remove calls in a dedicated subprocess.
Huge, massive unnecessary overhead.

[1]
https://trac.osgeo.org/grass/browser/grass/trunk/lib/python/pygrass/modules/interface/module.py#L791
[2]
https://trac.osgeo.org/grass/browser/grass/trunk/lib/python/pygrass/modules/interface/module.py#L951


> Moritz
>
>
> >
> >Just my 2 cent
> >Sören
> >
> >2016-09-25 20:49 GMT+01:00 Markus Neteler :
> >
> >> On Fri, Sep 23, 2016 at 11:30 PM, Markus Metz
> >>  wrote:
> >> > On Fri, Sep 23, 2016 at 11:22 PM, Markus Neteler
> >
> >> wrote:
> >> >> On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
> >> >>  wrote:
> >> >>> On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler
> >
> >> wrote:
> >> >> ...
> >> >>> Your motivation is to provide a specialized CLI interface for HPC
> >> >>> processing?
> >> >>
> >> >> No, not the case.
> >> >>
> >> >>> We used GRASS with HPC processing for years and the
> >> >>> problems we faced were causes by the HPC hardware and software
> >> >>> infrastructure, not by GRASS. What exactly is the problem with
> >using
> >> >>> GRASS and HPC processing?
> >> >>
> >> >> There is no problem. There is just the issue that with an
> >increasing
> >> >> amount of additions (e.g. maybe the need to provide
> >region/resolution
> >> >> to individual modules for independent parallel processing without
> >the
> >> >> overhead of always opening a new mapset)
> >> >
> >> > Getting closer it seems. Can you quantify "the overhead of always
> >> > opening a new mapset"?
> >>
> >> As an example, when aiming at processing all Sentinel-2 tiles
> >> globally, we speak about currently 73000 scenes * up-to-16 tiles
> >along
> >> with global data, analysis on top of other global data is more
> >complex
> >> when doing each job in its own mapset and reintegrate it in a single
> >> target mapset as if able to process then in parallel in one mapset by
> >> simply specifying the respective region to the command of interest.
> >> Yes, different from the current paradigm and not for G7.
> >>
> >> But my original comment was targeted at the increasing number of
> >> module parameters and how to handle that (with some new HPC related
> >> idea as an example).
> >>
> >> I'm fine to archive this question for now, it will likely come up
> >> again in the future.
> >>
> >> markusN
> >> ___
> >> grass-dev mailing list
> >> grass-dev@lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/grass-dev
> >>
> >
> >
> >
> >
> >___
> >grass-dev mailing list
> >grass-dev@lists.osgeo.org
> >http://lists.osgeo.org/mailman/listinfo/grass-dev
>
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Sören Gebbert]

Hi,
Here an example of hidden region settings for modules utilized in the
temporal framework:

Imagine you have a Landsat location that contains many thousand scenes
(time series) from all over the world including all bands. Each band of the
time series is managed in a specific space-time dataset. You may want to
use the temporal raster algebra to compute the NDVI in parallel on all
scenes at once. This is not possible at the moment, because of the current
region settings. Except you can set the current region to cover the whole
world, which is not preferable.

So the idea is to have options that will set the computational region for
each process independently, when the process is invoked. Hence the raster
computational window is set in the parser, so that the usual region get
methods in the modules will work untouched. This avoids the current
approach that is covering a single module call (like r.mapclalc, r.series,
r.neighbors, ... ) in several g.region and g.remove calls in a dedicated
subprocess.

This option is only useful for parallel processing of raster layers with
different spatial extents. A special feature for the temporal framework,
that should not be exposed to all the users, only to those who implement
massive parallel working temporal modules. Hence using the same approach as
for WPS, GUI and HTML output:

r.mapclac --raster-region= --north= --south= --west= --east= --res=
--ewres= --nsres= --vect-region --raster-align= ...

Just my 2 cent
Sören

2016-09-25 20:49 GMT+01:00 Markus Neteler :

> On Fri, Sep 23, 2016 at 11:30 PM, Markus Metz
>  wrote:
> > On Fri, Sep 23, 2016 at 11:22 PM, Markus Neteler 
> wrote:
> >> On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
> >>  wrote:
> >>> On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler 
> wrote:
> >> ...
> >>> Your motivation is to provide a specialized CLI interface for HPC
> >>> processing?
> >>
> >> No, not the case.
> >>
> >>> We used GRASS with HPC processing for years and the
> >>> problems we faced were causes by the HPC hardware and software
> >>> infrastructure, not by GRASS. What exactly is the problem with using
> >>> GRASS and HPC processing?
> >>
> >> There is no problem. There is just the issue that with an increasing
> >> amount of additions (e.g. maybe the need to provide region/resolution
> >> to individual modules for independent parallel processing without the
> >> overhead of always opening a new mapset)
> >
> > Getting closer it seems. Can you quantify "the overhead of always
> > opening a new mapset"?
>
> As an example, when aiming at processing all Sentinel-2 tiles
> globally, we speak about currently 73000 scenes * up-to-16 tiles along
> with global data, analysis on top of other global data is more complex
> when doing each job in its own mapset and reintegrate it in a single
> target mapset as if able to process then in parallel in one mapset by
> simply specifying the respective region to the command of interest.
> Yes, different from the current paradigm and not for G7.
>
> But my original comment was targeted at the increasing number of
> module parameters and how to handle that (with some new HPC related
> idea as an example).
>
> I'm fine to archive this question for now, it will likely come up
> again in the future.
>
> markusN
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Markus Neteler]

On Fri, Sep 23, 2016 at 11:30 PM, Markus Metz
 wrote:
> On Fri, Sep 23, 2016 at 11:22 PM, Markus Neteler  wrote:
>> On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
>>  wrote:
>>> On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler  wrote:
>> ...
>>> Your motivation is to provide a specialized CLI interface for HPC
>>> processing?
>>
>> No, not the case.
>>
>>> We used GRASS with HPC processing for years and the
>>> problems we faced were causes by the HPC hardware and software
>>> infrastructure, not by GRASS. What exactly is the problem with using
>>> GRASS and HPC processing?
>>
>> There is no problem. There is just the issue that with an increasing
>> amount of additions (e.g. maybe the need to provide region/resolution
>> to individual modules for independent parallel processing without the
>> overhead of always opening a new mapset)
>
> Getting closer it seems. Can you quantify "the overhead of always
> opening a new mapset"?

As an example, when aiming at processing all Sentinel-2 tiles
globally, we speak about currently 73000 scenes * up-to-16 tiles along
with global data, analysis on top of other global data is more complex
when doing each job in its own mapset and reintegrate it in a single
target mapset as if able to process then in parallel in one mapset by
simply specifying the respective region to the command of interest.
Yes, different from the current paradigm and not for G7.

But my original comment was targeted at the increasing number of
module parameters and how to handle that (with some new HPC related
idea as an example).

I'm fine to archive this question for now, it will likely come up
again in the future.

markusN
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Markus Metz]

On Fri, Sep 23, 2016 at 11:22 PM, Markus Neteler  wrote:
> On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
>  wrote:
>> On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler  wrote:
> ...
>> Your motivation is to provide a specialized CLI interface for HPC
>> processing?
>
> No, not the case.
>
>> We used GRASS with HPC processing for years and the
>> problems we faced were causes by the HPC hardware and software
>> infrastructure, not by GRASS. What exactly is the problem with using
>> GRASS and HPC processing?
>
> There is no problem. There is just the issue that with an increasing
> amount of additions (e.g. maybe the need to provide region/resolution
> to individual modules for independent parallel processing without the
> overhead of always opening a new mapset)

Getting closer it seems. Can you quantify "the overhead of always
opening a new mapset"?

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Markus Neteler]

On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
 wrote:
> On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler  wrote:
...
> Your motivation is to provide a specialized CLI interface for HPC
> processing?

No, not the case.

> We used GRASS with HPC processing for years and the
> problems we faced were causes by the HPC hardware and software
> infrastructure, not by GRASS. What exactly is the problem with using
> GRASS and HPC processing?

There is no problem. There is just the issue that with an increasing
amount of additions (e.g. maybe the need to provide region/resolution
to individual modules for independent parallel processing without the
overhead of always opening a new mapset) to the user interface it
becomes harder to understand for most users.

> You could ask on the GRASS user ML, or a
> GRASS developer who has experience with HPC processing.

I have 10+ years of experience in GRASS HPC :-) So that's fine.

 How do we make sure people understand the options were not
 removed, but are just hidden?
>>
>> For those being on command line, we could either show the advanced options,
>> or, if not show instead a hint that this particular command also offers
>> advanced options.
>
> I assume that most GRASS users regard the command line as advanced
> usage, therefore the CLI should always show all flags and options.

I don't think that all CLI users want to see always 40+ flags and
options per command if it could also be customizable.
Of course there may be different opinions.

...
> You wrote "The flags and parameters for advanced users should be
> hidden by default", and Anna referred to such advanced options.

Simple example, not HPC related:

* r.sun: "npartitions"  - not really relevant for day2day "normal" usage.
* various commands: "Use the low-memory version" - also more advanced
etc.

>> It is not about hiding, as I just tried to explain maybe better now.
>> It is about not adding so many more flags and parameters to the standard
>> interface, or, a way how to address this upcoming problem for the near
>> future.
>
> As you said, your problem is related to HPC processing. Why do you
> think more flags and options are a problem for HPC processing?

No, not a problem for using it.
But maybe not interesting for all users to be always seen when using
GRASS in "shallow mode".

> Particularly for HPC processing, I would assume that the more
> information and the more control you have, the better.

Exactly. Just not all (new) users want to see a huge interface of
which they only use a fraction.
That's why it could be structured better in order to distinguish
between "easy" and "advanced".
Maybe not hide then but at least group the parameters and flags rather
than alpha-order or random mix.

markusN
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Markus Metz]

On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler  wrote:
> On Fri, Sep 23, 2016 at 10:15 AM, Moritz Lennert
>  wrote:
>> On 23/09/16 02:37, Anna Petrášová wrote:
> ...
>>> How do we decide which options are advanced.
>
> We simply keep 95% of all as-is.
> But those we need for example for improved HPC processing (yet to be added)
> we may not want to clutter the standard interface with,

Your motivation is to provide a specialized CLI interface for HPC
processing? We used GRASS with HPC processing for years and the
problems we faced were causes by the HPC hardware and software
infrastructure, not by GRASS. What exactly is the problem with using
GRASS and HPC processing? You could ask on the GRASS user ML, or a
GRASS developer who has experience with HPC processing.

>
>>> How do we make sure people understand the options were not
>>> removed, but are just hidden?
>
> For those being on command line, we could either show the advanced options,
> or, if not show instead a hint that this particular command also offers
> advanced options.

I assume that most GRASS users regard the command line as advanced
usage, therefore the CLI should always show all flags and options.

>
>>> If the goal of this is to simplify life for new users, we can focus
>>> the efforts on GUI.
>
> Yes, that's easy. But I am indeed thinking about the command line here.

See above. Why exactly do you want to change the command line
interface? What is the problem?

>
>>> If we tag an option as advanced in parser, it
>>> could be moved to a special tab 'Advanced', which some modules already
>>> have.
>
> Yes.
>
>>> I am not sure hiding the options completely is a good idea.
>
> This is not what I intended (nor wrote I think).

You wrote "The flags and parameters for advanced users should be
hidden by default", and Anna referred to such advanced options.

> It is not about hiding, as I just tried to explain maybe better now.
> It is about not adding so many more flags and parameters to the standard
> interface, or, a way how to address this upcoming problem for the near
> future.

As you said, your problem is related to HPC processing. Why do you
think more flags and options are a problem for HPC processing?
Particularly for HPC processing, I would assume that the more
information and the more control you have, the better.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Moritz Lennert]

On 23/09/16 13:11, Markus Neteler wrote:

On Fri, Sep 23, 2016 at 10:15 AM, Moritz Lennert
> wrote:

On 23/09/16 02:37, Anna Petrášová wrote:

...

Probably not so complicated, but I would be more concerned with the
decisions coming with this. For example, what should the online manual
show?


Using the parser, the advanced options could automatically go into a
subsection of the description part.
Just to distinguish them easily from the "standard" options.


How do we decide which options are advanced.


We simply keep 95% of all as-is.
But those we need for example for improved HPC processing (yet to be
added) we may not want to clutter the standard interface with,


No possibility of embedding this into some configuration module (à la 
r.li.setup) which sets the relevant options and then just adding one 
flag to the other modules, e.g. '-h' for 'take into account the HPC 
settings ?






How do we make sure people understand the options were not
removed, but are just hidden?


Easy: since they show up in the manual, it is documented.
Having an "[ ] advanced" button in the GUI, it is obvious.
For those being on command line, we could either show the advanced
options, or, if not show instead a hint that this particular command
also offers advanced options.


I would plead for a separate, but visible 'advanced options' section.



Example: today d.vect comes with this mount of flags and parameters:
d.vect --interface-description | grep 'flag name\|parameter name' | wc -l
39

In a single line (dirty hack):

for i in `ls -1 bin/* scripts/*` ; do echo -n "$i," ; $i
--interface-description | grep 'flag name\|parameter name' | wc -l ;
done | sort -n -t',' -k2
[...]
bin/r.watershed,27
bin/r.sim.water,28
bin/v.in.ogr,28
bin/r.gwflow,29
bin/v.vol.rst,29
bin/r.solute.transport,32
bin/v.surf.rst,32
bin/d.legend,33
bin/r.sun,34
bin/d.vect,39
bin/g.region,40


So, would adding 25% of new flags and parameters to many modules in the
next years still be ok? I guess not. So we need to understand how to
handle that.
Hence my proposal to structure them a bit better.


Ok. We might also think about the option of subdividing those modules 
with too many flags into several separate modules sharing the same code 
base. Or go the QGIS way, i.e. create several pre-configured wrapper 
scripts for the most "complicated" modules with several options preset 
and thus not visible in the interface.


But, I see your point and it is definitely necessary to think about the 
options.


Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Markus Neteler]

On Fri, Sep 23, 2016 at 10:15 AM, Moritz Lennert <
mlenn...@club.worldonline.be> wrote:
> On 23/09/16 02:37, Anna Petrášová wrote:
...
>> Probably not so complicated, but I would be more concerned with the
>> decisions coming with this. For example, what should the online manual
>> show?

Using the parser, the advanced options could automatically go into a
subsection of the description part.
Just to distinguish them easily from the "standard" options.

>> How do we decide which options are advanced.

We simply keep 95% of all as-is.
But those we need for example for improved HPC processing (yet to be added)
we may not want to clutter the standard interface with,


>> How do we make sure people understand the options were not
>> removed, but are just hidden?

Easy: since they show up in the manual, it is documented.
Having an "[ ] advanced" button in the GUI, it is obvious.
For those being on command line, we could either show the advanced options,
or, if not show instead a hint that this particular command also offers
advanced options.

>> If the goal of this is to simplify life for new users, we can focus
>> the efforts on GUI.

Yes, that's easy. But I am indeed thinking about the command line here.

>> If we tag an option as advanced in parser, it
>> could be moved to a special tab 'Advanced', which some modules already
>> have.

Yes.

>> I am not sure hiding the options completely is a good idea.

This is not what I intended (nor wrote I think).

>> Also in manual pages, perhaps these advanced options could somehow be
>> visibly marked as advanced (or possibly hidden with some javascript?).

Exactly.

> If any, I would say +1 for Anna's suggestion. I don't really like the idea
> of "hiding" options behind an environment variable.

It is not about hiding, as I just tried to explain maybe better now.
It is about not adding so many more flags and parameters to the standard
interface, or, a way how to address this upcoming problem for the near
future.

> This said, I haven't really experienced that many problems with complexity
> of modules with newbies. Yes, some are a bit overwhelmed at first contact,
> but with a little guidance they quickly get the hang of things...

But extrapolating from the '90th, we tend to add more and more..

Example: today d.vect comes with this mount of flags and parameters:
d.vect --interface-description | grep 'flag name\|parameter name' | wc -l
39

In a single line (dirty hack):

for i in `ls -1 bin/* scripts/*` ; do echo -n "$i," ; $i
--interface-description | grep 'flag name\|parameter name' | wc -l ; done |
sort -n -t',' -k2
[...]
bin/r.watershed,27
bin/r.sim.water,28
bin/v.in.ogr,28
bin/r.gwflow,29
bin/v.vol.rst,29
bin/r.solute.transport,32
bin/v.surf.rst,32
bin/d.legend,33
bin/r.sun,34
bin/d.vect,39
bin/g.region,40


So, would adding 25% of new flags and parameters to many modules in the
next years still be ok? I guess not. So we need to understand how to handle
that.
Hence my proposal to structure them a bit better.

cheers,
Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Moritz Lennert]

On 23/09/16 02:37, Anna Petrášová wrote:

On Thu, Sep 22, 2016 at 6:00 PM, Markus Neteler  wrote:

Hi,

in order to not hinder the addition of important new flags/parameters
to modules while keeping things optionally easy it would be good to
implement some expert mode to the parser.

The flags and parameters for advanced users should be hidden by
default (maybe by using an extra definition in order to "tag" them in
the source code). Then e.g. by setting a variable they would become
visible in the help text and GUI.
Probably they should be always accepted when being invoked.

Any thoughts if such mechanism would be complicated to be implemented?


Probably not so complicated, but I would be more concerned with the
decisions coming with this. For example, what should the online manual
show? How do we decide which options are advanced. How do we make sure
people understand the options were not removed, but are just hidden?

If the goal of this is to simplify life for new users, we can focus
the efforts on GUI. If we tag an option as advanced in parser, it
could be moved to a special tab 'Advanced', which some modules already
have. I am not sure hiding the options completely is a good idea. Also
in manual pages, perhaps these advanced options could somehow be
visibly marked as advanced (or possibly hidden with some javascript?).
Let's see what others think about this.


If any, I would say +1 for Anna's suggestion. I don't really like the 
idea of "hiding" options behind an environment variable.


This said, I haven't really experienced that many problems with 
complexity of modules with newbies. Yes, some are a bit overwhelmed at 
first contact, but with a little guidance they quickly get the hang of 
things...


Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Adding an expert mode to the parser

[Anna Petrášová]

On Thu, Sep 22, 2016 at 6:00 PM, Markus Neteler  wrote:
> Hi,
>
> in order to not hinder the addition of important new flags/parameters
> to modules while keeping things optionally easy it would be good to
> implement some expert mode to the parser.
>
> The flags and parameters for advanced users should be hidden by
> default (maybe by using an extra definition in order to "tag" them in
> the source code). Then e.g. by setting a variable they would become
> visible in the help text and GUI.
> Probably they should be always accepted when being invoked.
>
> Any thoughts if such mechanism would be complicated to be implemented?

Probably not so complicated, but I would be more concerned with the
decisions coming with this. For example, what should the online manual
show? How do we decide which options are advanced. How do we make sure
people understand the options were not removed, but are just hidden?

If the goal of this is to simplify life for new users, we can focus
the efforts on GUI. If we tag an option as advanced in parser, it
could be moved to a special tab 'Advanced', which some modules already
have. I am not sure hiding the options completely is a good idea. Also
in manual pages, perhaps these advanced options could somehow be
visibly marked as advanced (or possibly hidden with some javascript?).
Let's see what others think about this.

Anna

>
> Markus
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] Adding an expert mode to the parser

[Markus Neteler]

Hi,

in order to not hinder the addition of important new flags/parameters
to modules while keeping things optionally easy it would be good to
implement some expert mode to the parser.

The flags and parameters for advanced users should be hidden by
default (maybe by using an extra definition in order to "tag" them in
the source code). Then e.g. by setting a variable they would become
visible in the help text and GUI.
Probably they should be always accepted when being invoked.

Any thoughts if such mechanism would be complicated to be implemented?

Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev