Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  closed
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  fixed   |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--

Comment (by Nikos Alexandris):

 I didn't open a new ticket for this because I (wrongly) thought it to be
 part of the implementation of --tmp-location. Will open one--though the
 problem might reside on my OS/File-System side.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  closed
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  fixed   |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--

Comment (by wenzeslaus):

 Hi Nikos, I'm looking at this now and:

 1) I can't reproduce the "if the file does not exist, the process hangs"
 behavior from your [https://lists.osgeo.org/pipermail/grass-
 user/2018-August/078970.html email]. I get `ERROR: ... xxx.xxx: No such
 file or directory`.

 2) This should be really a separate ticket because it is/would be a bug
 (this is a closed enhancement ticket).

 3) It is unrelated to `--tmp-location` unless you can show that it does
 not happen with `-c` (important only for testing and reporting).

 4) Your considerations about what to report based on what are the right
 thing to do, let's discuss it in a new ticket of you can reproduce the
 hanging or you have some other issues (like the message).

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  closed
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  fixed   |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--

Comment (by Nikos Alexandris):

 Just some extra thoughts: perhaps the error message should *not* say
 anything but what actually happens: the file is not found.

 What are your thoughts in general? Even this very check, in case the file
 does exist, does not ensure that the file will be available. Right after
 the check anything can happen at the OS level.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  closed
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  fixed   |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--
Changes (by Nikos Alexandris):

 * Attachment "grass.py_check_geostring.2.patch" added.

 Check if geostring exists and if target file is readable

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  closed
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  fixed   |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--
Changes (by Nikos Alexandris):

 * Attachment "grass.py_check_geostring.patch" added.

 Check if geostring exists and if target file is readable

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  closed
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  fixed   |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--

Comment (by Nikos Alexandris):

 A suggestion to check for if the 'geofile' (in `grass.py` named as
 `geostring`) exists and if it is is readable:
 
https://trac.osgeo.org/grass/attachment/ticket/3537/grass.py_check_geostring.patch.

 I believe to have tested this properly.
 I could not follow what is recommended at
 https://stackoverflow.com/a/82852/1172302.
 In example, something like

 {{{
 try:
 geostring = pathlib.Path(geostring)
 geostring.resolve()
 except IOError as e:
 fatal(_("{error}"
 "\n\n"
 "Check spelling "
 "or the file's read permissions.").format(error=e))
 else:
 # create location using georeferenced file
 gcore.create_location(gisdbase, location,
 filename=geostring)

 }}}

 would not test for read access. I could find a way to test for read access
 using something from https://docs.python.org/3/library/pathlib.html.

 Maybe this can be implemented by using "future" built-in exceptions as
 FileNotFoundError (see https://docs.python.org/3/library/exceptions.html)?

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  closed
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  fixed   |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--
Changes (by Nikos Alexandris):

 * Attachment "grass.py_check_geostring.patch" added.

 Check if geostring exists and if target file is readable

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  closed
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  fixed   |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--
Changes (by wenzeslaus):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 The main part, `--tmp-location`, was implemented in r72790 and improved in
 r73096 (#3585), 73099, and r73098 (#3586). Now we can do:

 Execute script in a loc created based on EPSG:

 {{{
 grass75 --tmp-location EPSG:3358 --exec script.py
 }}}

 Get CRS for a file:

 {{{
 grass75 --tmp-location ~/data/elevation.tiff --exec g.proj -p
 }}}

 Get help text for a module:

 {{{
 grass75 --tmp-location XY --exec r.neighbors --help
 }}}

 `--no-clean` is a case for a separate ticket (please open it if you are
 interested).

 Closing this one as fixed.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  new
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--

Comment (by wenzeslaus):

 I committed a modified `--tmp-location` portion of the patch (`grass-tmp-
 loc-2.diff​`) in r72790. I provided couple use cases in the documentation.

 I worked on the `--no-clean` portion a little bit, but that needs more
 work. The mapsets `.tmp` directory is deleted at many places.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  new
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--
Changes (by radeknovotny94):

 * Attachment "grass-tmp-loc-2.diff" added.


-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  new
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--

Comment (by radeknovotny94):

 Replying to [comment:4 wenzeslaus]:
 > Thank you for submitting a new version. Perhaps next time, leave there
 the old diff file, so we can refer to it if needed. About the diff itself:
 >
 > 1. I didn't catch it the first time, but it seems there was
 misunderstanding about what `--no-clean` should do. See the
 [wiki:GSoC/2018#Neweasy-to-useCLIandAPIforGRASSGIS wiki] again...
 Thank you for the better explanation, I tried to fix it in new patch.
 Please add your comments.
 > 2. Please provide documentation for this, ideally as part of the
 `grass7.html` file. This will make testing easier and addressing the use
 cases.
 I made the first version of the documentation and I am also waiting for
 your comments.

 > 3. It would be great of you come up with some more formal tests for it.
 It does not really fit into the `gunittest` tests but we can always start
 with a Bash script executing series of commands.
 I will make this Bash script in next days.

 > 4. Any particular reason to place `--no-clean` on the same line with
 `[[[GISDBASE/]LOCATION_NAME/]MAPSET]` in the help message? It should be
 probably with `--exec` or close to `-f`.
 It was because in my previous implementation `--no-clean` works for
 calling without `--exec` too. I changed it.

 > 5. In generally, please avoid unrelated whitespace changes in the
 patches/commits. Whitespace and other PEP8 things can and should be
 changed, but in separate commits. See your lines 375 and 403 (Trac view
 numbering) in your diff.

 Thanks, I know it, I was more attentive about it this time.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  new
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--

Comment (by wenzeslaus):

 Thank you for submitting a new version. Perhaps next time, leave there the
 old diff file, so we can refer to it if needed. About the diff itself:

 1. I didn't catch it the first time, but it seems there was
 misunderstanding about what `--no-clean` should do. See the
 [wiki:GSoC/2018#Neweasy-to-useCLIandAPIforGRASSGIS wiki] again, ''...if
 --exec should clean the .tmp directory in the Mapset.'' This relates to
 the `clean_temp()` function (it is not connected to the `Cleaner` class,
 the code is only partially re-factored and this is really something
 different anyway). The description at the wiki says ''for parallel
 execution''. Try to run two (longer running) processes in parallel, at
 least sometimes, one will fail because the other deleted its temporary
 files in the Mapset .tmp directory. Let us know if you need more
 explanation on this, perhaps directly on the mailing list, so that more
 people can answer. I'm not sure at this point if your `--no-clean` is
 useful for something or not. Did you have any use case?

 2. Please provide documentation for this, ideally as part of the
 `grass7.html` file. This will make testing easier and addressing the use
 cases.

 3. It would be great of you come up with some more formal tests for it. It
 does not really fit into the `gunittest` tests but we can always start
 with a Bash script executing series of commands.

 4. Any particular reason to place `--no-clean` on the same line with
 `[[[GISDBASE/]LOCATION_NAME/]MAPSET]` in the help message? It should be
 probably with `--exec` or close to `-f`.

 5. In generally, please avoid unrelated whitespace changes in the
 patches/commits. Whitespace and other PEP8 things can and should be
 changed, but in separate commits. See your lines 375 and 403 (Trac view
 numbering) in your diff.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  new
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--

Comment (by radeknovotny94):

 Replying to [comment:1 wenzeslaus]:
 > Thumbs up for creating a `find_tmp()` function, but you replaced\\
 > ...\\
 > Note esp. the for loop and the test using try.
 I looked at it again and I think that finally is all new function
 {{{find_tmp}}} unnecessary. Because it is set environmental variable
 TMPDIR at the end of function {{{create_tmp}}}.

 {{{
 # promoting the variable even if it was not defined before
 os.environ['TMPDIR'] = tmpdir
 }}}

 It means that TMPDIR have to be defined when we call function for creating
 temporary location.
 {{{
 gisdbase = os.getenv('TMPDIR')
 }}}

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  new
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--
Changes (by radeknovotny94):

 * Attachment "grass-tmp-loc.diff" added.


-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  new
  Priority:  normal  |  Milestone:  7.6.0
 Component:  Startup |Version:  svn-trunk
Resolution:  |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--
Changes (by martinl):

 * milestone:  8.0.0 => 7.6.0


-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  new
  Priority:  normal  |  Milestone:  8.0.0
 Component:  Startup |Version:  svn-trunk
Resolution:  |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--

Comment (by wenzeslaus):

 Thumbs up for creating a `find_tmp()` function, but you replaced:

 {{{
 #!python
 for ttmp in ("/tmp", "/var/tmp", "/usr/tmp"):
 tmp = ttmp
 tmpdir = os.path.join(tmp, "grass7-%(user)s-%(lock)s" % {
   'user': user, 'lock': gis_lock})
 try:
 os.mkdir(tmpdir, 0o700)
 except:
 tmp = None
 if tmp:
 break
 }}}

 by:

 {{{
 #!python
 def find_tmp():
 ...
 for ttmp in ("/tmp", "/var/tmp", "/usr/tmp"):
 tmp = ttmp
 ...
 tmp = find_tmp()
 tmpdir = os.path.join(tmp, "grass7-%(user)s-%(lock)s" % {'user': user,
  'lock':
 gis_lock})
 try:
 os.mkdir(tmpdir, 0o700)
 ...
 }}}

 Note esp. the for loop and the test using try.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3537: Add functionalities –tmp-location and –no-clean

[GRASS GIS]

#3537: Add functionalities –tmp-location and –no-clean
-+--
  Reporter:  radeknovotny94  |  Owner:  grass-dev@…
  Type:  enhancement | Status:  new
  Priority:  normal  |  Milestone:  8.0.0
 Component:  Startup |Version:  svn-trunk
Resolution:  |   Keywords:  tmp-location, no-clean, exec
   CPU:  All |   Platform:  All
-+--
Changes (by radeknovotny94):

 * Attachment "grass-tmp-loc.diff" added.


-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev