Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Neteler]

On Wed, Apr 9, 2014 at 4:07 AM, Glynn Clements  wrote:
> Packages which include command-line programs (e.g. compilers)
> typically either modify the environment variables (e.g. PATH) via the
> registry[1], or provide a .bat/.cmd file which must be run to set up
> the environment for a particular shell.

FYI - Just found this ticket:

optional package to register python in Windows registry
https://trac.osgeo.org/osgeo4w/ticket/114

Dunno if that's of any relevance here.

Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Vaclav Petras wrote:

> > I think this is very common in companies and public institutions. I
> > know several companies with much more restrictive guideline (no
> > internet connection, USB sticks not allowed, ...). So you can't expect
> > that user have a free choice.
> 
> Hi, I don't understand how this actually influence whether GRASS should
> have Python inside or outside. For user there is no difference since it
> should be managed by sysadmin, so what is sysadmin's decision process?

Well, one factor is that a sysadmin may be willing to install the
stock Python version from the MSI on the python.org site (although
there are plenty that wouldn't even allow that), but somewhat less
willing to install a second copy of Python that's bundled into a
relatively-obscure third-party package that no-one (other than one
user who's asking to install it) has ever heard of.

Sysadmins typically don't perform exhaustive security analysis using
debuggers, disassemblers, etc. Approval is more likely to be based
upon reputation, third-party reviews, and prejudice. The last one
tends to hurt anything that's perceived as opening the system up to
additional risks (e.g. languages or other "developer" tools).

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Newcomb, Doug]

On Thu, May 8, 2014 at 9:52 AM, Vaclav Petras  wrote:

>
> On Thu, May 8, 2014 at 9:22 AM, Sören Gebbert <
> soerengebb...@googlemail.com> wrote:
>
>> >
>> > On MS Windows, users do have a free choice of what they install. Third
>> > party software packages are on MS Windows completely independent of
>> > each other. If you don't like this, you have to change the MS Windows
>> > world first, then GRASS.
>> >
>> > On MS Windows, GRASS and a system-wide Python installation are third
>> > party software packages that are completely independent of each other.
>>
>> In my Institute the user do not have the free choice of what they
>> can install on their windows machines.
>> There are guidelines what software and version is allowed to be
>> installed. There are Institution specific software repositories that
>> must be used.
>> All software must be installed by the system administrator, user do
>> not have any permissions
>> to do this them self. This is for security and maintenance reasons.
>>
>> I think this is very common in companies and public institutions. I
>> know several companies with much more restrictive guideline (no
>> internet connection, USB sticks not allowed, ...). So you can't expect
>> that
>> user have a free choice.
>>
>
> Hi, I don't understand how this actually influence whether GRASS should
> have Python inside or outside. For user there is no difference since it
> should be managed by sysadmin, so what is sysadmin's decision process?
>

1) does it introduce any security risks ( open ports, install remotely
accessible services, change system file/directory permissions, etc)?
2) does it require administrative access to run ( i.e, write access to
install directories under c:\Program Files by normal users is generally
prohibited) ?
3) does it break existing software?
4) does it perform useful tasks ?

Doug



-- 
Doug Newcomb
USFWS
Raleigh, NC
919-856-4520 ext. 14 doug_newc...@fws.gov
-
The opinions I express are my own and are not representative of the
official policy of the U.S.Fish and Wildlife Service or Dept. of the
Interior.   Life is too short for undocumented, proprietary data formats.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Thu, May 8, 2014 at 9:22 AM, Sören Gebbert
wrote:

> >
> > On MS Windows, users do have a free choice of what they install. Third
> > party software packages are on MS Windows completely independent of
> > each other. If you don't like this, you have to change the MS Windows
> > world first, then GRASS.
> >
> > On MS Windows, GRASS and a system-wide Python installation are third
> > party software packages that are completely independent of each other.
>
> In my Institute the user do not have the free choice of what they
> can install on their windows machines.
> There are guidelines what software and version is allowed to be
> installed. There are Institution specific software repositories that
> must be used.
> All software must be installed by the system administrator, user do
> not have any permissions
> to do this them self. This is for security and maintenance reasons.
>
> I think this is very common in companies and public institutions. I
> know several companies with much more restrictive guideline (no
> internet connection, USB sticks not allowed, ...). So you can't expect
> that
> user have a free choice.
>

Hi, I don't understand how this actually influence whether GRASS should
have Python inside or outside. For user there is no difference since it
should be managed by sysadmin, so what is sysadmin's decision process?
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Sören Gebbert]

Hi Markus,

2014-05-07 21:50 GMT+02:00 Markus Metz :
> On Wed, Apr 30, 2014 at 9:51 AM, Glynn Clements
>  wrote:
>>
>> Markus Metz wrote:
>>
>>> >> > By all means provide fall-backs, workarounds, alternatives, or
>>> >> > whatever, but anything which tries to make such things mandatory is
>>> >> > going to get reverted. Again.
>>> >>
>>> >> really nice attitude ;-) Martin
>>> >
>>> > At least I'm not saying "you ARE going to use our version of Python,
>>> > whether you like it or not".
>>>
>>> People installing GRASS want to use GRASS. They want GRASS to work out
>>> of the box. They can use any Python version they want, as long as
>>> WinGRASS uses its embedded Python version. Users will not notice it.
>>
>> You're assuming that users have a free choice as to what they install.
>
> Oh, you say users should not have a free choice of what they install?
> Please explain this to MS Windows users who want to give GRASS a try.

I don't think that Glynn was trying to say that.

>
> On MS Windows, users do have a free choice of what they install. Third
> party software packages are on MS Windows completely independent of
> each other. If you don't like this, you have to change the MS Windows
> world first, then GRASS.
>
> On MS Windows, GRASS and a system-wide Python installation are third
> party software packages that are completely independent of each other.

In my Institute the user do not have the free choice of what they
can install on their windows machines.
There are guidelines what software and version is allowed to be
installed. There are Institution specific software repositories that
must be used.
All software must be installed by the system administrator, user do
not have any permissions
to do this them self. This is for security and maintenance reasons.

I think this is very common in companies and public institutions. I
know several companies with much more restrictive guideline (no
internet connection, USB sticks not allowed, ...). So you can't expect
that
user have a free choice.

>
> Anything but using GRASS_PYTHON to call Python scripts will cause
> trouble on MS WIndows. See this thread.
>
>> Some sites actually have policies about what software they'll allow on
>> their systems.
>
> How arrogant. Third party software packages are on MS Windows
> completely independent of each other.

This sounds like another misunderstanding. :)

Best regards
Soeren

>
> As soon as I have enough time to do this, I will implement the
> GRASS_PYTHON mechanism. GRASS_PYTHON can always be set to the
> system-wide .py file association, whatever that will be, some
> incompatible/incomplete Python version, some .bat script (which could
> work BTW), a text editor, or some general developer editor or SDK such
> as the one from MS .
>
> Markus M
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> >> People installing GRASS want to use GRASS. They want GRASS to work out
> >> of the box. They can use any Python version they want, as long as
> >> WinGRASS uses its embedded Python version. Users will not notice it.
> >
> > You're assuming that users have a free choice as to what they install.
> 
> Oh, you say users should not have a free choice of what they install?

No, I'm simply stating that they don't always have a choice.

> > Some sites actually have policies about what software they'll allow on
> > their systems.
> 
> How arrogant.

The issue of whether they "should" have that choice really doesn't
belong here.

> Third party software packages are on MS Windows completely
> independent of each other.

How is this related to the above?

Did you even understand what I was saying?

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Wed, Apr 30, 2014 at 9:51 AM, Glynn Clements
 wrote:
>
> Markus Metz wrote:
>
>> >> > By all means provide fall-backs, workarounds, alternatives, or
>> >> > whatever, but anything which tries to make such things mandatory is
>> >> > going to get reverted. Again.
>> >>
>> >> really nice attitude ;-) Martin
>> >
>> > At least I'm not saying "you ARE going to use our version of Python,
>> > whether you like it or not".
>>
>> People installing GRASS want to use GRASS. They want GRASS to work out
>> of the box. They can use any Python version they want, as long as
>> WinGRASS uses its embedded Python version. Users will not notice it.
>
> You're assuming that users have a free choice as to what they install.

Oh, you say users should not have a free choice of what they install?
Please explain this to MS Windows users who want to give GRASS a try.

On MS Windows, users do have a free choice of what they install. Third
party software packages are on MS Windows completely independent of
each other. If you don't like this, you have to change the MS Windows
world first, then GRASS.

On MS Windows, GRASS and a system-wide Python installation are third
party software packages that are completely independent of each other.

Anything but using GRASS_PYTHON to call Python scripts will cause
trouble on MS WIndows. See this thread.

> Some sites actually have policies about what software they'll allow on
> their systems.

How arrogant. Third party software packages are on MS Windows
completely independent of each other.

As soon as I have enough time to do this, I will implement the
GRASS_PYTHON mechanism. GRASS_PYTHON can always be set to the
system-wide .py file association, whatever that will be, some
incompatible/incomplete Python version, some .bat script (which could
work BTW), a text editor, or some general developer editor or SDK such
as the one from MS .

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Newcomb, Doug]

Hi folks,
A lot of the folks I introduce to GRASS are already GIS users, using a
different GIS software package.  They are unlikely to stop using their
existing software while trying GRASS, have a "spare" computer that would
only have GRASS loaded, or be able to install a virtual instance of windows
to run just GRASS .

In the managed Windows desktop  environment that I work in, there is an
"approved"  mainline proprietary gis that is installed with its own version
of python installed as the system python. I cannot recommend the
installation of any software that interferes with that combination of
software.  The mainline gis, when updated periodically, will also probably
have a new version of python installed, without regards to other geospatial
software that is installed.

If GRASS7 cannot operate reliably in that environment , it is unlikely that
it will be installed in my managed Windows work environment.

I appreciate the ongoing technical discussion and I'm looking forward to a
workable solution.  There are several ways that GRASS7  on Windows could be
( in my personal opinion) an effective tool in my organization.

Doug

On Wed, Apr 30, 2014 at 3:51 AM, Glynn Clements wrote:

>
> Markus Metz wrote:
>
> > >> > By all means provide fall-backs, workarounds, alternatives, or
> > >> > whatever, but anything which tries to make such things mandatory is
> > >> > going to get reverted. Again.
> > >>
> > >> really nice attitude ;-) Martin
> > >
> > > At least I'm not saying "you ARE going to use our version of Python,
> > > whether you like it or not".
> >
> > People installing GRASS want to use GRASS. They want GRASS to work out
> > of the box. They can use any Python version they want, as long as
> > WinGRASS uses its embedded Python version. Users will not notice it.
>
> You're assuming that users have a free choice as to what they install.
> Some sites actually have policies about what software they'll allow on
> their systems.
>
> When it comes to determining those policies, general-purpose
> interpreted languages get a rougher ride than most. Applications which
> bundle private copies of such get an even rougher ride, if they don't
> simply get rejected out of hand.
>
> --
> Glynn Clements 
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>



-- 
Doug Newcomb
USFWS
Raleigh, NC
919-856-4520 ext. 14 doug_newc...@fws.gov
-
The opinions I express are my own and are not representative of the
official policy of the U.S.Fish and Wildlife Service or Dept. of the
Interior.   Life is too short for undocumented, proprietary data formats.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> >> > By all means provide fall-backs, workarounds, alternatives, or
> >> > whatever, but anything which tries to make such things mandatory is
> >> > going to get reverted. Again.
> >>
> >> really nice attitude ;-) Martin
> >
> > At least I'm not saying "you ARE going to use our version of Python,
> > whether you like it or not".
> 
> People installing GRASS want to use GRASS. They want GRASS to work out
> of the box. They can use any Python version they want, as long as
> WinGRASS uses its embedded Python version. Users will not notice it.

You're assuming that users have a free choice as to what they install. 
Some sites actually have policies about what software they'll allow on
their systems.

When it comes to determining those policies, general-purpose
interpreted languages get a rougher ride than most. Applications which
bundle private copies of such get an even rougher ride, if they don't
simply get rejected out of hand.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Moritz Lennert wrote:

> Can everyone agree on that solution (not meaning that you consider it 
> the best solution, but that you can live with it) ?

I can live with anything that doesn't require significant work to undo
(i.e. revert to the state where the platform's native mechanisms are
used without interference, for better or for worse).

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> GRASS has to check each time it is started if a compatible system-wide
> .py file association is available. This is IMHO nonsense.

Performing that test would be nonsense.

Any application depends upon certain functionality being provided by
the underlying platform. There are an unlimited number of ways that
you can break the platform such that GRASS stops working. Singling out
one of them in particular as needing to be checked is just silly.

> Anyway, there are four valid Python installations for MS Windows:
> 
> - Python 3 64 bit
> - Python 3 32 bit
> - Python 2 64 bit
> - Python 2 32 bit
> 
> IMHO, we can not wait until all these versions are supported by the
> same GRASS7 wingrass installer.

I don't care about the installer. I do care about the suggestion of
making changes to core GRASS code such that you would actually have to
modify the internals to have Python scripts executed normally (i.e. in
by the same mechanism as for anything that isn't GRASS).

If I write a C program which executes a Python script using
system("/path/to/script.py"), it *is* going to use the interpreter
specified by the registry's file associations. I expect the same
interpreter to be used when that script is executed by something which
is part of GRASS.

> > Personally, I'd rather just execute the scripts directly, as is done
> > on Unix.
> 
> Huh?

I mean that you pass the path to the script and its arguments to the
execution function (system() or whatever), and leave the platform to
deal with the rest. You don't try to second-guess or bypass the
platform.

E.g. if .py is associated with the Python launcher, I expect to be
able to control which Python version is used via the launcher's
configuration mechanism, without first having to figure out how all
this is being bypassed then un-bypassing it.

> > We're not even demanding that our users use a specific Python package,
> 
> Yes we do. At the moment we require the same Python 2.x 32 bit version
> that is embedded in OSGEO4W and the standalone WinGRASS installer.
> Remember, we are taking about the status quo.

I'm talking about GRASS, not the installer.

I don't particularly care about the installer, as anyone can simply
provide a sane alternative. But they shouldn't have to patch the code,
which would be required if some of the suggestions (e.g. intercepting
grass.script.Popen()) gained traction.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

Hamish wrote:

>I think that's more a reflection on the way things are done in the MS
>Windows world: the last program to install itself wins.

yes

>Which leads to
>the usual 1st trouble shooting step on Windows after seeing if a
>rebooting helps: try uninstalling then reinstalling the program. Then
>the program you are trying to use becomes the last thing to be installed
>and magically wins again (but by breaking the previously installed
>conflicting software). 

a not unusal way of fixing things ;-)


Moritz Lennert wrote
> On 27/04/14 22:52, Markus Metz wrote:
>> I do not understand this ongoing discussion on WinGRASS7+Python. My
>> proposed solution caters for all possibilities, in particular:
>>
>> - an existing incompatible .py file assocation
>> [add your workaround here]
>>
>> - changes in any existing .py file association after GRASS has been
>> installed
>> [add your check-at-startup-mechanism here]
> 
> I think we just have to agree to disagree on this at this stage, and 
> implement a solution that if not satisfying everyone, at least does not 
> dissatisfy anyone more than necessary.
> 
>  From the current discussion I take that the .bat file solution is one 
> that will satisfy those who want GRASS to "just work" on MS Windows 
> without putting any burden on the user. In order to not dissatisfy those 
> who do not want GRASS to impose a specific version of Python, it should 
> be easy to "switch" off the .bat file solution, i.e. by simply resetting 
> PATH and/or erasing the directory with the .bat files.
> 
> Can everyone agree on that solution (not meaning that you consider it 
> the best solution, but that you can live with it) ?

regarding "who do not want GRASS to impose a specific version of Python",
have a look at:

http://trac.osgeo.org/grass/browser/grass/trunk/REQUIREMENTS.html

- Python >= 2.6 (for temporal framework, scripts, wxGUI, and ctypes
interface)
Note: Python 3 support is still in development
[...]
- wxPython >= 2.8.10.1 (for wxGUI)
[...]
- NumPy >= 1.0.4 (for various wxGUI components and pyGRASS)
[...]
- Python dateutil Library ("python-dateutil", needed for the tgrass modules
t.*)
[...]
- Python Imaging Library or PILLOW (highly recommended for wxGUI and
necessary for wxGUI Cartographic Composer)

so there are basically some kind of "minimum version of python and python
package" requirements; this has to be communicated in a clear way to users
"who do not want GRASS to impose a specific version of Python".

regarding "switch" off the .bat file solution, there may be some more work
than just resetting PATH and erasing bat files.



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5137614.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Neteler]

On Mon, Apr 28, 2014 at 9:37 AM, Moritz Lennert
 wrote:
...
> From the current discussion I take that the .bat file solution is one that
> will satisfy those who want GRASS to "just work" on MS Windows without
> putting any burden on the user. In order to not dissatisfy those who do not
> want GRASS to impose a specific version of Python, it should be easy to
> "switch" off the .bat file solution, i.e. by simply resetting PATH and/or
> erasing the directory with the .bat files.
>
> Can everyone agree on that solution (not meaning that you consider it the
> best solution, but that you can live with it) ?

For me that sounds most reasonable given the various scenarios.

markusN
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Moritz Lennert]

On 27/04/14 22:52, Markus Metz wrote:

I do not understand this ongoing discussion on WinGRASS7+Python. My
proposed solution caters for all possibilities, in particular:

- an existing incompatible .py file assocation
[add your workaround here]

- changes in any existing .py file association after GRASS has been installed
[add your check-at-startup-mechanism here]


I think we just have to agree to disagree on this at this stage, and 
implement a solution that if not satisfying everyone, at least does not 
dissatisfy anyone more than necessary.


From the current discussion I take that the .bat file solution is one 
that will satisfy those who want GRASS to "just work" on MS Windows 
without putting any burden on the user. In order to not dissatisfy those 
who do not want GRASS to impose a specific version of Python, it should 
be easy to "switch" off the .bat file solution, i.e. by simply resetting 
PATH and/or erasing the directory with the .bat files.


Can everyone agree on that solution (not meaning that you consider it 
the best solution, but that you can live with it) ?


Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Sun, Apr 27, 2014 at 7:34 PM, Hamish  wrote:

> Markus Metz wrote:
> > ESRI assumes that their system-wide Python installation will never
> > change. That is rather ignorant. I don't think GRASS should mimik the
> > aggnorant ignorance of ESRI.
>
> I think that's more a reflection on the way things are done in the MS
> Windows world: the last program to install itself wins. Which leads to
> the usual 1st trouble shooting step on Windows after seeing if a
> rebooting helps: try uninstalling then reinstalling the program. Then
> the program you are trying to use becomes the last thing to be installed
> and magically wins again (but by breaking the previously installed
> conflicting software). Of course with Linux rpm/deb package managed
> system uninstalling and reinstalling typically does not change
> anything, you usually would have more luck renaming away ~/.foorc/
> and starting fresh that way.
>

This is a good point as well as Markus's note about .py being associated
with editor instead of interpreter. Installing some general developer
editor or SDK such as the one from MS, always associates whatever
extensions it knows about and then later another tool does the same, time
to time you can opt-out, but this is seldom true.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Hamish]

Markus Metz wrote:
> ESRI assumes that their system-wide Python installation will never
> change. That is rather ignorant. I don't think GRASS should mimik the
> aggnorant ignorance of ESRI.

I think that's more a reflection on the way things are done in the MS
Windows world: the last program to install itself wins. Which leads to
the usual 1st trouble shooting step on Windows after seeing if a
rebooting helps: try uninstalling then reinstalling the program. Then
the program you are trying to use becomes the last thing to be installed
and magically wins again (but by breaking the previously installed
conflicting software). Of course with Linux rpm/deb package managed
system uninstalling and reinstalling typically does not change
anything, you usually would have more luck renaming away ~/.foorc/
and starting fresh that way.



Hamish

ps- wrt fubared email header handling explained in the links below, I
notice gmail is now complaining about gmail addresses resent from
Mailman too. (your post MarkusM showed up with a spoof warning in
gmail) See the last link for Mailman config workaround (they just put
out a new band-aid release).


-- 
Hamish 
.
Thought I should join the Yahoo mail diaspora before 30 days
worth of my emails got flushed from everyone's spam boxes never
to be seen again. In the last weeks some have made it to the ML
archives at least, if not to end recipients. Others seem to have
just disappeared into /dev/null. It didn't help that the web
interface had become an unusable gibberish of broken JavaScript
and their IMAP would only transfer the oldest 4% of my inbox.
.
http://www.ietf.org/mail-archive/web/ietf/current/msg87153.html
http://www.spamresource.com/2014/04/up-in-arms-about-yahoos-dmarc-policy.html
http://wiki.list.org/pages/viewpage.action?pageId=17891458
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

I do not understand this ongoing discussion on WinGRASS7+Python. My
proposed solution caters for all possibilities, in particular:

- an existing incompatible .py file assocation
[add your workaround here]

- changes in any existing .py file association after GRASS has been installed
[add your check-at-startup-mechanism here]

Moritz Lennert:
> Markus Metz:
>>
>> [...] An existing system Python on MS Windows can change or disappear
>> any time, and a GRASS installation will not be notified about this
>> change.
>
> But that's exactly the same on any OS, including GNU/Linux. It is up to the
> user/sysadmin/packagers to ensure that Python is installed. But I can
> uninstall Python at any time on my system and GRASS will never be notified.

On GNU/Linux|*BSD|UNIX, this would be a packaging error. Python must
be a dependency for GRASS7. The package manager will take care of it.
As long as you have installed GRASS using the system's package manager
(who takes the hand of the user and takes care of everything).

Glynn Clements wrote:
>
> Markus Metz wrote:
>> [...] an existing system Python on MS Windows can change or disappear
>> any time, and a GRASS installation will not be notified about this
>> change.
>
> Yes; and your point is?

GRASS has to check each time it is started if a compatible system-wide
.py file association is available. This is IMHO nonsense.

As long as you do not provide this test, I will revert any change that
enforces the use of system-wide .py file assocations.

>
> If my system Python were to suddenly change or disappear, I'd be far
> more concerned about "who did that, and how?" than about GRASS not
> working.

Ask your sysadmin.

Anyway, there are four valid Python installations for MS Windows:
- Python 3 64 bit
- Python 3 32 bit
- Python 2 64 bit
- Python 2 32 bit

IMHO, we can not wait until all these versions are supported by the
same GRASS7 wingrass installer.

>> How about ignoring .py file associations and set GRASS_PYTHON to the
>> system's Python? If that does not work or disappeared, make
>> GRASS_PYTHON fall back to the embedded version? IOW, always use
>> GRASS_PYTHON, but let GRASS_PYTHON be the system's Python if possible.
>
> 1. GRASS_PYTHON is just a pathname (it needs to be quoted in the batch
> file because it may contain spaces), while the registry entry can
> specify switches. This could be worked around by setting GRASS_PYTHON
> to a batch file, but ...

In this case GRASS_PYTHON must be a path name, something like asking
the user to select one of several possibilities.
>
> 2. Batch files just add another source of potential problems. E.g.
> anything beyond the simplest cases fails on non-ASCII data (more
> precisely, whenever you use bytes which don't have identical
> interpretations in the "ANSI" and "OEM" codepages).
>
> Personally, I'd rather just execute the scripts directly, as is done
> on Unix.

Huh?

On MS Windows, .exe and .bat files are executed directly, but not .py
files. Any .py files are not executed on MS Windows, they are an
argument to the .py file's registry association. Ideally, .py files
are an argument to the system-wide Python Installation, whichever that
will be, but probably (75%) not compatible. The default .py file
association can also be a text editor wchich makes perfect sense
(probably not only) on MS Windows.

Also on *NIX systems, the scripts are not executed directly, instead
the shebang "#!/usr/bin/env python" is used which could be changed to
"#!/usr/bin/env python2" as long as GRASS7 does not comply with
Python3.

>
>> The downside is, GRASS would need a new mechanism to test if python
>> scripts can be executed on the current system. This mechanism would
>> need to be run before any GRASS modules. We would also need clear
>> messages to users about any problems and how these problems can be
>> resolved. Further on, any solution should not require administrative
>> privileges.
>>
>> Anyway, IMHO we really must have a mechanism to avoid the system's .py
>> associations because an existing Python file association on MS Windows
>> can change or disappear any time, and a GRASS installation will not be
>> notified about this change.
>
> ESRI don't appear to be concerned about this "problem", and their
> users are paying them (rather heavily, I might add). And they don't
> even have to consider portability; it's Windows-only.

ESRI assumes that their system-wide Python installation will never
change. That is rather ignorant. I don't think GRASS should mimik the
aggnorant ignorance of ESRI.

>
> We're not even demanding that our users use a specific Python package,

Yes we do. At the moment we require the same Python 2.x 32 bit version
that is embedded in OSGEO4W and the standalone WinGRASS installer.
Remember, we are taking about the status quo.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Sat, Apr 26, 2014 at 1:16 PM, Glynn Clements
 wrote:
>
> Martin Landa wrote:
>
>> > By all means provide fall-backs, workarounds, alternatives, or
>> > whatever, but anything which tries to make such things mandatory is
>> > going to get reverted. Again.
>>
>> really nice attitude ;-) Martin
>
> At least I'm not saying "you ARE going to use our version of Python,
> whether you like it or not".

People installing GRASS want to use GRASS. They want GRASS to work out
of the box. They can use any Python version they want, as long as
WinGRASS uses its embedded Python version. Users will not notice it.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

wenzeslaus wrote
> Please consider this scenario:
> 
> 1. user installs ArcGIS which installs Python system-wide
> 2. user installs GRASS GIS which will use system-wide Python

2. may or may not work out of the box ... there are a lot of uncertain
issues to consider (version, dependencies, etc. etc.)



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5137324.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Sat, Apr 26, 2014 at 6:42 AM, Glynn Clements wrote:

> > >> It's easier to check if the file is a python script and if so than
> > >> to force to use bundled version of Python.
> > >
> > > So long as I have commit access, GRASS isn't going to be "forcing" the
> > > use of a non-system Python.
> >
> > But an existing system Python on MS Windows can change or disappear
> > any time, and a GRASS installation will not be notified about this
> > change.
>
> Yes; and your point is?
>
> If my system Python were to suddenly change or disappear, I'd be far
> more concerned about "who did that, and how?" than about GRASS not
> working.


Please consider this scenario:

1. user installs ArcGIS which installs Python system-wide
2. user installs GRASS GIS which will use system-wide Python
3. user is happy with free and open source GIS, so user uninstalls ArcGIS
4. GRASS does not work
5 a. user does not understand and returns to ArcGIS
5 b. user understands that GRASS depends on ArcGIS and returns to ArcGIS
5 c. user asks on mailing list and installs Python or re-installs GRASS
which would install Python system-wide (nice but how probable?)
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Martin Landa]

2014-04-26 13:16 GMT+02:00 Glynn Clements :

> At least I'm not saying "you ARE going to use our version of Python,
> whether you like it or not".

please ask the Windows users, 99.9% they don't care or they will not
understand your question. The question is completely pointless. The
most of users care about simple installation and working software,
nothing else. Thanks completely OK.

If you are not able to change your opinion you are probably the only
one who can implement your ideas in winGRASS installer (including all
Python dependencies which we use [1,2]). Otherwise it's just
never-ending so-called discussion with *no* result. If you going to
revert a commit you should simply introduce a better solution.
Otherwise it's just blocking us (the community).

Martin

[1] http://download.osgeo.org/osgeo4w/release/grass/grass70/setup.hint
[2] http://download.osgeo.org/osgeo4w/release/grass/msys-grass/setup.hint

-- 
Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Martin Landa wrote:

> > By all means provide fall-backs, workarounds, alternatives, or
> > whatever, but anything which tries to make such things mandatory is
> > going to get reverted. Again.
> 
> really nice attitude ;-) Martin

At least I'm not saying "you ARE going to use our version of Python,
whether you like it or not".

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> >> It's easier to check if the file is a python script and if so than
> >> to force to use bundled version of Python.
> >
> > So long as I have commit access, GRASS isn't going to be "forcing" the
> > use of a non-system Python.
> 
> But an existing system Python on MS Windows can change or disappear
> any time, and a GRASS installation will not be notified about this
> change.

Yes; and your point is?

If my system Python were to suddenly change or disappear, I'd be far
more concerned about "who did that, and how?" than about GRASS not
working.

> > By all means provide fall-backs, workarounds, alternatives, or
> > whatever, but anything which tries to make such things mandatory is
> > going to get reverted. Again.
> 
> How about ignoring .py file associations and set GRASS_PYTHON to the
> system's Python? If that does not work or disappeared, make
> GRASS_PYTHON fall back to the embedded version? IOW, always use
> GRASS_PYTHON, but let GRASS_PYTHON be the system's Python if possible.

1. GRASS_PYTHON is just a pathname (it needs to be quoted in the batch
file because it may contain spaces), while the registry entry can
specify switches. This could be worked around by setting GRASS_PYTHON
to a batch file, but ...

2. Batch files just add another source of potential problems. E.g. 
anything beyond the simplest cases fails on non-ASCII data (more
precisely, whenever you use bytes which don't have identical
interpretations in the "ANSI" and "OEM" codepages).

Personally, I'd rather just execute the scripts directly, as is done
on Unix.

> The downside is, GRASS would need a new mechanism to test if python
> scripts can be executed on the current system. This mechanism would
> need to be run before any GRASS modules. We would also need clear
> messages to users about any problems and how these problems can be
> resolved. Further on, any solution should not require administrative
> privileges.
> 
> Anyway, IMHO we really must have a mechanism to avoid the system's .py
> associations because an existing Python file association on MS Windows
> can change or disappear any time, and a GRASS installation will not be
> notified about this change.

ESRI don't appear to be concerned about this "problem", and their
users are paying them (rather heavily, I might add). And they don't
even have to consider portability; it's Windows-only.

We're not even demanding that our users use a specific Python package,
only a relatively recent official Python 2.x release (i.e. something
which actually deserves to "own" the .py association). Or something
which is at least minimally compatible.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Fri, Apr 25, 2014 at 4:18 AM, Moritz Lennert <
mlenn...@club.worldonline.be> wrote:

> On 25/04/14 09:36, Markus Metz wrote:
>
>> On Fri, Apr 25, 2014 at 5:18 AM, Glynn Clements
>>  wrote:
>>
>>>
>>> Martin Landa wrote:
>>>
>>>  It's easier to check if the file is a python script and if so than
 to force to use bundled version of Python.

>>>
>>> So long as I have commit access, GRASS isn't going to be "forcing" the
>>> use of a non-system Python.
>>>
>>
>> But an existing system Python on MS Windows can change or disappear
>> any time, and a GRASS installation will not be notified about this
>> change.
>>
>> I am not opposed to using the system's Python, I just don't trust it.
>>
>
> But that's exactly the same on any OS, including GNU/Linux. It is up to
> the user/sysadmin/packagers to ensure that Python is installed. But I can
> uninstall Python at any time on my system and GRASS will never be notified.
>
> On my Linux I cannot uninstall Python without notifying (i.e.
uninstalling) all other dependent packages. The only GRASS which will not
know about it is the one I compiled myself. If I compiled myself, I
probably understand the issues, so nothing to worry about from point of
view of GRASS developer. While on MS Windows user can be in the same
situation as I'm with compiled GRASS with the difference that he or she is
installed standard version and thus is possibly very unexperienced.

I just wanted to point out that there is a significant difference thanks to
packaging (and packagers!). No historical reasons, this is how it is.


> So the issue is that for historical reasons of usage, we don't want the
> Windows user to have that responsibility, while we trust the Linux user
> with it, partly because he is helped by packagers.
>


> Moritz
>
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Moritz Lennert]

On 25/04/14 09:36, Markus Metz wrote:

On Fri, Apr 25, 2014 at 5:18 AM, Glynn Clements
 wrote:


Martin Landa wrote:


It's easier to check if the file is a python script and if so than
to force to use bundled version of Python.


So long as I have commit access, GRASS isn't going to be "forcing" the
use of a non-system Python.


But an existing system Python on MS Windows can change or disappear
any time, and a GRASS installation will not be notified about this
change.

I am not opposed to using the system's Python, I just don't trust it.


But that's exactly the same on any OS, including GNU/Linux. It is up to 
the user/sysadmin/packagers to ensure that Python is installed. But I 
can uninstall Python at any time on my system and GRASS will never be 
notified.


So the issue is that for historical reasons of usage, we don't want the 
Windows user to have that responsibility, while we trust the Linux user 
with it, partly because he is helped by packagers.


Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Martin Landa]

2014-04-25 5:18 GMT+02:00 Glynn Clements :
> By all means provide fall-backs, workarounds, alternatives, or
> whatever, but anything which tries to make such things mandatory is
> going to get reverted. Again.

really nice attitude ;-) Martin

-- 
Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Martin Landa]

Hi,

2014-04-25 9:36 GMT+02:00 Markus Metz :
>>> It's easier to check if the file is a python script and if so than
>>> to force to use bundled version of Python.
>>
>> So long as I have commit access, GRASS isn't going to be "forcing" the
>> use of a non-system Python.
>
> But an existing system Python on MS Windows can change or disappear
> any time, and a GRASS installation will not be notified about this
> change.
>
> I am not opposed to using the system's Python, I just don't trust it.

I strongly agree with that! Martin
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Fri, Apr 25, 2014 at 5:18 AM, Glynn Clements
 wrote:
>
> Martin Landa wrote:
>
>> It's easier to check if the file is a python script and if so than
>> to force to use bundled version of Python.
>
> So long as I have commit access, GRASS isn't going to be "forcing" the
> use of a non-system Python.

But an existing system Python on MS Windows can change or disappear
any time, and a GRASS installation will not be notified about this
change.

I am not opposed to using the system's Python, I just don't trust it.
>
> By all means provide fall-backs, workarounds, alternatives, or
> whatever, but anything which tries to make such things mandatory is
> going to get reverted. Again.

How about ignoring .py file associations and set GRASS_PYTHON to the
system's Python? If that does not work or disappeared, make
GRASS_PYTHON fall back to the embedded version? IOW, always use
GRASS_PYTHON, but let GRASS_PYTHON be the system's Python if possible.

The downside is, GRASS would need a new mechanism to test if python
scripts can be executed on the current system. This mechanism would
need to be run before any GRASS modules. We would also need clear
messages to users about any problems and how these problems can be
resolved. Further on, any solution should not require administrative
privileges.

Anyway, IMHO we really must have a mechanism to avoid the system's .py
associations because an existing Python file association on MS Windows
can change or disappear any time, and a GRASS installation will not be
notified about this change.

>
> Batch files aren't a problem, as they can just be deleted or the
> directory removed from PATH.

Sure, the idea is to have any wrappers in one directory and the actual
python scripts in another directory. Then it is just a matter of
setting PATH accordingly: hide the one directory, include the other
directory. run_command() etc can stay as they are right now.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Martin Landa wrote:

> It's easier to check if the file is a python script and if so than
> to force to use bundled version of Python.

So long as I have commit access, GRASS isn't going to be "forcing" the
use of a non-system Python.

By all means provide fall-backs, workarounds, alternatives, or
whatever, but anything which tries to make such things mandatory is
going to get reverted. Again.

Batch files aren't a problem, as they can just be deleted or the
directory removed from PATH.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Hamish]

Glynn:

>>>  Batch files are a better solution, as they don't affect anything else,
>>>  and it's a simple matter to either delete them or modify PATH to
>>>  ignore them, and use the Python scripts directly.

MMetz:
>>  Sounds good.

Martin:
> I am not sure, what about user scripts?

a new g.batwrap helper program (written in C) to create a .bat wrapper and
put it in the same dir as the target python script?

> It's easier to check if the file is a python script and if so than to
> force to use bundled version of Python. This seems to be a better
> workarround than using bat files.

what would do the checking, g.parser.exe? remember that any solution also needs 
to work 100% from the command line too, so having the GUI's pseudo-command line 
taking care of those things is not a real solution. Even if a user never uses 
the command line on Windows, they will at some point run into the old 
script-called-from-another-script problem.

?


regards,
Hamish

ps- this part of g.extension.py is fundamentally broken:
  line = line.replace("%GISBASE%", "%GRASS_ADDON_PATH%") # options['prefix'])

It's equivalent to trying to do  s+/usr/bin/+$PATH/+  and then run 
"$PATH/g.module". The approach used in g.extension.sh is to split 
$GRASS_ADDON_PATH and use the first dir in the list as "$GRASS_ADDON_PATH1", 
and then use that dir as the first-choice userland exedir destination.

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Martin Landa]

Hi,

2014-04-23 21:56 GMT+02:00 Markus Metz :

>> Batch files are a better solution, as they don't affect anything else,
>> and it's a simple matter to either delete them or modify PATH to
>> ignore them, and use the Python scripts directly.
>
> Sounds good.

I am not sure, what about user scripts? It's easier to check if the
file is a python script and if so than to force to use bundled version
of Python. This seems to be a better workarround than using bat files.
Martin


-- 
Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Wed, Apr 23, 2014 at 1:37 AM, Glynn Clements
 wrote:
>
> Markus Metz wrote:
>
>> Therefore it is IMHO not a good idea to rely on a system-wide Python
>> file association on MS Windows,
>
> Regardless of whether or not it's a good idea, it's not entirely
> avoidable.

Sure it is.
>
> If a process "executes" a command using the system's interfaces
> (system(), CreateProcess(), subprocess.Popen(), batch files, etc), and
> that command refers to a Python script, the registry's .py file
> association *will* be used.

You ignore my suggestion to use .bat files calling the embedded script
interpreter, like in GRASS 6.

>
>> and therefore I propose to use the
>> embedded Python version already coming for some years with the
>> WinGRASS installer.
>
> You can only force the use of a specific Python interpreter when
> you control the execution mechanism. Otherwise, the system's file
> associations will be used.

You ignore my suggestion to use .bat files calling the embedded script
interpreter, like in GRASS 6.

>
> Users who aren't willing to live within whatever walled garden you
> provide will have to have a valid system-wide Python installation. In
> that situation, that installation should be used always, not just for
> those cases where you can't interpose your workaround.

You missed the point. I do not want to use a walled garden. Just set
the GRASS environment as currently done and off you go (with .bat
files, not with .py files).

>>
>> I suspect the whole discussion boils down to the question whether we
>> can rely on GRASS-compatible Python file associations on MS Windows or
>> not. I say, we can not.

[Your answer here]

>
> For me, the discussion boils down to:
>
> 1. How invasive is this workaround going to be with regard to the rest
> of GRASS?

For using system-wide .py file associations, nobody tried, so nobody
knows, thus maximal.
>
> 2. How easy is it to disable this workaround and just use the system
> Python instead?.

IMHO an invalid question because an existing Python file association
on MS Windows can change or disappear any time, and a GRASS
installation will not be notified about this change.

>
> Batch files are a better solution, as they don't affect anything else,
> and it's a simple matter to either delete them or modify PATH to
> ignore them, and use the Python scripts directly.

Sounds good.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> There seems to be agreement that changing an existing Python file
> association is not a good idea.

The problem is that every possible solution has some aspect which is
"not a good idea".

> An existing Python file association on MS Windows can change or
> disappear any time, and a GRASS installation will not be notified
> about this change.

Note that the same is also true of the .bat file association. On Unix,
"python" may change from 2.x to 3.x.

There is a limit to how far we can (or should) go to protect against
defects in the underlying platform. Every workaround has an associated
cost.

> Therefore it is IMHO not a good idea to rely on a system-wide Python
> file association on MS Windows,

Regardless of whether or not it's a good idea, it's not entirely
avoidable.

If a process "executes" a command using the system's interfaces
(system(), CreateProcess(), subprocess.Popen(), batch files, etc), and
that command refers to a Python script, the registry's .py file
association *will* be used.

> and therefore I propose to use the
> embedded Python version already coming for some years with the
> WinGRASS installer.

You can only force the use of a specific Python interpreter when
you control the execution mechanism. Otherwise, the system's file
associations will be used.

Users who aren't willing to live within whatever walled garden you
provide will have to have a valid system-wide Python installation. In
that situation, that installation should be used always, not just for
those cases where you can't interpose your workaround.

> With .bat file wrappers, you would not even need to set a particular
> python environment, it would just work. Just as on any other OS, all
> GRASS commands would be available after running the grassxy command.
> If, as Glynn suggests, running grassxy should not be required, instead
> the GRASS environment is set permanently at login time, there would be
> no difference, except that you don't need to bother about funny
> (non-)existing Python file associations on MS Windows.
> 
> I suspect the whole discussion boils down to the question whether we
> can rely on GRASS-compatible Python file associations on MS Windows or
> not. I say, we can not.

For me, the discussion boils down to:

1. How invasive is this workaround going to be with regard to the rest
of GRASS?

2. How easy is it to disable this workaround and just use the system
Python instead?.

IMHO, "solutions" such as making run_command() detect Python scripts
and apply special treatment are non-starters on both fronts.

Batch files are a better solution, as they don't affect anything else,
and it's a simple matter to either delete them or modify PATH to
ignore them, and use the Python scripts directly.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

A short comment about GRASS as a monolithic application or not:
On every OS where I tested various GRASS versions (I tested on various
Linux distros, on various FreeBSD versions, on various NetBSD
versions, on 2 Solaris versions and helped give IBM AIX a try) and on
all these OS's I started GRASS with the grassxy command, usually not
bothering about a GUI but using the command line exclusively. All I
propose is that GRASS on Windows behaves the same, and that all GRASS
commands work, no matter if they are real executables or some scripts.

I hope we can agree on that.

I think this discussion has gone astray. This is about how GRASS
scripts are executed on Windows, not whether GRASS on WIndows should
behave any different than on other OS's.

There seems to be agreement that changing an existing Python file
association is not a good idea.
An existing Python file association on MS Windows can change or
disappear any time, and a GRASS installation will not be notified
about this change.
Therefore it is IMHO not a good idea to rely on a system-wide Python
file association on MS Windows, and therefore I propose to use the
embedded Python version already coming for some years with the
WinGRASS installer.

With .bat file wrappers, you would not even need to set a particular
python environment, it would just work. Just as on any other OS, all
GRASS commands would be available after running the grassxy command.
If, as Glynn suggests, running grassxy should not be required, instead
the GRASS environment is set permanently at login time, there would be
no difference, except that you don't need to bother about funny
(non-)existing Python file associations on MS Windows.

I suspect the whole discussion boils down to the question whether we
can rely on GRASS-compatible Python file associations on MS Windows or
not. I say, we can not.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Helmut Kudrnovsky wrote:

> not including python, have a look here:
> 
> http://trac.osgeo.org/grass/browser/grass/trunk/mswindows/GRASS-Packager.bat.tmpl#L113
> 
>   @echo Copy Python content to PACKAGE_DIR\Python27
> 
> and then maybe you have to adapt the starting script and all other ?!

I'm fairly sure you'll have to adapt something. Simply dumping a copy
of Python onto the disk won't cause it to be used, so something must
be setting up e.g. PATH to use it. That would need to change.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Moritz Lennert wrote:

> the argument that GRASS4Win should be a different beast 
> than GRASS4NIX in that it should become a monolithic integrated 
> application, which, IMHO, fundamentally alters the very nature of what 
> GRASS is.

There's no fundamental reason why we can't have "Lite" and "Full"
versions.

I don't have a problem with having a "Lite" version; it's the lack of
a "Full" version that I have a problem with.

The main point is that GRASS should not automatically switch to
"idiot mode" upon detecting that it's running on Windows.

> But whatever the solution, unless we want to turn GRASS4Win into a 
> different application than GRASS on other platforms, there will have to 
> be a minimal amount of intervention by the user to chose how she wants 
> to handle Python on her machine.

Well, in theory at least, we could adopt ESRI's "solution" and just
force "our" Python to be the system Python, stomping on any existing
installation.

While I'm not recommending that approach, it's worth noting that I'm
(apparently) not the only one who considers having a "system Python"
to be a necessity. And ESRI doesn't even need to maintain
compatibility between platforms, ArcGIS being Windows-only.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

>So, there is a vision by some that for the reasons of apparent
>"incompatibility" between the way GRASS works and Windows, 
>GRASS has to
>be different on windows. I.e. that GRASS has to become a monolithic
>application on Windows. 

I wouldn't call it incompatibility and I think GRASS hasn't to be different
on windows. 

I like mounteering, in winter and in summer; it's always
climbing, but depending of the season, you need different/other clothes,
tools etc ... so
maybe windows is some kind of icy winter time ... ;-)

>I also think that some of the debate comes from the question of whether
>the wxGUI should have a special status or whether it should just be
>treated as one of the hundreds of modules that GRASS offers. 

maybe; but what about the scripts?

>And AFAIK, no one has suggested that the GRASS installer should just
>install Python system-wide, but rather that a correct Python
>installation would be left to the user, with possible help in the
>installer (e.g. suggesting installation if it detects no installation at
>all, clear documentation on different cases, etc). 
[...]
>There is quite a large margin between creating a monolithic application
>with everything bundled and leaving the user alone... 

what will you suggest to the user, if there is already an system wide python
(installed
by another software, sometimes adapted to their needs), but incompatible for
winGRASS?

deinstall/break the other software and install winGRASS?

some operating system design flaws comes in here  ... which we can't change
(... just remember dll-hell :-))

>To answer Martin's call for concrete action: how difficult would it be
>for you, Helmut, to create a GRASS installer without Python, so that we
>can test that scenario ? I don't have the Windows build toolchain set up
>at the moment, but if it's more work for you than a few lines of code to
>change and command to create a new package, then I'll try to find the
>time to do that. 

not including python, have a look here:

http://trac.osgeo.org/grass/browser/grass/trunk/mswindows/GRASS-Packager.bat.tmpl#L113

@echo Copy Python content to PACKAGE_DIR\Python27

and then maybe you have to adapt the starting script and all other ?! is
having a starting script monolithic or non-monolithic? ;-)

and if you want winGRASS operating really from everywhere in windows, you
have to add entries accordingly to the windows registry.






-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5134852.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Martin Landa]

2014-04-14 9:53 GMT+02:00 Moritz Lennert :
> There is quite a large margin between creating a monolithic application with
> everything bundled and leaving the user alone...
>
> To answer Martin's call for concrete action: how difficult would it be for
> you, Helmut, to create a GRASS installer without Python, so that we can test
> that scenario ? I don't have the Windows build toolchain set up at the
> moment, but if it's more work for you than a few lines of code to change and
> command to create a new package, then I'll try to find the time to do that.

I simply miss a reason. I would say that it will be more than one line
to change. Good luck if you want to spend energy/time on that. Martin

-- 
Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Moritz Lennert]

On 14/04/14 00:21, Helmut Kudrnovsky wrote:

Hi Moritz,


1) Should GRASS be the same (i.e. feature parity) on all platforms ?


for sure, why not?


Not to single out MarkusM, but just since he formulated it so clearly:

>On 09/04/14 08:16, Markus Metz wrote:
>> On Windows, GRASS should behave like a
>> self-contained GUI application. There were even suggestions to remove
>> the command line interface completely from winGRASS because it
>> confuses users.


keep in mind windows and *nix are working as operating
systems sometimes in the same way and sometimes in a (quite) different way.

IMHO the discussion is not/should not be about feature parity etc.; it is
about how to handle/tackle as
software project these differences how operating systems do their job with
monolithic/non-monolithic applications, file extensions associations, global
system variables etc. etc.



So, there is a vision by some that for the reasons of apparent 
"incompatibility" between the way GRASS works and Windows, GRASS has to 
be different on windows. I.e. that GRASS has to become a monolithic 
application on Windows.


I also think that some of the debate comes from the question of whether 
the wxGUI should have a special status or whether it should just be 
treated as one of the hundreds of modules that GRASS offers.




I try to outline some scenarios with software using/depending on python in
the windows side ;-)
of the world which I effectively know in reality:

(a)

software A bundles/embeds python X.x.x
software B bundles/embeds python X.x.x

they can be installed in whatever order

(b)

software A bundles/embeds python X.x.x
software B bundles/embeds python Y.y.y



[snip]


(d)

software A installs system wide python X.x.x
or
software A installs system wide python X.x.y
AND
software B installs system wide python X.x.x
or
software B installs system wide python Y.y.y

...and a lot other variations are already out in the windows world ...

a system wide installation in windows defines the file extension
associations, global system variables, etc.;
additionally the applications need maybe the same site-packages or different
ones (and not always the same versions)

as a system wide installation defines the file extension associations,
global system variables, the installation order A then B or B then A may
break the installation of the other software and so on...


As has been mentioned often enough: there is a big difference between 
types of software, between monolithic GUI applications and others.


And AFAIK, no one has suggested that the GRASS installer should just 
install Python system-wide, but rather that a correct Python 
installation would be left to the user, with possible help in the 
installer (e.g. suggesting installation if it detects no installation at 
all, clear documentation on different cases, etc).



I'm pretty sure, if we - the GRASS project - let potential users in the
windows world alone to tackle/handle all these challenges of different
/installations versions etc., we will lose these people ... and this would
be a really pity!!!


There is quite a large margin between creating a monolithic application 
with everything bundled and leaving the user alone...


To answer Martin's call for concrete action: how difficult would it be 
for you, Helmut, to create a GRASS installer without Python, so that we 
can test that scenario ? I don't have the Windows build toolchain set up 
at the moment, but if it's more work for you than a few lines of code to 
change and command to create a new package, then I'll try to find the 
time to do that.


Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Moritz Lennert]

On 13/04/14 20:33, Martin Landa wrote:

2014-04-13 17:03 GMT+02:00 Moritz Lennert :

On 11/04/14 13:05, Martin Landa wrote:
I don't think this a very helpful attitude. This whole debate is not about
one bad guy keeping the rest of the good guys from moving forward. First of
all, Glynn is not alone in his position.


nobody is saying anything about a bad gay, it's your feeling not mine
I can say. The fact is that this debate started more than one year ago
(probably two years ago or more) with *no* consensus or accepted
*solution*. I was playing with virtualenv, python launcher. I will be
more than satisfied if we find acceptable solution. We need to
continue and focus our energy on other important topics or we just
keep discussing this topic for other two or three years and blocking
any of releases. If you find a better solution in the near future we
can easily to integrate it and avoid current dirty workarounds. I am
looking forward to see real-working-solid solution as the outcome of
this debate. Till now I can hardy see it.


I believe, and that's what I've been trying to get at, that our problem 
is not finding the solution, but defining the problem.


If we want to build a more monolithic GRASSWin then some solutions are 
on the table, but they go against the aim of allowing to run GRASS as a 
library of tools which is available from everywhere, not only from 
within the monolithic tool. For this latter aim, some solutions are on 
the table, but they go against the aim of creating a one-click installer 
for GRASS which "just works" without any need for the user to understand 
issues of system-wide Python installation.


So, unless we decide which of these aims (i.e. problems) we make our 
priority, we will never be able to decide on a solution.


Moritz

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Moritz Lennert]

On 13/04/14 20:32, Jürgen E. Fischer wrote:

And IMHO the broader topics would have been better addressed in (a) separate
thread(s) and not side-track this one.  Your original problem got out of focus,
but still isn't solved, is it?


IMHO, the broader topics are the core of this thread !
;-)

Moritz

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Sun, Apr 13, 2014 at 12:52 PM, Benjamin Ducke wrote:

> But please: Let's keep (minor) version-specific Python support
> and all of its associated problems out of the core release.
>
> I really believe that this whole conundrum was caused by not
> taking that step earlier.
>

The problem is caused by not clear selection of Python, its DLL and
packages in GRASS session, so than two Python installations can be mixed.
This together with *internal* incompatibility between Python minor versions
causes the problem. One of the following minor Python versions fixed this
internal incompatibility.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

Hi Moritz,

>1) Should GRASS be the same (i.e. feature parity) on all platforms ?

for sure, why not?

>2) How much less effort can we ask of Windows users than we ask of users 
>of other platforms ?
>
>GRASS was developed in a time when Windows didn't even exist. And it was 
>developed in a very typical *nix style and form. This makes for its 
>special character and for use forms which are quite different from most 
>recent programs. For long years the only way GRASS could be run on 
>Windows was by imitating a *nix environment using cygwin.
>
>Growth in interest in free software, accompanied by a continued 
>dominance of Windows on the desktop, and the desire of some of us to 
>provide the possibility for our colleagues to run GRASS on their Windows 
>machines prompted the beginning of a long period of making GRASS 
>compilable and runnable natively in Windows. And the switch to Python 
>for scripts was, amongst others, motivated by the desire to make these 
>scripts more platform-independent.
>
>At no point, however, was there ever the question of creating a special 
>Windows version of GRASS, a version that would function differently from 
>the *nix version.

keep in mind windows and *nix are working as operating
systems sometimes in the same way and sometimes in a (quite) different way.

IMHO the discussion is not/should not be about feature parity etc.; it is
about how to handle/tackle as
software project these differences how operating systems do their job with
monolithic/non-monolithic applications, file extensions associations, global
system variables etc. etc.

> IIUC, this latter desire seems to have come from 
>experiences of some of us with windows users who were not accustomed to 
>*nix-style programs/libraries/toolboxes. Thus the debate about whether 
>or not we should even show a command line environment outside the GUI 

I don't support to drop the command line in windows and I don't see a reason
why we should do this!

>and thus, now, the argument that GRASS4Win should be a different beast 
>than GRASS4NIX in that it should become a monolithic integrated 
>application, which, IMHO, fundamentally alters the very nature of what 
>GRASS is. 

see my comments two paragraphs above.

>Generally, to put it more bluntly, there seems to be an 
>attempt to dumb GRASS down for Windows users, who, on average, are less 
>computer-savvy than *nix-users (as a side note, this argument is quickly 
>losing its grounds when I see the number of GNU/Linux uptakes around me, 
>especially now in the wake of the discontinuation of XP-support). The 
>idea is that *nix users can solve Python installation issues so as to 
>have their system-wide python work with GRASS (to be totally honest life 
>is a bit easier for them thanks to the /bin/env solution).
>
>[Side note: The same discussion should also constantly be held about 
>functionality which is specific to the GUI, because specifically 
>developed for it), i.e. not just a GUI layer above command line 
>functionality, something I'm afraid to see creep in more and more...]
>
>Now, as Glynn has pointed out, with distributions soon to ship Python3 
>as default (and I'll throw in: with the *nix crowd changing), we will 
>quite probably see similar issues in *nix installation as we are seeing 
>now in Windows, so the question is how to find a solution which will 
>work across all platforms without the need for constant manual 
>intervention by developers to solve yet another special case. And, IMHO, 
>part of this solution will be through trusting a GRASS user on Windows 
>to be capable of following basic instructions concerning the steps to 
>follow to install GRASS on their system.

I try to outline some scenarios with software using/depending on python in
the windows side ;-) 
of the world which I effectively know in reality:

(a)

software A bundles/embeds python X.x.x
software B bundles/embeds python X.x.x

they can be installed in whatever order

(b)

software A bundles/embeds python X.x.x
software B bundles/embeds python Y.y.y

they can be installed in whatever order

(c)

software A bundles/embeds python X.x.x
software B installs systemwide python X.x.x

they may/may not interfere, depending of file extension associations, global
system variables, 
have the same/different site-packages versions/some missing...

(d)

software A installs system wide python X.x.x
or
software A installs system wide python X.x.y
AND
software B installs system wide python X.x.x
or
software B installs system wide python Y.y.y

...and a lot other variations are already out in the windows world ...

a system wide installation in windows defines the file extension
associations, global system variables, etc.;
additionally the applications need maybe the same site-packages or different
ones (and not always the same versions)

as a system wide installation defines the file extension associations,
global system variables, the installation order A then B or B then A may
break the installat

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Martin Landa]

> nobody is saying anything about a bad gay, it's your feeling not mine

well, I meant "a guy" of course :-) I hope that there is at least a
reason to smile ;-) Martin

-- 
Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Jürgen E . Fischer]

Hi Moritz,

On Sun, 13. Apr 2014 at 17:03:17 +0200, Moritz Lennert wrote:
> On 11/04/14 13:05, Martin Landa wrote:
>> 2014-04-11 12:42 GMT+02:00 Jürgen E. :

> >> That way you could use the stock sources and wouldn't have to use patches
> >> when packaging (which would have been my pragmatic approach to solve the
> >> issue at hand without risking that someone pulls a Glynn on it ;)).

> I don't think this a very helpful attitude. This whole debate is not about
> one bad guy keeping the rest of the good guys from moving forward.  First of
> all, Glynn is not alone in his position.

I couldn't resist to express some dislike about instantly reverting potentially
useful stuff without having a better solution - but en passant.

But discussing attitude doesn't help with running .py scripts on windows.
Martin probably didn't quote that part, because he knew better than us.

But that's was what started this thread - and to me that's still what it is
about.  I agree that this aspect wasn't too helpful, but it wasn't my main
point, otherwise I wouldn't have posted anything - it also had a pointer to a
potentially better approach.

And IMHO the broader topics would have been better addressed in (a) separate
thread(s) and not side-track this one.  Your original problem got out of focus,
but still isn't solved, is it?


Jürgen

-- 
Jürgen E. Fischer norBIT GmbH   Tel. +49-4931-918175-31
Dipl.-Inf. (FH)   Rheinstraße 13Fax. +49-4931-918175-50
Software Engineer D-26506 Norden   http://www.norbit.de
QGIS PSC member (RM)  Germany  IRC: jef on FreeNode 


-- 
norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH
Rheinstrasse 13, 26506 Norden
GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Martin Landa]

2014-04-13 17:03 GMT+02:00 Moritz Lennert :
> On 11/04/14 13:05, Martin Landa wrote:
> I don't think this a very helpful attitude. This whole debate is not about
> one bad guy keeping the rest of the good guys from moving forward. First of
> all, Glynn is not alone in his position.

nobody is saying anything about a bad gay, it's your feeling not mine
I can say. The fact is that this debate started more than one year ago
(probably two years ago or more) with *no* consensus or accepted
*solution*. I was playing with virtualenv, python launcher. I will be
more than satisfied if we find acceptable solution. We need to
continue and focus our energy on other important topics or we just
keep discussing this topic for other two or three years and blocking
any of releases. If you find a better solution in the near future we
can easily to integrate it and avoid current dirty workarounds. I am
looking forward to see real-working-solid solution as the outcome of
this debate. Till now I can hardy see it. Martin
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Benjamin Ducke]

On 13/04/14 17:03, Moritz Lennert wrote:

[...]

> 
> I don't think this a very helpful attitude. This whole debate is not
> about one bad guy keeping the rest of the good guys from moving forward.
> First of all, Glynn is not alone in his position.
> 

I agree. I tend to take a conservative view, because I am
old enough to have seen several trendy technologies and programming
styles come and go. I think that GRASS has done very well and
acquired an excellent reputation exactly because it has not
compromised on its cleanly engineered architecture.

> But secondly, and more importantly, the debate is about two fundamental
> questions and these questions need to be debated with patience:
> 
> 1) Should GRASS be the same (i.e. feature parity) on all platforms ?

I would say: Yes. I'd be a big step backward if users would
be tied to a certain operating system to get their work done.

> 2) How much less effort can we ask of Windows users than we ask of users
> of other platforms ?

We can ask them to put in just as much effort as everybody else.
I do think that most GRASS users on Windows don't mind, actually.
However, this is not true when it comes to running GRASS from
a host GIS such as QGIS (see my comments below).

> 
> GRASS was developed in a time when Windows didn't even exist. And it was
> developed in a very typical *nix style and form. This makes for its
> special character and for use forms which are quite different from most
> recent programs. For long years the only way GRASS could be run on
> Windows was by imitating a *nix environment using cygwin.
> 
> Growth in interest in free software, accompanied by a continued
> dominance of Windows on the desktop, and the desire of some of us to
> provide the possibility for our colleagues to run GRASS on their Windows
> machines prompted the beginning of a long period of making GRASS
> compilable and runnable natively in Windows. And the switch to Python
> for scripts was, amongst others, motivated by the desire to make these
> scripts more platform-independent.
> 
> At no point, however, was there ever the question of creating a special
> Windows version of GRASS, a version that would function differently from
> the *nix version. IIUC, this latter desire seems to have come from
> experiences of some of us with windows users who were not accustomed to
> *nix-style programs/libraries/toolboxes. Thus the debate about whether
> or not we should even show a command line environment outside the GUI
> and thus, now, the argument that GRASS4Win should be a different beast
> than GRASS4NIX in that it should become a monolithic integrated
> application, which, IMHO, fundamentally alters the very nature of what
> GRASS is. Generally, to put it more bluntly, there seems to be an
> attempt to dumb GRASS down for Windows users, who, on average, are less
> computer-savvy than *nix-users (as a side note, this argument is quickly
> losing its grounds when I see the number of GNU/Linux uptakes around me,
> especially now in the wake of the discontinuation of XP-support). The
> idea is that *nix users can solve Python installation issues so as to
> have their system-wide python work with GRASS (to be totally honest life
> is a bit easier for them thanks to the /bin/env solution).
> 
> [Side note: The same discussion should also constantly be held about
> functionality which is specific to the GUI, because specifically
> developed for it), i.e. not just a GUI layer above command line
> functionality, something I'm afraid to see creep in more and more...]
> 

Agreed. Once more, I plead for a clean separation between GUI
and CLI developments, and a disconnection of their release cycles.
The wxPython GUI can be considered a monolithic application,
and FWIW it can pull every trick in the book to integrate a
Python interpreter, and to make it all easier for Windows users.

But please: Let's keep (minor) version-specific Python support
and all of its associated problems out of the core release.

I really believe that this whole conundrum was caused by not
taking that step earlier.

> Now, as Glynn has pointed out, with distributions soon to ship Python3
> as default (and I'll throw in: with the *nix crowd changing), we will
> quite probably see similar issues in *nix installation as we are seeing
> now in Windows, so the question is how to find a solution which will
> work across all platforms without the need for constant manual
> intervention by developers to solve yet another special case. And, IMHO,
> part of this solution will be through trusting a GRASS user on Windows
> to be capable of following basic instructions concerning the steps to
> follow to install GRASS on their system.
> 

I would say: Consider the wxGUI, the QGIS and gvSIG plug-ins etc.
monolithic applications and let their maintainers figure out how
to deal with this. In the gvSIG CE project, we do a lot of hair-
raising stuff to hide the complexity of GRASS and its dependencies
from the end user, and I susp

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Moritz Lennert]

On 11/04/14 13:05, Martin Landa wrote:

Hi Jurgen,

2014-04-11 12:42 GMT+02:00 Jürgen E. :


although the architecture may differ, we may get some inspiration for python
windows handling from other GIS projects like QGIS?

in QGIS python is also in heavy use, e.g. QGIS processing framework, python
addons, integrated python shell, etc. AFAIK they're bundling python in their
standalone winQGIS ...


Not sure if I want to participate in this thread.


you are probably not alone...



[...]

>> That way you could use the stock sources and wouldn't have to use
>> patches when packaging (which would have been my pragmatic approach
>> to solve the issue at hand without risking that someone pulls a
>> Glynn on it ;)).

I don't think this a very helpful attitude. This whole debate is not 
about one bad guy keeping the rest of the good guys from moving forward. 
First of all, Glynn is not alone in his position.


But secondly, and more importantly, the debate is about two fundamental 
questions and these questions need to be debated with patience:


1) Should GRASS be the same (i.e. feature parity) on all platforms ?
2) How much less effort can we ask of Windows users than we ask of users 
of other platforms ?


GRASS was developed in a time when Windows didn't even exist. And it was 
developed in a very typical *nix style and form. This makes for its 
special character and for use forms which are quite different from most 
recent programs. For long years the only way GRASS could be run on 
Windows was by imitating a *nix environment using cygwin.


Growth in interest in free software, accompanied by a continued 
dominance of Windows on the desktop, and the desire of some of us to 
provide the possibility for our colleagues to run GRASS on their Windows 
machines prompted the beginning of a long period of making GRASS 
compilable and runnable natively in Windows. And the switch to Python 
for scripts was, amongst others, motivated by the desire to make these 
scripts more platform-independent.


At no point, however, was there ever the question of creating a special 
Windows version of GRASS, a version that would function differently from 
the *nix version. IIUC, this latter desire seems to have come from 
experiences of some of us with windows users who were not accustomed to 
*nix-style programs/libraries/toolboxes. Thus the debate about whether 
or not we should even show a command line environment outside the GUI 
and thus, now, the argument that GRASS4Win should be a different beast 
than GRASS4NIX in that it should become a monolithic integrated 
application, which, IMHO, fundamentally alters the very nature of what 
GRASS is. Generally, to put it more bluntly, there seems to be an 
attempt to dumb GRASS down for Windows users, who, on average, are less 
computer-savvy than *nix-users (as a side note, this argument is quickly 
losing its grounds when I see the number of GNU/Linux uptakes around me, 
especially now in the wake of the discontinuation of XP-support). The 
idea is that *nix users can solve Python installation issues so as to 
have their system-wide python work with GRASS (to be totally honest life 
is a bit easier for them thanks to the /bin/env solution).


[Side note: The same discussion should also constantly be held about 
functionality which is specific to the GUI, because specifically 
developed for it), i.e. not just a GUI layer above command line 
functionality, something I'm afraid to see creep in more and more...]


Now, as Glynn has pointed out, with distributions soon to ship Python3 
as default (and I'll throw in: with the *nix crowd changing), we will 
quite probably see similar issues in *nix installation as we are seeing 
now in Windows, so the question is how to find a solution which will 
work across all platforms without the need for constant manual 
intervention by developers to solve yet another special case. And, IMHO, 
part of this solution will be through trusting a GRASS user on Windows 
to be capable of following basic instructions concerning the steps to 
follow to install GRASS on their system.



if you take a look on all threads related to this topic in grass-dev
ML you will count more that 150 messages, but without real consensus.


Well, there have been different proposals for possible solutions, 
amongst which the Python launcher which I tested last summer [1], but 
except for Helmut, none of those who are pushing for a quick solution of 
the issue seem to have been sufficiently interested to test that path. I 
know I haven't been very active on that front, and so will not throw 
stones, but others shouldn't either...


But whatever the solution, unless we want to turn GRASS4Win into a 
different application than GRASS on other platforms, there will have to 
be a minimal amount of intervention by the user to chose how she wants 
to handle Python on her machine.


To come back to a general note about this debate: As should be clear 
from this and previous intervent

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

>thanks also for the link! that's all about we're discussing here.
>
>Python Launcher 

just forgotten, some ugly tests for Python Launcher:

http://trac.osgeo.org/grass/ticket/2015#comment:6



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5134733.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

Hi Jürgen,

> Hi Helmut, 
[...]
> Not sure if I want to participate in this thread. 

thanks for sharing your  experience anyway.

> I think a system installation of python is uncommon.

yes I'm also thinking so if windows comes in ...

[...]
>And
>thereby avoid having special treatment for .py scripts in GRASS itself (see
>http://stackoverflow.com/questions/5583024/temporary-file-association-for-single-cmd-exe-session)
> 

thanks also for the link! that's all about we're discussing here. 

Python Launcher and virtualenv are mentioned there ... maybe some worthy
approaches to satisfy a non-monolithic application in an
non-monolithic-application-agnostic operating system ;-)



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5134732.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Helmut Kudrnovsky wrote:

> although the architecture may differ, we may get some inspiration for python
> windows handling from other GIS projects like QGIS? 
> 
> in QGIS python is also in heavy use, e.g. QGIS processing framework, python
> addons, integrated python shell, etc. AFAIK they're bundling python in their
> standalone winQGIS ...

QGIS is a monolithic application. GRASS is a set of command-line
programs (plus an optional GUI front-end).

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Martin Landa]

Hi Jurgen,

2014-04-11 12:42 GMT+02:00 Jürgen E. :

>> although the architecture may differ, we may get some inspiration for python
>> windows handling from other GIS projects like QGIS?
>>
>> in QGIS python is also in heavy use, e.g. QGIS processing framework, python
>> addons, integrated python shell, etc. AFAIK they're bundling python in their
>> standalone winQGIS ...
>
> Not sure if I want to participate in this thread.

you are probably not alone...

> I think a system installation of python is uncommon.  You'd have to deal with 
> a
> variety of possibilities (python.org, ESRI, ...), you would need to make sure
> all extensions you need are available for each of them - and have them play
> nice with the rest of your install.
>
> Shipping you own python with all the extensions you need, probably saves a lot
> of headaches and lifetime.   That's the approach OSGeo4W (and in turn QGIS)
> takes.  I wouldn't require a system wide python install or try to use an
> existing one.

if you take a look on all threads related to this topic in grass-dev
ML you will count more that 150 messages, but without real consensus.
I completely agree with you. To avoid bundled Python it would require
a lot of work, we simply do not have such man-power, resources or
fundings. All projects around us shifts their own Python for Windows,
and of course beasts like Esri installs their Python system-wide, they
simply don't care. We are in different situation, happily. So if we
want to move forward (and to release GRASS 7) we would be simply
forced to stick with the current solution (forcing Python script which
are launched within GRASS environment to use Python version provided
by the installer or by OSGeo4W framework). Or we need someone who will
invest his/her energy to implement something better, it will take
probably take not days, but weeks.
Then we will probably first project who will avoid bundled Python on
Windows, cool ;-) Just discussing this topic forever is not a solution
or real problems.

Martin
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Jürgen E . Fischer]

Hi Helmut,

On Thu, 10. Apr 2014 at 14:34:04 -0700, Helmut Kudrnovsky wrote:
> although the architecture may differ, we may get some inspiration for python
> windows handling from other GIS projects like QGIS? 
> 
> in QGIS python is also in heavy use, e.g. QGIS processing framework, python
> addons, integrated python shell, etc. AFAIK they're bundling python in their
> standalone winQGIS ...

Not sure if I want to participate in this thread.

I think a system installation of python is uncommon.  You'd have to deal with a
variety of possibilities (python.org, ESRI, ...), you would need to make sure
all extensions you need are available for each of them - and have them play
nice with the rest of your install.

Shipping you own python with all the extensions you need, probably saves a lot
of headaches and lifetime.   That's the approach OSGeo4W (and in turn QGIS)
takes.  I wouldn't require a system wide python install or try to use an
existing one.

You could establish (or modify a pre-existing) file association to python files
that is tied to an environment variable.   And thereby make the association use
GRASS' python within GRASS and the system-wide installation otherwise.  And
thereby avoid having special treatment for .py scripts in GRASS itself (see
http://stackoverflow.com/questions/5583024/temporary-file-association-for-single-cmd-exe-session)

That way you could use the stock sources and wouldn't have to use patches when
packaging (which would have been my pragmatic approach to solve the issue at
hand without risking that someone pulls a Glynn on it ;)).


Jürgen

-- 
Jürgen E. Fischer norBIT GmbH   Tel. +49-4931-918175-31
Dipl.-Inf. (FH)   Rheinstraße 13Fax. +49-4931-918175-50
Software Engineer D-26506 Norden   http://www.norbit.de
QGIS PSC member (RM)  Germany  IRC: jef on FreeNode 


-- 
norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH
Rheinstrasse 13, 26506 Norden
GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

[...]
>> GRASS 6 creates a .bat file for each shell script.
>
>And this .bat file specifies the script interpreter. Looks like a good
>solution to also select the correct Python version. 
[...]
>Other projects also bypass the standard execution mechanism of python
>scripts, and they work just fine in MS Windows.

although the architecture may differ, we may get some inspiration for python
windows handling from other GIS projects like QGIS? 

in QGIS python is also in heavy use, e.g. QGIS processing framework, python
addons, integrated python shell, etc. AFAIK they're bundling python in their
standalone winQGIS ...




-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5134463.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Vaclav Petras wrote:

> > And this .bat file specifies the script interpreter. Looks like a good
> > solution to also select the correct Python version.
> 
> 
> I'm afraid how will work for user scripts. Typical case will be that user
> has some Python, so he or she will try to run from outside if he or she
> sets the environment correctly, running of modules (exe or bat) will be OK,
> but grass.pygrass and grass.lib might not be OK.
> 
> The other typical case (hopefully more common) is when a user writes his or
> her own GRASS Python module/script. This does not contain any environment
> settings (same as .py in scripts/ and temporal/) because it is intended to
> be run inside of GRASS session. However, it will not run because there is
> no .bat. Should user create one? Should GUI do some workaround for that
> case? But what about Python command line, wxGUI command console and
> standard Windows command line?
> 
> My point is that nobody was writing shell scripts on Windows but people are
> writing Python scripts/modules. So, this new problem to solve.

My view is that .bat files may have some value as a workaround to make
GRASS mostly functional on systems which lack a usable (or any) Python
installation. A significant advantage is that they're a lot less
invasive than having hard-coded treatment for Python scripts littered
throughout GRASS.

On systems with a suitable Python installation, neither .bat files nor
the use of a bundled Python interpreter should be forced upon the
user.

My main criterion for any workaround for Python-on-Windows "issues" is
"First, do no harm". Users who want feature-parity with Unix (and are
willing to satisfy the requirement of having Python as a "system
feature") shouldn't have to suffer in order to make the packager's job
easier.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> > In spite of wxGUI, GRASS remains fundamentally a collection of
> > command-line modules, more like a library than an application.
> 
> I make use of this property daily. On MS Windows, GRASS should by
> default behave like a stand-alone application. You can make use of the
> collection of command-line modules through the msys shell, but that is
> really only for power users.

You can use any shell. On Windows, there's no particular reason to use
MSys. It's more powerful than cmd.exe, but also has a lot of drawbacks
(MSys doesn't register it as an interpreter for the .sh extension, it
requires Unix-style pathnames, it doesn't honour file associations,
etc).

We certainly shouldn't be bundling MSys with GRASS 7. A very large
part of the reason for the switch to Python was to eliminate that
particular dependency (the remaining part being that, on any platform,
Bourne shell is simply unfit for "scripts" which need to perform
non-trivial data processing themselves, as opposed to simply gluing
other programs together).

If a script is trivial enough not to prefer using Python, it's trivial
enough for cmd.exe.

> > These are self-contained GUI applications. Which is a major
> > limitation; if I need to manipulate images, I'm more likely to use
> > NetPBM or PIL/NumPy than GIMP (on any platform). Apart from anything
> > else, the job will be over and done while GIMP would still be showing
> > the splash screen.
> >
> > If I'm likely to perform the same operation more than once, I'm *much*
> > more likely to use the scripted approach than the pointy-clicky GUI.
> 
> As an experienced *NIX user, yes. Not so Windows users, many feel
> alienated by a command line. On Windows, GRASS should behave like a
> self-contained GUI application. There were even suggestions to remove
> the command line interface completely from winGRASS because it
> confuses users.

Such suggestions are entirely misplaced. The end result would be so
watered-down as to make GRASS almost useless.

It's clear that we're never going to agree on this. You're willing to
substntially cripple GRASS in the hope that it will be usable by the
"... For Dummies" crowd, and I'm not.

And I don't even think that such a goal realistic. However much you
simplify the program, you can't simplify the underlying concepts. It
has been noted that the "For Dummies" series doesn't include
"Neurosurgery for Dummies", "Civil Aviation for Dummies", or the like.

Do we really need more users who not only don't know which projection
their data is in, but don't even know what a projection is? And,
regardless of how much the interface is simplified, is GRASS even
going to be of any use to them?

tl;dr: if you want Google Earth, you know where to find it.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> > I basically agree with user expectations you stated. But I would like to
> > note that recently, I met several users which wanted and expected that GRASS
> > script will run outside GRASS without any special environment setup in the
> > script itself.
> 
> AFAIK, GRASS scripts explicitly test if they are run inside GRASS,
> otherwise they exit with an error message saying "You must be in GRASS
> GIS to run this program." You are inside GRASS as soon as the
> environment variables for the GRASS version are set and as soon as a
> proper GRASS session has been established (location and mapset). No
> GRASS script should run outside GRASS, independent of the OS.

This is conflating "inside GRASS" with "necessary environment
variables set".

It's entirely possible to configure the system so that the environment
variables are always set.

As I've mentioned before, the only time I run the grass7 script is if
I'm testing it.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Wed, Apr 9, 2014 at 2:16 AM, Markus Metz
wrote:

> On Wed, Apr 9, 2014 at 3:44 AM, Glynn Clements 
> wrote:
> >
> > Markus Metz wrote:
> >
> >> > All of this goes out the window if you want to provide a command-line
> >> > environment, whether an interactive shell or the ability to execute
> >> > commands via system() or CreateProcess().
> >>
> >> It works in GRASS 6 with shell scripts. I am sure the same mechanism
> >> will work just as well with python scripts.
> >
> > GRASS 6 creates a .bat file for each shell script.
>
> And this .bat file specifies the script interpreter. Looks like a good
> solution to also select the correct Python version.


I'm afraid how will work for user scripts. Typical case will be that user
has some Python, so he or she will try to run from outside if he or she
sets the environment correctly, running of modules (exe or bat) will be OK,
but grass.pygrass and grass.lib might not be OK.

The other typical case (hopefully more common) is when a user writes his or
her own GRASS Python module/script. This does not contain any environment
settings (same as .py in scripts/ and temporal/) because it is intended to
be run inside of GRASS session. However, it will not run because there is
no .bat. Should user create one? Should GUI do some workaround for that
case? But what about Python command line, wxGUI command console and
standard Windows command line?

My point is that nobody was writing shell scripts on Windows but people are
writing Python scripts/modules. So, this new problem to solve.

Vaclav
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Moritz Lennert]

On 09/04/14 08:16, Markus Metz wrote:

On Wed, Apr 9, 2014 at 3:44 AM, Glynn Clements  wrote:

[...]

In spite of wxGUI, GRASS remains fundamentally a collection of
command-line modules, more like a library than an application.


I make use of this property daily. On MS Windows, GRASS should by
default behave like a stand-alone application. You can make use of the
collection of command-line modules through the msys shell, but that is
really only for power users.


[...]


If I'm likely to perform the same operation more than once, I'm *much*
more likely to use the scripted approach than the pointy-clicky GUI.


As an experienced *NIX user, yes. Not so Windows users, many feel
alienated by a command line. On Windows, GRASS should behave like a
self-contained GUI application. There were even suggestions to remove
the command line interface completely from winGRASS because it
confuses users.



So, actually here the debate is about whether GRASS for Windows should 
be a different beast than GRASS for *nix/Mac. In my practical teaching 
experience I would strongly plead for having them as similar as 
possible. Learning on one platform should allow you to use GRASS on any 
platform.


And, in my experience, I've never seen users confused by the command 
line in Windows. They might ignore it, it might scare them a bit, but 
I've never seen it cause any problems. On the contrary, I've always 
found that with a little gentle introduction, students very quickly see 
the advantage of the command line over the GUI.



IMHO, what you say is that GRASS and MS Windows are incompatible by
principle,


Not at all. Windows does actually have a command-execution mechanism
(even the Mac has one nowadays).


Any installed Python interpreter does not know about any other
packages relying on it. The Python interpreter can change any time
(version, architecture), or disappear, without other packages knowing
about it, then there is a problem. Windows applications are by nature
stand-alone applications, they don't know about each other's
existence.


In what ways is this different from *nix ? GRASS has no knowledge or 
control over the Python interpreter. It's up to the user to install the 
correct interpreter and make sure that GRASS can find it. Luckily this 
is often taken care of by distribution packaging, but it's still 
independent of GRASS.


Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Wed, Apr 9, 2014 at 3:44 AM, Glynn Clements  wrote:
>
> Markus Metz wrote:
>
>> > All of this goes out the window if you want to provide a command-line
>> > environment, whether an interactive shell or the ability to execute
>> > commands via system() or CreateProcess().
>>
>> It works in GRASS 6 with shell scripts. I am sure the same mechanism
>> will work just as well with python scripts.
>
> GRASS 6 creates a .bat file for each shell script.

And this .bat file specifies the script interpreter. Looks like a good
solution to also select the correct Python version.
>
>> > The only way that you can "configure" Windows' command execution
>> > mechanism is by registering extensions.
>>
>> I suggest to not even attempt to configure a Windows' command
>> execution mechanism.
>
> If you mean "don't mess with the existing .py association", I agree
> with you. If you mean "make every attempt to bypass the standard
> execution mechanisms", that will make WinGRASS a second-rate imitation
> of the Unix version.

Other projects also bypass the standard execution mechanism of python
scripts, and they work just fine in MS Windows.

>
> In spite of wxGUI, GRASS remains fundamentally a collection of
> command-line modules, more like a library than an application.

I make use of this property daily. On MS Windows, GRASS should by
default behave like a stand-alone application. You can make use of the
collection of command-line modules through the msys shell, but that is
really only for power users.

>
> The big advantage of this is that that it can be scripted, in any
> language capable of executing commands. Python, Bourne Shell, cmd.exe,
> Visual Basic, or whatever.

That means creating a new command, a script in the language of choice.
Executing this command and executing GRASS commands are two different
things.

>
> You can either attempt to "capture" and bypass as many of the possible
> interfaces to the system's execution mechanism as possible, or you can
> make the system's execution mechanism work.
>
> The latter one requires dealing with factors beyond our control, but
> ultimately it results in a superior product (it's also less work).
>
>> >> Instead of battling the MS Windows software management system,
>> >
>> > But that's exactly what trying to "bundle" Python is doing. You don't
>> > trust the OS to be configured correctly so you're trying to construct
>> > an isolated sandbox.
>>
>> Exactly! E.g. LibreOffice and Gimp also operate as isolated sandboxes.
>> Actually, pretty much every MS Windows software is an isolated
>> sandbox. This is the safest way to ensure proper operation on MS
>> Windows.
>
> These are self-contained GUI applications. Which is a major
> limitation; if I need to manipulate images, I'm more likely to use
> NetPBM or PIL/NumPy than GIMP (on any platform). Apart from anything
> else, the job will be over and done while GIMP would still be showing
> the splash screen.
>
> If I'm likely to perform the same operation more than once, I'm *much*
> more likely to use the scripted approach than the pointy-clicky GUI.

As an experienced *NIX user, yes. Not so Windows users, many feel
alienated by a command line. On Windows, GRASS should behave like a
self-contained GUI application. There were even suggestions to remove
the command line interface completely from winGRASS because it
confuses users.

>
>> IMHO, what you say is that GRASS and MS Windows are incompatible by
>> principle,
>
> Not at all. Windows does actually have a command-execution mechanism
> (even the Mac has one nowadays).

Any installed Python interpreter does not know about any other
packages relying on it. The Python interpreter can change any time
(version, architecture), or disappear, without other packages knowing
about it, then there is a problem. Windows applications are by nature
stand-alone applications, they don't know about each other's
existence.

>
>> and you will not succeed in making MS Windows compatible
>> with these GRASS principles. I assume WinGRASS users expect a typical
>> MS Windows software like clicking on the GRASS icon which executes a
>> command (grassXY) and starts the software. I strongly suggest to
>> respect the users' expectations. It does not matter in this respect if
>> the command actually only sets the environment for GRASS modules and
>> if this environment is like a sandbox or system-wide. FWIW, Linux
>> users also execute a grassxy command to start GRASS.
>
> No matter how much GUI you wrap around it, GRASS is never going to be
> a "point-and-drool" application aimed at users who want a "Solve my
> problem for me" icon on the toolbar.
>
> I would expect anything beyond the simplest cases to be scripted. So
> that you can easily repeat the exact process with next month's data,
> or with a different value for a parameter, or so that you can easily
> communicate your methodology to others

I fully agree.

Markus M
___
grass-dev mailing list
grass

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Tue, Apr 8, 2014 at 11:23 PM, Vaclav Petras  wrote:
>
> On Tue, Apr 8, 2014 at 3:22 PM, Markus Metz 
> wrote:
>>
>> IMHO, what you say is that GRASS and MS Windows are incompatible by
>> principle, and you will not succeed in making MS Windows compatible
>> with these GRASS principles. I assume WinGRASS users expect a typical
>> MS Windows software like clicking on the GRASS icon which executes a
>> command (grassXY) and starts the software. I strongly suggest to
>> respect the users' expectations. It does not matter in this respect if
>> the command actually only sets the environment for GRASS modules and
>> if this environment is like a sandbox or system-wide. FWIW, Linux
>> users also execute a grassxy command to start GRASS.
>
>
> I basically agree with user expectations you stated. But I would like to
> note that recently, I met several users which wanted and expected that GRASS
> script will run outside GRASS without any special environment setup in the
> script itself.

AFAIK, GRASS scripts explicitly test if they are run inside GRASS,
otherwise they exit with an error message saying "You must be in GRASS
GIS to run this program." You are inside GRASS as soon as the
environment variables for the GRASS version are set and as soon as a
proper GRASS session has been established (location and mapset). No
GRASS script should run outside GRASS, independent of the OS.

Markus M


> Perhaps, ArcGIS sets the environment, so this might be the
> reason they expect the same from GRASS. I wonder if we can make setting the
> environment in Python easier.
>
> Vaclav
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Vaclav Petras wrote:

> I basically agree with user expectations you stated. But I would like to
> note that recently, I met several users which wanted and expected that
> GRASS script will run outside GRASS without any special environment setup
> in the script itself. Perhaps, ArcGIS sets the environment, so this might
> be the reason they expect the same from GRASS. I wonder if we can make
> setting the environment in Python easier.

Windows' environment handling mimics that of Unix: a process can set
its own environment, which is inherited by child processes, or it can
explicitly set the environment of a child process at creation time.

But you can't write a program which sets the environment for the shell
(or other program) from which it was invoked, for the benefit of
subsequent child processes started from the same shell.

Packages which include command-line programs (e.g. compilers)
typically either modify the environment variables (e.g. PATH) via the
registry[1], or provide a .bat/.cmd file which must be run to set up
the environment for a particular shell.

We could certainly reduce the number of environment variables
involved, e.g. by having G_gisinit() check for missing variables and
initialising them from settings in the registry. Even GISRC could fall
back to e.g. %APPDATA%\GRASS\gisrc.

Supporting multiple installed versions and/or multiple concurrent
sessions would be marginally more complex. But then many Windows
applications entirely prohibit doing either of those.

[1] HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Environment
for the system-wide settings and HKCU\Environment for the per-user
settings.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> > All of this goes out the window if you want to provide a command-line
> > environment, whether an interactive shell or the ability to execute
> > commands via system() or CreateProcess().
> 
> It works in GRASS 6 with shell scripts. I am sure the same mechanism
> will work just as well with python scripts.

GRASS 6 creates a .bat file for each shell script.

> > The only way that you can "configure" Windows' command execution
> > mechanism is by registering extensions.
> 
> I suggest to not even attempt to configure a Windows' command
> execution mechanism.

If you mean "don't mess with the existing .py association", I agree
with you. If you mean "make every attempt to bypass the standard
execution mechanisms", that will make WinGRASS a second-rate imitation
of the Unix version.

In spite of wxGUI, GRASS remains fundamentally a collection of
command-line modules, more like a library than an application.

The big advantage of this is that that it can be scripted, in any
language capable of executing commands. Python, Bourne Shell, cmd.exe,
Visual Basic, or whatever.

You can either attempt to "capture" and bypass as many of the possible
interfaces to the system's execution mechanism as possible, or you can
make the system's execution mechanism work.

The latter one requires dealing with factors beyond our control, but
ultimately it results in a superior product (it's also less work).

> >> Instead of battling the MS Windows software management system,
> >
> > But that's exactly what trying to "bundle" Python is doing. You don't
> > trust the OS to be configured correctly so you're trying to construct
> > an isolated sandbox.
> 
> Exactly! E.g. LibreOffice and Gimp also operate as isolated sandboxes.
> Actually, pretty much every MS Windows software is an isolated
> sandbox. This is the safest way to ensure proper operation on MS
> Windows.

These are self-contained GUI applications. Which is a major
limitation; if I need to manipulate images, I'm more likely to use
NetPBM or PIL/NumPy than GIMP (on any platform). Apart from anything
else, the job will be over and done while GIMP would still be showing
the splash screen.

If I'm likely to perform the same operation more than once, I'm *much*
more likely to use the scripted approach than the pointy-clicky GUI.

> IMHO, what you say is that GRASS and MS Windows are incompatible by
> principle,

Not at all. Windows does actually have a command-execution mechanism
(even the Mac has one nowadays).

Some people even run web servers on Windows (and I don't believe that
Microsoft has yet found a way to use arbitrary GUI programs in place
of PHP, ASP or whatever).

> and you will not succeed in making MS Windows compatible
> with these GRASS principles. I assume WinGRASS users expect a typical
> MS Windows software like clicking on the GRASS icon which executes a
> command (grassXY) and starts the software. I strongly suggest to
> respect the users' expectations. It does not matter in this respect if
> the command actually only sets the environment for GRASS modules and
> if this environment is like a sandbox or system-wide. FWIW, Linux
> users also execute a grassxy command to start GRASS.

No matter how much GUI you wrap around it, GRASS is never going to be
a "point-and-drool" application aimed at users who want a "Solve my
problem for me" icon on the toolbar.

I would expect anything beyond the simplest cases to be scripted. So
that you can easily repeat the exact process with next month's data,
or with a different value for a parameter, or so that you can easily
communicate your methodology to others (I find pasting a script
fragment somewhat easier than "... then click on the icon that looks
like, uh, ...").

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Tue, Apr 8, 2014 at 3:22 PM, Markus Metz
wrote:

> IMHO, what you say is that GRASS and MS Windows are incompatible by
> principle, and you will not succeed in making MS Windows compatible
> with these GRASS principles. I assume WinGRASS users expect a typical
> MS Windows software like clicking on the GRASS icon which executes a
> command (grassXY) and starts the software. I strongly suggest to
> respect the users' expectations. It does not matter in this respect if
> the command actually only sets the environment for GRASS modules and
> if this environment is like a sandbox or system-wide. FWIW, Linux
> users also execute a grassxy command to start GRASS.
>

I basically agree with user expectations you stated. But I would like to
note that recently, I met several users which wanted and expected that
GRASS script will run outside GRASS without any special environment setup
in the script itself. Perhaps, ArcGIS sets the environment, so this might
be the reason they expect the same from GRASS. I wonder if we can make
setting the environment in Python easier.

Vaclav
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Thu, Mar 6, 2014 at 10:08 AM, Glynn Clements
 wrote:
>
> Markus Metz wrote:
>
>> > .py is supposed to be associated with a Python interpreter, and
>> > the stock Python installer will do that.
>>
>> .py is not supposed to be associated with a Python interpreter that is
>> installed system-wide
>
> It certainly is, because that's what the stock Python installer will
> do by default.

I was unclear, I mean that any .py association must be ignored because
GRASS might be might be incompatible with the stock Python installer .

>
>> because this assumes some kind of package
>> manager as available on Linux/BSD/Unix. MS Windows does not have a
>> package manager. Any software installed on Windows must include any
>> script interpreters and enforce the use of these for the respective
>> scripts.
>>
>> The fundamental assumption of MS Windows software installers is that
>> no other software installer will interfere with its installation. This
>> assumption is violated as soon as a software installer invokes another
>> software installer. This means that the WinGRASS software installer
>> should not invoke or require a Python installation on MS Windows.
>
> All of this goes out the window if you want to provide a command-line
> environment, whether an interactive shell or the ability to execute
> commands via system() or CreateProcess().

It works in GRASS 6 with shell scripts. I am sure the same mechanism
will work just as well with python scripts.
>
> The only way that you can "configure" Windows' command execution
> mechanism is by registering extensions.

I suggest to not even attempt to configure a Windows' command
execution mechanism.
>
>> Instead of battling the MS Windows software management system,
>
> But that's exactly what trying to "bundle" Python is doing. You don't
> trust the OS to be configured correctly so you're trying to construct
> an isolated sandbox.

Exactly! E.g. LibreOffice and Gimp also operate as isolated sandboxes.
Actually, pretty much every MS Windows software is an isolated
sandbox. This is the safest way to ensure proper operation on MS
Windows.

>
>> I would
>> rather follow the approach of other projects that have been running
>> successfully on MS Windows with their embedded python interpreter for
>> a couple of years.
>
> You're referring to monolithic applications which require that any
> "scripting" is done from within the application itself. They typically
> also require the use of a specific language for scripting, and have
> entirely separate mechanisms for extension via scripts and extension
> via compiled code (the latter typically being in the form of DLL
> "plug-ins").
>
> IOW, the exact opposite of GRASS.

IMHO, what you say is that GRASS and MS Windows are incompatible by
principle, and you will not succeed in making MS Windows compatible
with these GRASS principles. I assume WinGRASS users expect a typical
MS Windows software like clicking on the GRASS icon which executes a
command (grassXY) and starts the software. I strongly suggest to
respect the users' expectations. It does not matter in this respect if
the command actually only sets the environment for GRASS modules and
if this environment is like a sandbox or system-wide. FWIW, Linux
users also execute a grassxy command to start GRASS.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> > .py is supposed to be associated with a Python interpreter, and
> > the stock Python installer will do that.
> 
> .py is not supposed to be associated with a Python interpreter that is
> installed system-wide

It certainly is, because that's what the stock Python installer will
do by default.

> because this assumes some kind of package
> manager as available on Linux/BSD/Unix. MS Windows does not have a
> package manager. Any software installed on Windows must include any
> script interpreters and enforce the use of these for the respective
> scripts.
> 
> The fundamental assumption of MS Windows software installers is that
> no other software installer will interfere with its installation. This
> assumption is violated as soon as a software installer invokes another
> software installer. This means that the WinGRASS software installer
> should not invoke or require a Python installation on MS Windows.

All of this goes out the window if you want to provide a command-line
environment, whether an interactive shell or the ability to execute
commands via system() or CreateProcess().

The only way that you can "configure" Windows' command execution
mechanism is by registering extensions.

> Instead of battling the MS Windows software management system,

But that's exactly what trying to "bundle" Python is doing. You don't
trust the OS to be configured correctly so you're trying to construct
an isolated sandbox.

> I would
> rather follow the approach of other projects that have been running
> successfully on MS Windows with their embedded python interpreter for
> a couple of years.

You're referring to monolithic applications which require that any
"scripting" is done from within the application itself. They typically
also require the use of a specific language for scripting, and have
entirely separate mechanisms for extension via scripts and extension
via compiled code (the latter typically being in the form of DLL
"plug-ins").

IOW, the exact opposite of GRASS.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Wed, Feb 19, 2014 at 3:56 PM, Glynn Clements
 wrote:
>
> Markus Metz wrote:
>
>> >> > It's fairly trivial to set a valid GRASS environment globally, so that
>> >> > commands are usable in any shell. That's how I've had it on Linux
>> >> > since roughly forever; I don't run the grass70 script unless I'm
>> >> > testing it.
>> >>
>> >> At the very least, GRASS needs to clean up temporary files on exit.
>> >
>> > Temporary files need to be cleaned up periodically. The term "on exit"
>> > doesn't make sense for something which doesn't start or stop.
>>
>> I thought that is the way GRASS operates currently: a process is
>> started and stopped when the user types "exit". At least I do that on
>> Linux when calling grass64 or grass70.
>
> And I'm suggesting eliminating that mechanism because it's
> inconvenient and unnecessary (at one point, the only way to change
> mapsets was to exit the current session and start a new one, but that
> hasn't been the case for a long while).

Typing "exit" invokes some clean up which I do not want to miss.

We are using something similar to your suggested new mechanism on a
high performance cluster, but struggle with the cleaning up. "We"
means long-term GRASS users and developers. Leaving the cleaning up of
any temp files to the user is IMHO very user-unfriendly.

> .py is supposed to be associated with a Python interpreter, and
> the stock Python installer will do that.

.py is not supposed to be associated with a Python interpreter that is
installed system-wide because this assumes some kind of package
manager as available on Linux/BSD/Unix. MS Windows does not have a
package manager. Any software installed on Windows must include any
script interpreters and enforce the use of these for the respective
scripts.

The fundamental assumption of MS Windows software installers is that
no other software installer will interfere with its installation. This
assumption is violated as soon as a software installer invokes another
software installer. This means that the WinGRASS software installer
should not invoke or require a Python installation on MS Windows.

Instead of battling the MS Windows software management system, I would
rather follow the approach of other projects that have been running
successfully on MS Windows with their embedded python interpreter for
a couple of years.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> >> > It's fairly trivial to set a valid GRASS environment globally, so that
> >> > commands are usable in any shell. That's how I've had it on Linux
> >> > since roughly forever; I don't run the grass70 script unless I'm
> >> > testing it.
> >>
> >> At the very least, GRASS needs to clean up temporary files on exit.
> >
> > Temporary files need to be cleaned up periodically. The term "on exit"
> > doesn't make sense for something which doesn't start or stop.
> 
> I thought that is the way GRASS operates currently: a process is
> started and stopped when the user types "exit". At least I do that on
> Linux when calling grass64 or grass70.

And I'm suggesting eliminating that mechanism because it's
inconvenient and unnecessary (at one point, the only way to change
mapsets was to exit the current session and start a new one, but that
hasn't been the case for a long while).

> >> We use hard-coded special treatment for shell scripts and still they
> >> behave like a GRASS module.
> >
> > Where?
> 
> We create .bat wrappers for shell scripts here:
> https://trac.osgeo.org/grass/browser/grass/branches/releasebranch_6_4/include/Make/Script.make#L15

Ah, that.

Shell scripts are somewhat abnormal, largely because MSys (and
similar) try to emulate aspects of Unix beyond the shell itself. But
the main reason we have to do that is that MSys itself doesn't
associate its sh.exe with the .sh extension; it assumes that shell
scripts will always be started from within the shell, which has its
own execution mechanism which recognises scripts and uses the #! line.

> >> I don't see how users are bothered whether
> >> a module is an executable or a shell script. Internally, GRASS can
> >> always use system() or subprocess.Popen() for executables and .bat
> >> files, but not for shell or Python scripts, Therefore we need
> >> hard-coded special treatment for shell and Python scripts in order to
> >> make sure that the correct interpreter is used.
> >
> > IIRC, .bat files were used because there were issues with associating
> > .sh with MSys' shell (one of them being that MSys itself doesn't do
> > that, so we'd effectively be forced to make global changes to
> > something which doesn't "belong" to GRASS).
> 
> Msys is not a Windows shell script interpreter. IIUC, msys provides a
> minimal, isolated system environment which needs to be started
> explicitly.

Not really. Although setting PATH is even more important than with
other command-line programs, because the shell itself doesn't provide
most of the facilities found in other programming languages, relying
upon external commands for all but the most basic operations; e.g. 
"expr" for arithmetic operation ("test" aka "[" is only built-in for
efficiency; historically, it was external command, and coreutils still
provides it).

> Just as there were issues with associating .sh with MSys' shell, there
> are issues when .py files are associated with the system's Python.
> IMHO, any attempt of GRASS to associate .sh files or .py files on
> Windows with a particular interpreter is doomed to fail. WinGRASS
> includes all the needed interpreters and should use them.

WinGRASS can't make the OS use them other than by ensuring that the
extension is registered.

Shell scripts calling shell scripts works (sort of) because the shell
deliberately emulates the Unix #! mechanism. Although it still falls
down in the case of a script calling an EXE which tries to execute a
script directly.

And with .sh, we had no real choice. MSys doesn't even try to
associate it, and GRASS shouldn't be overriding that.

But .py is supposed to be associated with a Python interpreter, and
the stock Python installer will do that.

I don't particularly like working around bugs (which is what
"bundling" Python tries to do), but I'll do it ... except when the
workaround ends up breaking systems which are actually configured
correctly and/or imposing a maintenance burden on the system as a
whole (and "bundling" Python ends up doing both of those).

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Tue, Feb 11, 2014 at 5:39 PM, Glynn Clements
 wrote:
>
> Markus Metz wrote:
>
>> This is the fundamental difference and the reason why using a
>> system-wide Python can only cause trouble on Windows. On Windows, a
>> software package typically includes everything it needs to run,
>
> On Windows, a software package is typically a monolithic executable
> with no ability to be controlled by other programs (including the
> command interpreter).

Windows users expect software packages typical for Windows.

GRASS is the contrary of a monolithic executable, and should stay so
for various reasons. However, on Windows we need to fake a monolithic
system, because that is how Windows works, if we like it or not.
Having a command interpreter with a GRASS session running on Windows
is IMHO something for power users. On Linux/BSD/Unix, a command
interpreter is started first, then GRASS. On Windows it is the other
way around. The environment established by starting GRASS is (should
be) only valid for the current GRASS session.
>
>> > It's fairly trivial to set a valid GRASS environment globally, so that
>> > commands are usable in any shell. That's how I've had it on Linux
>> > since roughly forever; I don't run the grass70 script unless I'm
>> > testing it.
>>
>> At the very least, GRASS needs to clean up temporary files on exit.
>
> Temporary files need to be cleaned up periodically. The term "on exit"
> doesn't make sense for something which doesn't start or stop.

I thought that is the way GRASS operates currently: a process is
started and stopped when the user types "exit". At least I do that on
Linux when calling grass64 or grass70.
>
>> We use hard-coded special treatment for shell scripts and still they
>> behave like a GRASS module.
>
> Where?

We create .bat wrappers for shell scripts here:
https://trac.osgeo.org/grass/browser/grass/branches/releasebranch_6_4/include/Make/Script.make#L15
>
>> I don't see how users are bothered whether
>> a module is an executable or a shell script. Internally, GRASS can
>> always use system() or subprocess.Popen() for executables and .bat
>> files, but not for shell or Python scripts, Therefore we need
>> hard-coded special treatment for shell and Python scripts in order to
>> make sure that the correct interpreter is used.
>
> IIRC, .bat files were used because there were issues with associating
> .sh with MSys' shell (one of them being that MSys itself doesn't do
> that, so we'd effectively be forced to make global changes to
> something which doesn't "belong" to GRASS).

Msys is not a Windows shell script interpreter. IIUC, msys provides a
minimal, isolated system environment which needs to be started
explicitly.

Just as there were issues with associating .sh with MSys' shell, there
are issues when .py files are associated with the system's Python.
IMHO, any attempt of GRASS to associate .sh files or .py files on
Windows with a particular interpreter is doomed to fail. WinGRASS
includes all the needed interpreters and should use them.

I don't want to defend the way typical Windows software packages work,
but I think it is not worth the trouble to force Unix principles onto
Windows.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Fri, Feb 14, 2014 at 3:41 AM, Enrico Gallo wrote:

> Dear list,
>
> in my very limited experience, the same Python script for GRASS always
> runs without problems inside QGIS-GRASS Processing Plugin but
> sometimes fails in GRASS Enviroment, if other Python are installed on
> the PC (Stand alone, ARCGIS)
> Could be QGIS-GRASS Processing Initialization investigated?
>
> This is a good point. Why this is working? And also OSGeo4W was working
when WinGRASS was broken (with ArcGIS installed). So, what's the trick
there, anyway?

Vaclav

regards
>
> enrico gallo
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] Handling of Python scripts on MS Windows

[Enrico Gallo]

Dear list,

in my very limited experience, the same Python script for GRASS always
runs without problems inside QGIS-GRASS Processing Plugin but
sometimes fails in GRASS Enviroment, if other Python are installed on
the PC (Stand alone, ARCGIS)
Could be QGIS-GRASS Processing Initialization investigated?

regards

enrico gallo
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> This is the fundamental difference and the reason why using a
> system-wide Python can only cause trouble on Windows. On Windows, a
> software package typically includes everything it needs to run,

On Windows, a software package is typically a monolithic executable
with no ability to be controlled by other programs (including the
command interpreter).

> > It's fairly trivial to set a valid GRASS environment globally, so that
> > commands are usable in any shell. That's how I've had it on Linux
> > since roughly forever; I don't run the grass70 script unless I'm
> > testing it.
> 
> At the very least, GRASS needs to clean up temporary files on exit.

Temporary files need to be cleaned up periodically. The term "on exit"
doesn't make sense for something which doesn't start or stop.

> We use hard-coded special treatment for shell scripts and still they
> behave like a GRASS module.

Where?

> I don't see how users are bothered whether
> a module is an executable or a shell script. Internally, GRASS can
> always use system() or subprocess.Popen() for executables and .bat
> files, but not for shell or Python scripts, Therefore we need
> hard-coded special treatment for shell and Python scripts in order to
> make sure that the correct interpreter is used.

IIRC, .bat files were used because there were issues with associating
.sh with MSys' shell (one of them being that MSys itself doesn't do
that, so we'd effectively be forced to make global changes to
something which doesn't "belong" to GRASS).

> > Attempting to side-step the system's execution mechanism will *always*
> > be an incomplete solution.
> 
> I disagree because there is no default system execution mechanism for
> Python scripts: Python is not part of the Windows OS.
> The recommended Python version as of today would be Python 3.3 64 bit.

It's not necessarily "part of" all Linux distributions either, and
some will move to 3.x by default quite soon.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Benjamin Ducke wrote:

> How would users choose mapsets or create new ones in this
> scenario?

With g.mapset.

> > Things get marginally more complex if you either need multiple
> > versions installed concurrently, or multiple concurrent sessions.
> > 
> > The former is seldom supported, particularly for packages consisting
> > of many distinct utilities (having python2 and python3 executables is
> > one thing, versioning dozens or hundreds of distinct commands is
> > another).
> 
> Well, at this point I have several versions of GRASS 6.4 and
> a version of GRASS 7 installed. I need them all, because they
> run as testing versions or to provide processing functionality
> for another "host" GIS.

Right; but in the first situation, you're a "developer", not a "user". 
As such, you have to expect to do some customisation which normal
users wouldn't.

> How would I switch between them?

Modify GISBASE and environment variables which depend upon it (PATH,
LD_LIBRARY_PATH, PYTHONPATH).

> > The latter is fairly trivial to implement on Unix (just use $$ in the
> > setting of GISRC), and could be implemented on Windows (which doesn't
> > have the equivalent of ~/.profile etc) with fairly minor changes to
> > lib/gis/env.c.
> 
> But that would again require some user-friendly software, unless
> people are supposed to start editing configuration files, right?

That doesn't involve any user-visible software; it's part of the
installation (if you're installing an environment file as part of the
package, you install one which sets GISRC either per-user or
per-shell).

In the worst case, you can use the grass70 script to start a new
session which is disconnected from any existing session.

> Whatever the decision may be: Please make sure that it will always be
> possible to bundle a completely independent distribution of GRASS
> with 3rd party GIS. Otherwise, this project will no longer be able to
> benefit from the growing user base that projects such as QGIS have
> recently attracted.

A higher-level system which uses GRASS can configure its environment
however it wishes.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Mon, Feb 10, 2014 at 8:29 PM, Vaclav Petras  wrote:
>
>
>
> On Mon, Feb 10, 2014 at 2:17 PM, Markus Metz 
> wrote:
>>
>> On Mon, Feb 10, 2014 at 6:44 PM, Vaclav Petras 
>> wrote:
>> >
>> >
>> >
>> > On Mon, Feb 10, 2014 at 8:41 AM, Markus Metz
>> > 
>> > wrote:
>> >>
>> >> On Mon, Feb 10, 2014 at 12:04 PM, Moritz Lennert
>> >>  wrote:
>> >> > On 10/02/14 11:46, Markus Metz wrote:
>> >> >>
>> >> >> On Mon, Feb 10, 2014 at 11:26 AM, Helmut Kudrnovsky 
>> >> >> wrote:
>> >> >
>> >> > Therefore we need
>> >> > hard-coded special treatment for shell and Python scripts in
>> >> > order
>> >> > to
>> >> > make sure that the correct interpreter is used.
>> >> >>>
>> >> >>>
>> >>  Just for my understanding: When you say hard-coded special
>> >>  treatment
>> >>  for
>> >>  shell scripts, are you speaking about the .bat files ?
>> >> >>>
>> >> >>>
>> >> >>> I think yes.
>> >> >>
>> >> >>
>> >> >> Or more generally, any mechanism explicitly using %GRASS_PYTHON%
>> >> >> script.py.
>> >> >
>> >> >
>> >> > But as far as I've seen, this might not be sufficient since this only
>> >> > indicates which Python executable to use for launching the Python
>> >> > script,
>> >> > but any library calls linked to that execution will involve the
>> >> > system-wide
>> >> > installed Python. Which is different from bash scripts, where this is
>> >> > not an
>> >> > issue.
>> >>
>> >> GRASS Python scripts are currently executed using the system-wide
>> >> installed Python if it exists. No attempt has been made to explicitly
>> >> use GRASS_PYTHON, therefore it is not possible to say if the system's
>> >> Python would really be completely ignored.
>> >
>> >
>> > If I remember correctly, Python scripts were not working from Python
>> > scripts, they were working from command line.
>>
>> Which command line? If you used the msys command line, it should work
>> because within msys, the embedded GRASS_PYTHON version is readily
>> available:
>>
>> GRASS 7.0.svn> which python
>> /c/Programme/GRASS GIS 7.0.svn/extrabin/python.exe
>>
>> > And we were not able to
>> > explain why the right Python (or Python DLL) is used at one point but
>> > not
>> > the other.
>>
>> If you used the GUI, you are outside the msys shell and the system's
>> Python is used. Scripts invoke other GRASS modules with
>> grass.run_command() which uses the system's script interpreter if the
>> module is a script.
>>
> So, msys is able to configure our Python in the way that it is usable
> without interference with the system Python.

Maybe, but I did not test it thoroughly. Anyway, G7 is supposed to run
without the msys console which is optional.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Mon, Feb 10, 2014 at 2:17 PM, Markus Metz
wrote:

> On Mon, Feb 10, 2014 at 6:44 PM, Vaclav Petras 
> wrote:
> >
> >
> >
> > On Mon, Feb 10, 2014 at 8:41 AM, Markus Metz <
> markus.metz.gisw...@gmail.com>
> > wrote:
> >>
> >> On Mon, Feb 10, 2014 at 12:04 PM, Moritz Lennert
> >>  wrote:
> >> > On 10/02/14 11:46, Markus Metz wrote:
> >> >>
> >> >> On Mon, Feb 10, 2014 at 11:26 AM, Helmut Kudrnovsky 
> >> >> wrote:
> >> >
> >> > Therefore we need
> >> > hard-coded special treatment for shell and Python scripts in order
> >> > to
> >> > make sure that the correct interpreter is used.
> >> >>>
> >> >>>
> >>  Just for my understanding: When you say hard-coded special
> treatment
> >>  for
> >>  shell scripts, are you speaking about the .bat files ?
> >> >>>
> >> >>>
> >> >>> I think yes.
> >> >>
> >> >>
> >> >> Or more generally, any mechanism explicitly using %GRASS_PYTHON%
> >> >> script.py.
> >> >
> >> >
> >> > But as far as I've seen, this might not be sufficient since this only
> >> > indicates which Python executable to use for launching the Python
> >> > script,
> >> > but any library calls linked to that execution will involve the
> >> > system-wide
> >> > installed Python. Which is different from bash scripts, where this is
> >> > not an
> >> > issue.
> >>
> >> GRASS Python scripts are currently executed using the system-wide
> >> installed Python if it exists. No attempt has been made to explicitly
> >> use GRASS_PYTHON, therefore it is not possible to say if the system's
> >> Python would really be completely ignored.
> >
> >
> > If I remember correctly, Python scripts were not working from Python
> > scripts, they were working from command line.
>
> Which command line? If you used the msys command line, it should work
> because within msys, the embedded GRASS_PYTHON version is readily
> available:
>
> GRASS 7.0.svn> which python
> /c/Programme/GRASS GIS 7.0.svn/extrabin/python.exe
>
> > And we were not able to
> > explain why the right Python (or Python DLL) is used at one point but not
> > the other.
>
> If you used the GUI, you are outside the msys shell and the system's
> Python is used. Scripts invoke other GRASS modules with
> grass.run_command() which uses the system's script interpreter if the
> module is a script.
>
> So, msys is able to configure our Python in the way that it is usable
without interference with the system Python.

Markus M
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Mon, Feb 10, 2014 at 6:44 PM, Vaclav Petras  wrote:
>
>
>
> On Mon, Feb 10, 2014 at 8:41 AM, Markus Metz 
> wrote:
>>
>> On Mon, Feb 10, 2014 at 12:04 PM, Moritz Lennert
>>  wrote:
>> > On 10/02/14 11:46, Markus Metz wrote:
>> >>
>> >> On Mon, Feb 10, 2014 at 11:26 AM, Helmut Kudrnovsky 
>> >> wrote:
>> >
>> > Therefore we need
>> > hard-coded special treatment for shell and Python scripts in order
>> > to
>> > make sure that the correct interpreter is used.
>> >>>
>> >>>
>>  Just for my understanding: When you say hard-coded special treatment
>>  for
>>  shell scripts, are you speaking about the .bat files ?
>> >>>
>> >>>
>> >>> I think yes.
>> >>
>> >>
>> >> Or more generally, any mechanism explicitly using %GRASS_PYTHON%
>> >> script.py.
>> >
>> >
>> > But as far as I've seen, this might not be sufficient since this only
>> > indicates which Python executable to use for launching the Python
>> > script,
>> > but any library calls linked to that execution will involve the
>> > system-wide
>> > installed Python. Which is different from bash scripts, where this is
>> > not an
>> > issue.
>>
>> GRASS Python scripts are currently executed using the system-wide
>> installed Python if it exists. No attempt has been made to explicitly
>> use GRASS_PYTHON, therefore it is not possible to say if the system's
>> Python would really be completely ignored.
>
>
> If I remember correctly, Python scripts were not working from Python
> scripts, they were working from command line.

Which command line? If you used the msys command line, it should work
because within msys, the embedded GRASS_PYTHON version is readily
available:

GRASS 7.0.svn> which python
/c/Programme/GRASS GIS 7.0.svn/extrabin/python.exe

> And we were not able to
> explain why the right Python (or Python DLL) is used at one point but not
> the other.

If you used the GUI, you are outside the msys shell and the system's
Python is used. Scripts invoke other GRASS modules with
grass.run_command() which uses the system's script interpreter if the
module is a script.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

> And we were not able to explain why the right Python (or Python DLL) is
used at one point but not the other. 

it's maybe a path issue (?).

http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/path.mspx?mfr=true

"The operating system always searches in the current directory first, before
it searches the directories in the command path."



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5103011.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

wenzeslaus wrote
> If I remember correctly, Python scripts were not working from Python
> scripts, they were working from command line. And we were not able to
> explain why the right Python (or Python DLL) is used at one point but not
> the other. If there wouldn't be shell=True [1], I would say that we need
> to
> add it.
> 
> [1]
> http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/script/core.py?rev=58952#L52

please see
http://lists.osgeo.org/pipermail/grass-dev/2014-February/067307.html and 
http://trac.osgeo.org/grass/ticket/2138

Python scripts _are_ working from Python scripts, if e.g. in a related
bat-file GRASS_PYTHON is defined.



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5103009.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

>GRASS Python scripts are currently executed using the system-wide
>installed Python if it exists. No attempt has been made to explicitly
>use GRASS_PYTHON, therefore it is not possible to say if the system's
>Python would really be completely ignored. 

it is (partly) implemented ( and tested on Win7 with a system-wide
ArcGIS-python) at least for python-addons for winGRASS6.4.svn/6.5.svn:

see

http://trac.osgeo.org/grass/ticket/2138#comment:8

and some local tests here shows (see
http://trac.osgeo.org/grass/ticket/2138):

"the python script (which invokes two other python scripts) finishes now."

how to test if the system's Python would really be completely ignored?



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5103005.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Mon, Feb 10, 2014 at 8:41 AM, Markus Metz
wrote:

> On Mon, Feb 10, 2014 at 12:04 PM, Moritz Lennert
>  wrote:
> > On 10/02/14 11:46, Markus Metz wrote:
> >>
> >> On Mon, Feb 10, 2014 at 11:26 AM, Helmut Kudrnovsky 
> wrote:
> >
> > Therefore we need
> > hard-coded special treatment for shell and Python scripts in order to
> > make sure that the correct interpreter is used.
> >>>
> >>>
>  Just for my understanding: When you say hard-coded special treatment
> for
>  shell scripts, are you speaking about the .bat files ?
> >>>
> >>>
> >>> I think yes.
> >>
> >>
> >> Or more generally, any mechanism explicitly using %GRASS_PYTHON%
> >> script.py.
> >
> >
> > But as far as I've seen, this might not be sufficient since this only
> > indicates which Python executable to use for launching the Python script,
> > but any library calls linked to that execution will involve the
> system-wide
> > installed Python. Which is different from bash scripts, where this is
> not an
> > issue.
>
> GRASS Python scripts are currently executed using the system-wide
> installed Python if it exists. No attempt has been made to explicitly
> use GRASS_PYTHON, therefore it is not possible to say if the system's
> Python would really be completely ignored.
>

If I remember correctly, Python scripts were not working from Python
scripts, they were working from command line. And we were not able to
explain why the right Python (or Python DLL) is used at one point but not
the other. If there wouldn't be shell=True [1], I would say that we need to
add it.

[1]
http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/script/core.py?rev=58952#L52



> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Mon, Feb 10, 2014 at 12:04 PM, Moritz Lennert
 wrote:
> On 10/02/14 11:46, Markus Metz wrote:
>>
>> On Mon, Feb 10, 2014 at 11:26 AM, Helmut Kudrnovsky  wrote:
>
> Therefore we need
> hard-coded special treatment for shell and Python scripts in order to
> make sure that the correct interpreter is used.
>>>
>>>
 Just for my understanding: When you say hard-coded special treatment for
 shell scripts, are you speaking about the .bat files ?
>>>
>>>
>>> I think yes.
>>
>>
>> Or more generally, any mechanism explicitly using %GRASS_PYTHON%
>> script.py.
>
>
> But as far as I've seen, this might not be sufficient since this only
> indicates which Python executable to use for launching the Python script,
> but any library calls linked to that execution will involve the system-wide
> installed Python. Which is different from bash scripts, where this is not an
> issue.

GRASS Python scripts are currently executed using the system-wide
installed Python if it exists. No attempt has been made to explicitly
use GRASS_PYTHON, therefore it is not possible to say if the system's
Python would really be completely ignored.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Moritz Lennert]

On 10/02/14 11:46, Markus Metz wrote:

On Mon, Feb 10, 2014 at 11:26 AM, Helmut Kudrnovsky  wrote:

Therefore we need
hard-coded special treatment for shell and Python scripts in order to
make sure that the correct interpreter is used.



Just for my understanding: When you say hard-coded special treatment for
shell scripts, are you speaking about the .bat files ?


I think yes.


Or more generally, any mechanism explicitly using %GRASS_PYTHON% script.py.


But as far as I've seen, this might not be sufficient since this only 
indicates which Python executable to use for launching the Python 
script, but any library calls linked to that execution will involve the 
system-wide installed Python. Which is different from bash scripts, 
where this is not an issue.


Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Mon, Feb 10, 2014 at 11:26 AM, Helmut Kudrnovsky  wrote:
>>>Therefore we need
>>> hard-coded special treatment for shell and Python scripts in order to
>>> make sure that the correct interpreter is used.
>
>>Just for my understanding: When you say hard-coded special treatment for
>>shell scripts, are you speaking about the .bat files ?
>
> I think yes.

Or more generally, any mechanism explicitly using %GRASS_PYTHON% script.py.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

>>Therefore we need
>> hard-coded special treatment for shell and Python scripts in order to
>> make sure that the correct interpreter is used.

>Just for my understanding: When you say hard-coded special treatment for
>shell scripts, are you speaking about the .bat files ? 

I think yes.



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5102860.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Moritz Lennert]

On 10/02/14 09:57, Markus Metz wrote:

Glynn Clements wrote:

This is my entire point. But that cannot be achieved if you're relying
upon hard-coded special treatment for Python scripts. If Python
scripts cannot simply be executed (via system() or subprocess.Popen()
or whatever) in the same manner as executables or other scripts, the
user will have to be bothered with whether something is an executable
or a Python script.


We use hard-coded special treatment for shell scripts and still they
behave like a GRASS module. I don't see how users are bothered whether
a module is an executable or a shell script. Internally, GRASS can
always use system() or subprocess.Popen() for executables and .bat
files, but not for shell or Python scripts, Therefore we need
hard-coded special treatment for shell and Python scripts in order to
make sure that the correct interpreter is used.


Just for my understanding: When you say hard-coded special treatment for 
shell scripts, are you speaking about the .bat files ?


Moritz

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

Glynn Clements wrote:
>
> Markus Metz wrote:
>
>> >> Other projects such as gimp or libreoffice are AFAICT reasonably
>> >> bundled with Python, without a Python installer.
>> >
>> > They aren't attempting to support Python scripts as stand-alone
>> > programs (i.e. something which can be run from the command prompt,
>> > batch files, etc).
>>
>> GRASS scripts, just like GRASS C modules, only run as stand-alone
>> programs if the GRASS environment is set properly and if a valid GRASS
>> session has been established.
>
> This is a matter of creating a file and setting some environment
> variables.
>
> It's also not that specific to GRASS. As the Windows filesystem layout
> groups files by origin (i.e. .../Program Files//)
> rather than by function (.../bin, .../lib, etc), it's fairly typical
> to have to at least modify PATH for anything which is meant to be used
> from the command line.

This is the fundamental difference and the reason why using a
system-wide Python can only cause trouble on Windows. On Windows, a
software package typically includes everything it needs to run,
embedding any script interpreters, while Linux, Unix, BSD systems use
package managers. The fact that ArcGIS installs a system-wide Python
is maybe a curiosity coming from the origins of ArcInfo as a Unix
application.

>
> But you can't configure interpreters on a per-process basis on Windows.
> You can only do it on Unix because we use the "#!/usr/bin/env python"
> hack (which was originally invented to deal with Python being either
> in /usr/bin or /usr/local/bin).
>
>> Thus, GRASS modules do not run from any
>> command prompt, neither from the native Windows command prompt nor
>> from a *NIX shell, without setting the GRASS environment first. Real
>> stand-alone programs are for example the GDAL utilities, but not GRASS
>> modules.
>
> It's fairly trivial to set a valid GRASS environment globally, so that
> commands are usable in any shell. That's how I've had it on Linux
> since roughly forever; I don't run the grass70 script unless I'm
> testing it.

At the very least, GRASS needs to clean up temporary files on exit.
>
> FWIW, I want to make that the default mode of operation.
>
> I.e. on Unix, set the environment variables in /etc/profile.d/grass7
> or whatever a given distribution uses, so that you only need to
> explicitly start GRASS sessions if you want multiple, independent
> sessions.

I have multiple, independent sessions running pretty much every day.
>
>> > A compiled program which "embeds" Python (links against the Python
>> > DSO/DLL) to use Python for "internal" scripting is a much simpler
>> > case. And much less useful.
>>
>> Why much less useful? As far as GRASS modules are concerned, users
>> should not be bothered and should not care if a GRASS module is a C
>> module or a shell script or a python script. First of all, a GRASS
>> module must behave like a GRASS module, independent of the language in
>> which it is written.
>
> This is my entire point. But that cannot be achieved if you're relying
> upon hard-coded special treatment for Python scripts. If Python
> scripts cannot simply be executed (via system() or subprocess.Popen()
> or whatever) in the same manner as executables or other scripts, the
> user will have to be bothered with whether something is an executable
> or a Python script.

We use hard-coded special treatment for shell scripts and still they
behave like a GRASS module. I don't see how users are bothered whether
a module is an executable or a shell script. Internally, GRASS can
always use system() or subprocess.Popen() for executables and .bat
files, but not for shell or Python scripts, Therefore we need
hard-coded special treatment for shell and Python scripts in order to
make sure that the correct interpreter is used.

IMHO, the argument to have GRASS as a GIS processing engine in the
background (e.g. for Sextante) that includes everything it needs and
does not interfere with other software packages is a pretty strong
argument.
>
> Attempting to side-step the system's execution mechanism will *always*
> be an incomplete solution.

I disagree because there is no default system execution mechanism for
Python scripts: Python is not part of the Windows OS.
The recommended Python version as of today would be Python 3.3 64 bit.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

hi,

> But there are also downsides to this: We would need to think about
> different mechanisms for each supported OS; we could get tangled up in
> all sorts of flawed decisions by the OS designers;

I think we are already there regarding MS windos OS and python ...

>Whatever the decision may be: Please make sure that it will always be
>possible to bundle a completely independent distribution of GRASS
>with 3rd party GIS. Otherwise, this project will no longer be able to
>benefit from the growing user base that projects such as QGIS have
>recently attracted. 

I second this.



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5102711.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Benjamin Ducke]

On 08/02/14 20:41, Glynn Clements wrote:
> 
> Markus Neteler wrote:
> 
>>> The difference is that you don't "start" GRASS. You set the required
>>> environment variables from e.g. ~/.profile so that GRASS commands work
>>> in any shell (or via any other execution mechanism, e.g. M-! from
>>> within Emacs).
>>
>> This is hardly feasible for the majority of the users. Perhaps I don't
>> understand the suggestion, at least for Windows OS users (nad most
>> Linux/Mac newcomers).
> 
> It's not something users should need to concern themselves with.
> 
> Many packages require certain environment settings (which is why most
> modern Linux systems have e.g. /etc/profile.d or /etc/env.d, with each
> file belonging to a different package).
> 
> Most packages made up of command-line utilities don't require you to
> "start" the package before you can use the commands which comprise it.
> 
> It would be quite straightforward for a GRASS package to install an
> environment file which sets GISBASE and GISRC so that GRASS commands
> just work anywhere.
> 

I have been trying to wrap my head around this.
Please tell me if I still misunderstand something,
but this is what I think you are suggesting:

Instead of having to log into a GRASS session
explicitely, the GRASS environment gets configured
with the login (shell) and then the user can just run
e.g. "g.list" and it will just work.

How would users choose mapsets or create new ones in this
scenario? Wouldn't that require new user level software?

> Things get marginally more complex if you either need multiple
> versions installed concurrently, or multiple concurrent sessions.
> 
> The former is seldom supported, particularly for packages consisting
> of many distinct utilities (having python2 and python3 executables is
> one thing, versioning dozens or hundreds of distinct commands is
> another).
> 

Well, at this point I have several versions of GRASS 6.4 and
a version of GRASS 7 installed. I need them all, because they
run as testing versions or to provide processing functionality
for another "host" GIS.

How would I switch between them?

The same situation arises with different Java versions on the
same system. It is a very frequent situation and IMHO there
is no really good, convenient solution for this. Every Linux
distribution seems to use its own frontend for switching and
let's not even get started about Windows.

The alternatives here are to either manipulate PATH etc.
directly or to provide some more user-friendly software that
take care of this. I find neither ideal.

> The latter is fairly trivial to implement on Unix (just use $$ in the
> setting of GISRC), and could be implemented on Windows (which doesn't
> have the equivalent of ~/.profile etc) with fairly minor changes to
> lib/gis/env.c.
> 

But that would again require some user-friendly software, unless
people are supposed to start editing configuration files, right?

I understand the benefits of tighter system integration that (I think)
you are aiming for. It would make everything more seamless.
But there are also downsides to this: We would need to think about
different mechanisms for each supported OS; we could get tangled up in
all sorts of flawed decisions by the OS designers; we would depend on
3rd party dependencies to work with GRASS. Surely, all of these issues
could be worked around, but are the potential benefits worth it?

Whatever the decision may be: Please make sure that it will always be
possible to bundle a completely independent distribution of GRASS
with 3rd party GIS. Otherwise, this project will no longer be able to
benefit from the growing user base that projects such as QGIS have
recently attracted.

Best,

Ben

-- 
Dr. Benjamin Ducke, M.A.
{*} Geospatial Consultant
{*} GIS Developer

  bendu...@fastmail.fm
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Neteler wrote:

> > The difference is that you don't "start" GRASS. You set the required
> > environment variables from e.g. ~/.profile so that GRASS commands work
> > in any shell (or via any other execution mechanism, e.g. M-! from
> > within Emacs).
> 
> This is hardly feasible for the majority of the users. Perhaps I don't
> understand the suggestion, at least for Windows OS users (nad most
> Linux/Mac newcomers).

It's not something users should need to concern themselves with.

Many packages require certain environment settings (which is why most
modern Linux systems have e.g. /etc/profile.d or /etc/env.d, with each
file belonging to a different package).

Most packages made up of command-line utilities don't require you to
"start" the package before you can use the commands which comprise it.

It would be quite straightforward for a GRASS package to install an
environment file which sets GISBASE and GISRC so that GRASS commands
just work anywhere.

Things get marginally more complex if you either need multiple
versions installed concurrently, or multiple concurrent sessions.

The former is seldom supported, particularly for packages consisting
of many distinct utilities (having python2 and python3 executables is
one thing, versioning dozens or hundreds of distinct commands is
another).

The latter is fairly trivial to implement on Unix (just use $$ in the
setting of GISRC), and could be implemented on Windows (which doesn't
have the equivalent of ~/.profile etc) with fairly minor changes to
lib/gis/env.c.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Neteler]

On Fri, Feb 7, 2014 at 5:49 PM, Glynn Clements  wrote:
>
> Markus Neteler wrote:
>
>> > It's fairly trivial to set a valid GRASS environment globally, so that
>> > commands are usable in any shell. That's how I've had it on Linux
>> > since roughly forever; I don't run the grass70 script unless I'm
>> > testing it.
>> >
>> > FWIW, I want to make that the default mode of operation.
>>
>> Could you please elaborate? Say, what would be the difference to the
>> current way?
>
> The difference is that you don't "start" GRASS. You set the required
> environment variables from e.g. ~/.profile so that GRASS commands work
> in any shell (or via any other execution mechanism, e.g. M-! from
> within Emacs).

This is hardly feasible for the majority of the users. Perhaps I don't
understand the suggestion, at least for Windows OS users (nad most
Linux/Mac newcomers).

Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Fri, Feb 7, 2014 at 11:49 AM, Glynn Clements wrote:

>
> Markus Neteler wrote:
>
> > > It's fairly trivial to set a valid GRASS environment globally, so that
> > > commands are usable in any shell. That's how I've had it on Linux
> > > since roughly forever; I don't run the grass70 script unless I'm
> > > testing it.
> > >
> > > FWIW, I want to make that the default mode of operation.
> >
> > Could you please elaborate? Say, what would be the difference to the
> > current way?
>
> The difference is that you don't "start" GRASS. You set the required
> environment variables from e.g. ~/.profile so that GRASS commands work
> in any shell (or via any other execution mechanism, e.g. M-! from
> within Emacs).
>
>
I'm still not sure if this is the best practice. I though I saw on mailing
list that somebody claimed that it is not safe. Not sure why, perhaps
because you are not getting the cleaning up procedures at the beginning and
at the end of GRASS session? Or how this is supposed to work with multiple
grass installations, which is a pretty common case?

Vaclav

> --
> Glynn Clements 
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Neteler wrote:

> > It's fairly trivial to set a valid GRASS environment globally, so that
> > commands are usable in any shell. That's how I've had it on Linux
> > since roughly forever; I don't run the grass70 script unless I'm
> > testing it.
> >
> > FWIW, I want to make that the default mode of operation.
> 
> Could you please elaborate? Say, what would be the difference to the
> current way?

The difference is that you don't "start" GRASS. You set the required
environment variables from e.g. ~/.profile so that GRASS commands work
in any shell (or via any other execution mechanism, e.g. M-! from
within Emacs).

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Neteler]

On Wed, Feb 5, 2014 at 5:19 PM, Glynn Clements  wrote:
...
> It's fairly trivial to set a valid GRASS environment globally, so that
> commands are usable in any shell. That's how I've had it on Linux
> since roughly forever; I don't run the grass70 script unless I'm
> testing it.
>
> FWIW, I want to make that the default mode of operation.

Could you please elaborate? Say, what would be the difference to the
current way?

> I.e. on Unix, set the environment variables in /etc/profile.d/grass7
> or whatever a given distribution uses, so that you only need to
> explicitly start GRASS sessions if you want multiple, independent
> sessions.
>
> On Windows, GISBASE should be set from the registry (with the
> environment variables allowing this to be overriden, mainly as a
> developer feature).

Not sure what this would mean to the average user...

thanks for some more insights,
Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Helmut Kudrnovsky]

Glynn Clements wrote
> Markus Metz wrote:
> 
>> >> Other projects such as gimp or libreoffice are AFAICT reasonably
>> >> bundled with Python, without a Python installer.
>> >
>> > They aren't attempting to support Python scripts as stand-alone
>> > programs (i.e. something which can be run from the command prompt,
>> > batch files, etc).
>> 
>> GRASS scripts, just like GRASS C modules, only run as stand-alone
>> programs if the GRASS environment is set properly and if a valid GRASS
>> session has been established.
> 
> This is a matter of creating a file and setting some environment
> variables.
> 
> It's also not that specific to GRASS. As the Windows filesystem layout
> groups files by origin (i.e. .../Program Files/
> 
> /
> 
> )
> rather than by function (.../bin, .../lib, etc), it's fairly typical
> to have to at least modify PATH for anything which is meant to be used
> from the command line.
> 
> But you can't configure interpreters on a per-process basis on Windows.
> You can only do it on Unix because we use the "#!/usr/bin/env python"
> hack (which was originally invented to deal with Python being either
> in /usr/bin or /usr/local/bin).
> 
>> Thus, GRASS modules do not run from any
>> command prompt, neither from the native Windows command prompt nor
>> from a *NIX shell, without setting the GRASS environment first. Real
>> stand-alone programs are for example the GDAL utilities, but not GRASS
>> modules.
> 
> It's fairly trivial to set a valid GRASS environment globally, so that
> commands are usable in any shell. That's how I've had it on Linux
> since roughly forever; I don't run the grass70 script unless I'm
> testing it.
> 
> FWIW, I want to make that the default mode of operation.
> 
> I.e. on Unix, set the environment variables in /etc/profile.d/grass7
> or whatever a given distribution uses, so that you only need to
> explicitly start GRASS sessions if you want multiple, independent
> sessions.
> 
> On Windows, GISBASE should be set from the registry (with the
> environment variables allowing this to be overriden, mainly as a
> developer feature).
> 
>> > A compiled program which "embeds" Python (links against the Python
>> > DSO/DLL) to use Python for "internal" scripting is a much simpler
>> > case. And much less useful.
>> 
>> Why much less useful? As far as GRASS modules are concerned, users
>> should not be bothered and should not care if a GRASS module is a C
>> module or a shell script or a python script. First of all, a GRASS
>> module must behave like a GRASS module, independent of the language in
>> which it is written.
> 
> This is my entire point. But that cannot be achieved if you're relying
> upon hard-coded special treatment for Python scripts. If Python
> scripts cannot simply be executed (via system() or subprocess.Popen()
> or whatever) in the same manner as executables or other scripts, the
> user will have to be bothered with whether something is an executable
> or a Python script.
> 
> Attempting to side-step the system's execution mechanism will *always*
> be an incomplete solution.
> 
> -- 
> Glynn Clements

please keep in mind that we are providing WinGRASS in different ways (most
used as WinGRASS-standalone and OSGeo4W-WinGRASS; others are QGIS, gvSIGce,
etc) for our users aka community. 

IMHO it would be really a pity to lo lose any of those for our community.

thanks




-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Handling-of-Python-scripts-on-MS-Windows-tp5081335p5102228.html
Sent from the Grass - Dev mailing list archive at Nabble.com.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Markus Metz wrote:

> >> Other projects such as gimp or libreoffice are AFAICT reasonably
> >> bundled with Python, without a Python installer.
> >
> > They aren't attempting to support Python scripts as stand-alone
> > programs (i.e. something which can be run from the command prompt,
> > batch files, etc).
> 
> GRASS scripts, just like GRASS C modules, only run as stand-alone
> programs if the GRASS environment is set properly and if a valid GRASS
> session has been established.

This is a matter of creating a file and setting some environment
variables.

It's also not that specific to GRASS. As the Windows filesystem layout
groups files by origin (i.e. .../Program Files//)
rather than by function (.../bin, .../lib, etc), it's fairly typical
to have to at least modify PATH for anything which is meant to be used
from the command line.

But you can't configure interpreters on a per-process basis on Windows.
You can only do it on Unix because we use the "#!/usr/bin/env python"
hack (which was originally invented to deal with Python being either
in /usr/bin or /usr/local/bin).

> Thus, GRASS modules do not run from any
> command prompt, neither from the native Windows command prompt nor
> from a *NIX shell, without setting the GRASS environment first. Real
> stand-alone programs are for example the GDAL utilities, but not GRASS
> modules.

It's fairly trivial to set a valid GRASS environment globally, so that
commands are usable in any shell. That's how I've had it on Linux
since roughly forever; I don't run the grass70 script unless I'm
testing it.

FWIW, I want to make that the default mode of operation.

I.e. on Unix, set the environment variables in /etc/profile.d/grass7
or whatever a given distribution uses, so that you only need to
explicitly start GRASS sessions if you want multiple, independent
sessions.

On Windows, GISBASE should be set from the registry (with the
environment variables allowing this to be overriden, mainly as a
developer feature).

> > A compiled program which "embeds" Python (links against the Python
> > DSO/DLL) to use Python for "internal" scripting is a much simpler
> > case. And much less useful.
> 
> Why much less useful? As far as GRASS modules are concerned, users
> should not be bothered and should not care if a GRASS module is a C
> module or a shell script or a python script. First of all, a GRASS
> module must behave like a GRASS module, independent of the language in
> which it is written.

This is my entire point. But that cannot be achieved if you're relying
upon hard-coded special treatment for Python scripts. If Python
scripts cannot simply be executed (via system() or subprocess.Popen()
or whatever) in the same manner as executables or other scripts, the
user will have to be bothered with whether something is an executable
or a Python script.

Attempting to side-step the system's execution mechanism will *always*
be an incomplete solution.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Glynn Clements]

Vaclav Petras wrote:

> > But with the
> > latest Python developments, it will be a real challenge to
> > integrate GRASS 7 in the same way that we could integrate
> > GRASS 6 into a "host" application (i.e. without messing
> > around with any system settings).
> 
> Can you be more specific? It sounds like something we should be concern
> about. We should consider how GRASS 7 will be integrated into other GIS
> packages when Python scripts are called this or that way.

If Python is "installed", then there shouldn't be any difference
between a Python script, a shell script (or batch file), or a compiled
executable.

Personally, I consider this to be non-optional. Python scripts have to
work like any other script or compiled program, therefore the Python
interpreter must be "installed" on the system and we must rely upon
the system's native execution mechanism(s).

Any approach which tries to sidestep this requirement will always be
incomplete. It can only work so long as it controls the entire chain
of execution down to the point where the script is invoked.

E.g. checking whether a command is an EXE or a Python script and
providing different treatment for each case will work in and of
itself, but it will then fail if it tries to execute a Python script
and doesn't use the same mechanism itself (e.g. because it's a
third-party program rather than part of GRASS).

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Neteler]

On Tue, Feb 4, 2014 at 10:08 PM, Markus Metz
 wrote:
...
> We should use the embedded Python explicitly to avoid this problem. We
> were here already. From a user's perspective, the easiest would be if
> GRASS ignores any system-wide Python installation.

This view is IMHO already confirmed by the fact that ArcGIS users
which comes with an own Python interpreter still continuously run into
troubles when having GRASS installed instead of not interfering...
(just got another offlist confirmation some hours ago).

markusN
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Markus Metz]

On Tue, Feb 4, 2014 at 11:29 AM, Glynn Clements
 wrote:
>
> Markus Metz wrote:
>
>> Other projects such as gimp or libreoffice are AFAICT reasonably
>> bundled with Python, without a Python installer.
>
> They aren't attempting to support Python scripts as stand-alone
> programs (i.e. something which can be run from the command prompt,
> batch files, etc).

GRASS scripts, just like GRASS C modules, only run as stand-alone
programs if the GRASS environment is set properly and if a valid GRASS
session has been established. Thus, GRASS modules do not run from any
command prompt, neither from the native Windows command prompt nor
from a *NIX shell, without setting the GRASS environment first. Real
stand-alone programs are for example the GDAL utilities, but not GRASS
modules.
>
> A compiled program which "embeds" Python (links against the Python
> DSO/DLL) to use Python for "internal" scripting is a much simpler
> case. And much less useful.

Why much less useful? As far as GRASS modules are concerned, users
should not be bothered and should not care if a GRASS module is a C
module or a shell script or a python script. First of all, a GRASS
module must behave like a GRASS module, independent of the language in
which it is written.
>
>> > Most of the troubles arise from attempting to use a mixture of a
>> > bundled version and a system-wide installation.
>>
>> I agree. It seems that the wxGUI always uses the bundled GRASS_PYTHON,
>> the reason why the wxGUI works fine without a system-wide Python.
>
> That was done so that the system Python doesn't need to have wxPython
> installed. Note that wxpyimgview does something similar: the top-level
> script invokes the wxPython program via GRASS_PYTHON explicitly.

Not a good idea, as long as the embedded Python is not always used
explicitely. This mixes the embedded Python with any existing system
Python which in turn causes the infamous MAXREPEAT error.

>>
>> Other projects do bundle Python, and there it seems to work just fine.
>
> They don't merely bundle it, they embed it, which is a much simpler
> case. The downside is that Python can only be used within a monolithic
> GUI application, which isn't how GRASS works.

I disagree. On GRASS 6, the shell interpreter is embedded, and GRASS
can be used in a perfectly modular way, e.g. by the Sextante plugin.
Wrapping Python scripts in .bat files, just like shell scripts are
wrapped in .bat files for GRASS 6, preserves the modular operation of
GRASS.

>
>> I don't understand why we can not also bundle Python, including all
>> required Python packages.

I meant embed, not bundle.

>>
>> Otherwise, users would need to install Python, wxPython, numpy, scipy,
>> matplotlib, datetime themselves (not sure if the list is correct and
>> complete).
>
> We can facilitate this in most cases. If there isn't already a Python
> installation, we can just install the latest 2.x version from the MSI,
> and all required packages. If there is a suitable Python installation,
> we can leave that and just install any required packages.

But after the next upgrade of the system Python, GRASS might no longer
be working, and the user is confused.
>
> The problem only arises when there is an existing installation which
> is unsuitable (incompatible version of Python or additional packages).
> In that situation, either the user has to make a choice or we need to
> figure out how to use the launcher.

We should use the embedded Python explicitly to avoid this problem. We
were here already. From a user's perspective, the easiest would be if
GRASS ignores any system-wide Python installation.

Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Benjamin Ducke]

On 04/02/14 16:42, Vaclav Petras wrote:
> 
> 
> 
> On Tue, Feb 4, 2014 at 10:14 AM, Benjamin Ducke  > wrote:
> 
> On 04/02/14 15:09, Vaclav Petras wrote:
> >
> > On Tue, Feb 4, 2014 at 8:25 AM, Benjamin Ducke
> mailto:bendu...@fastmail.fm>
> > >> wrote:
> >
> > But with the
> > latest Python developments, it will be a real challenge to
> > integrate GRASS 7 in the same way that we could integrate
> > GRASS 6 into a "host" application (i.e. without messing
> > around with any system settings).
> >
> >
> > Can you be more specific? It sounds like something we should be
> concern
> > about. We should consider how GRASS 7 will be integrated into
> other GIS
> > packages when Python scripts are called this or that way.
> 
> Sure. At this moment, we (i.e. the gvSIG CE dev team) distribute
> GRASS 6 binaries along with gvSIG CE. On Windows, we also package
> MSYS with "sh.exe" for the scripted modules.
> 
> So we can deliever gvSIG CE + GRASS as a completely self-contained
> (portable) package.
> 
> Of course, we do not ship any GRASS GUI (or any of the d.* modules).
> All GRASS modules are available seamlessly from the gvSIG CE GUI via a
> Plug-in, similar to that in QGIS.
> 
> Now, with the new Python-scripted modules in GRASS 7 this will
> become more difficult, since Python adds a more heavy dependency.
> In order to stay portable, we need to bundle a completely isolated
> Python interpreter with gvSIG CE. And then we also need to make
> sure that no GRASS Python module will call the system Python. The
> latter is something that can only be taken care of on the GRASS
> side.
> 
> 
> So, what would be the consequences for gvSIG, QGIS, OSGeo4W, ... if we
> decide to use system Python in (standalone) WinGRASS?

As far as gvSIG CE is concerned, there would be no consequences,
as long as it is possible to also make a GRASS distribution that
does not depend on system Python.

We ship our own, bundled GRASS binaries and users can just install
a WinGRASS (or any other GRASS version) in parallel. However,
in order to be able to bundle GRASS 7 in the future, none
of its modules should depend on system Python. If there were such
a dependency, then we would have to drop the Python-scripted
modules from the set of GRASS modules supported in gvSIG CE, as
we do not want to compromise our software's portability.

Best,

Ben


-- 
Dr. Benjamin Ducke, M.A.
{*} Geospatial Consultant
{*} GIS Developer

  bendu...@fastmail.fm
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Handling of Python scripts on MS Windows

[Vaclav Petras]

On Tue, Feb 4, 2014 at 10:14 AM, Benjamin Ducke wrote:

> On 04/02/14 15:09, Vaclav Petras wrote:
> >
> > On Tue, Feb 4, 2014 at 8:25 AM, Benjamin Ducke  > > wrote:
> >
> > But with the
> > latest Python developments, it will be a real challenge to
> > integrate GRASS 7 in the same way that we could integrate
> > GRASS 6 into a "host" application (i.e. without messing
> > around with any system settings).
> >
> >
> > Can you be more specific? It sounds like something we should be concern
> > about. We should consider how GRASS 7 will be integrated into other GIS
> > packages when Python scripts are called this or that way.
>
> Sure. At this moment, we (i.e. the gvSIG CE dev team) distribute
> GRASS 6 binaries along with gvSIG CE. On Windows, we also package
> MSYS with "sh.exe" for the scripted modules.
>
> So we can deliever gvSIG CE + GRASS as a completely self-contained
> (portable) package.
>
> Of course, we do not ship any GRASS GUI (or any of the d.* modules).
> All GRASS modules are available seamlessly from the gvSIG CE GUI via a
> Plug-in, similar to that in QGIS.
>
> Now, with the new Python-scripted modules in GRASS 7 this will
> become more difficult, since Python adds a more heavy dependency.
> In order to stay portable, we need to bundle a completely isolated
> Python interpreter with gvSIG CE. And then we also need to make
> sure that no GRASS Python module will call the system Python. The
> latter is something that can only be taken care of on the GRASS
> side.
>

So, what would be the consequences for gvSIG, QGIS, OSGeo4W, ... if we
decide to use system Python in (standalone) WinGRASS?
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev