[issue467455] Enhanced environment variables

[admin]

Change by admin :


--
github: None -> 35272

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue451285] distutils ignores environment variables

[admin]

Change by admin :


--
github: None -> 34969

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24493] subprocess with env=os.environ doesn't preserve environment variables when calling a 32bit process on Windows 8.1

[Eryk Sun]

Change by Eryk Sun :


--
resolution:  -> third party
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Steve Dower]

Steve Dower  added the comment:

Distutils is now deprecated (see PEP 632) and all tagged issues are being 
closed. From now until removal, only release blocking issues will be considered 
for distutils.

If this issue does not relate to distutils, please remove the component and 
reopen it. If you believe it still requires a fix, most likely the issue should 
be re-reported at https://github.com/pypa/setuptools

--
resolution:  -> out of date
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42983] sys.argv incorrectly parses command lines with args in environment variables.

[Guido van Rossum]

Guido van Rossum  added the comment:

That looks like you need to learn more about the shell. It is not a Python bug.

--
nosy: +gvanrossum
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42983] sys.argv incorrectly parses command lines with args in environment variables.

[Jason Williams]

Change by Jason Williams :


Added file: https://bugs.python.org/file49754/private_key.pem

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42983] sys.argv incorrectly parses command lines with args in environment variables.

[Jason Williams]

Change by Jason Williams :


--
components: +Library (Lib)
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42983] sys.argv incorrectly parses command lines with args in environment variables.

[Jason Williams]

New submission from Jason Williams :

Passing arguments to a Python script using an environment variable like:

export en_auth="--arg1 test --arg2 \"$(https://bugs.python.org/file49753/test.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42761] Why does python's Popen fail to pass environment variables on Mac OS X?

[Benjamin Peterson]

Change by Benjamin Peterson :


--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42761] Why does python's Popen fail to pass environment variables on Mac OS X?

[AustEcon]

New submission from AustEcon :

I have described the issue in a stackoverflow question here:

https://stackoverflow.com/questions/65466303/why-does-pythons-popen-fail-to-pass-environment-variables-on-mac-os-x


In summary, I want to pass environment variables to a child process on Mac OS 
(Big Sur) with Popen but it is not working (works on Win32 and linux platforms).

Any assistance much appreciated,
AustEcon

--
components: Library (Lib)
messages: 383877
nosy: AustEcon
priority: normal
severity: normal
status: open
title: Why does python's Popen fail to pass environment variables on Mac OS X?
type: behavior
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue42761>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Senthil Kumaran]

Senthil Kumaran  added the comment:

I spent some time reviewing and researching the specification. It also says

   The server is not required to create meta-variables for all the
   header fields that it receives.  

And this in issue, open since 2010, we have issue two different set of 
variables one from Apache and from Nginx. So, it Is not certain if http.server 
should be alinged it any or all, and plus if anything is required.

The discussion on QUERY_STRING was noted, but as Pierre pointed out it was set 
too

for k in ('QUERY_STRING', 'REMOTE_HOST', 'CONTENT_LENGTH',
  'HTTP_USER_AGENT', 'HTTP_COOKIE', 'HTTP_REFERER'):
env.setdefault(k, "")

For cosmetic purpose, I could remove the existing if condition - 
https://github.com/python/cpython/pull/23604

I am not sure if we need to add other variables with an empty string value for 
any reason. 

As a maintainer, I think, we should close this issue.

If there is a bug report, like issue5054, then that is a valid issue, and we 
should fix it.  If there any specific issues raised with parsing or lack of 
"required" meta variable that caused the application to break, even that could 
be fixed.

I am closing this issue with a cosmetic change that stemmed out from the 
discussion - https://github.com/python/cpython/pull/23604

--
resolution:  -> wont fix
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.10 -Python 2.7, Python 3.2, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Senthil Kumaran]

Change by Senthil Kumaran :


--
keywords: +patch
pull_requests: +22473
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/23604

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Senthil Kumaran]

Senthil Kumaran  added the comment:

Hello Maarten,

> Using cgitb, I found out that the webdriver expects the environment variable 
> `HTTP_X_URWID_METHOD` to be set. The javascript sets the "X-Urwid-Method" 
> header (using XmlHttpRequest), but these are not visible by the CGI python 
> script.

> So some scripts extra Meta-Variables neet to be set

Thanks for your comment on this old issue. The topic under discussion was about 
some existing "more standard" CGI variables than special meta variables. 

Even if the first standard CGI variables issue get exposed, I doubt the meta 
variables will get added. I will think about considering the minimal change 
required to accomplish the task and close the issue.

--
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Maarten]

Maarten  added the comment:

The CGI examples of urwid (see 
http://urwid.org/manual/displaymodules.html#cgi-web-display-module-web-display) 
don't work on http.server because of missing meta variables.

Using cgitb, I found out that the webdriver expects the environment variable 
`HTTP_X_URWID_METHOD` to be set. The javascript sets the "X-Urwid-Method" 
header (using XmlHttpRequest), but these are not visible by the CGI python 
script.

So some scripts extra Meta-Variables neet to be set.

I think section 4.1.18 applied because it is a http header that is being set. 
The sections says that these meta-variables are optional though.

I argue that having access to extra headers is useful.

--
nosy: +maarten

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41500] InterpolationSyntaxError reading the (windows) environment with unresolved environment variables

[Pauser]

Pauser  added the comment:

Thanks a lot for the fast response ... did not expect this ;)
I have looked into many threads and I will try to implement my own 
interpolation to use the current basic interpolation and process the current 
environment.

So thanks a lot, stay healthy and have a nice day!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41500] InterpolationSyntaxError reading the (windows) environment with unresolved environment variables

[Ronald Oussoren]

Ronald Oussoren  added the comment:

There's multiple options:

1. Escape '%' in environment variables (%%not_existing%%) to avoid the exception

2. Switch to configparser.ExtendedInterpolation for the interpolation, which 
uses a different syntax that doesn't conflict with env. variables on Windows 
(but does conflict with that on Unix)

Note that regardless of the option ConfigParser will not expand the value of 
the environment variable reference. If you want to be able to interpolate env 
variables you need a different solution, for example by using the 'defaults' 
argument to ConfigParser and expand references using the ConfigParser 
interpolation syntax.

--

___
Python tracker 
<https://bugs.python.org/issue41500>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41500] InterpolationSyntaxError reading the (windows) environment with unresolved environment variables

[Pauser]

Pauser  added the comment:

So I have to use two config parser objects to merge the configuration of one 
config.ini (needs interpolation) and the environment on windows ?

So at the end choose wether to use environment variables or config parser 
interpolation, right? (only on windows) 

Not the solution I was looking for.

--

___
Python tracker 
<https://bugs.python.org/issue41500>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41500] InterpolationSyntaxError reading the (windows) environment with unresolved environment variables

[Ronald Oussoren]

Ronald Oussoren  added the comment:

The exception is due to the interpolation feature of ConfigParser, which 
expects a different syntax. The default interpolation syntax is "%(key)s".

Interpolation can be turned off by passing "interpolation=None" when creating 
the ConfigParser object.

See also 
https://docs.python.org/3/library/configparser.html#interpolation-of-values

--
nosy: +ronaldoussoren
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41500] InterpolationSyntaxError reading the (windows) environment with unresolved environment variables

[Pauser]

New submission from Pauser :

Hello,
I tried to extend our config processing using the environment variables and the 
configparser module. 

On some machines (all windows) the configparser failed with an 
InterpolationSyntaxError ...
It seems that not resolved environment variables (e.g. "test":"%not_existing%") 
lead to this error.
I wrote little python test file reproducing the error and providing a very 
simple workaround to not fix all environments ;) Attention the bug is 
reproduced but the test is green ^^

I would be happy to remove the workaround as soon as possible ;)

--
components: Library (Lib)
files: test_EnvironmentProcessing.py
messages: 374991
nosy: c_panser, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: InterpolationSyntaxError reading the (windows) environment with 
unresolved environment variables
type: behavior
versions: Python 3.8
Added file: https://bugs.python.org/file49375/test_EnvironmentProcessing.py

___
Python tracker 
<https://bugs.python.org/issue41500>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Steve Dower]

Steve Dower  added the comment:

Fixed via issue39439

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Steve Dower]

Steve Dower  added the comment:


New changeset 6990d1b6131873c7f0913908162e4c723d00ea19 by Steve Dower (Adam 
Meily) in branch '3.7':
bpo-38092: Reduce overhead when using multiprocessing in a Windows virtual 
environment (GH-16098)
https://github.com/python/cpython/commit/6990d1b6131873c7f0913908162e4c723d00ea19


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39420] Windows: convertenviron() doesn't parse environment variables properly

[STINNER Victor]

STINNER Victor  added the comment:

You're right, Windows works as expected. I close the issue.

See https://github.com/python/cpython/pull/18115 for more information.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Adam Meily]

Change by Adam Meily :


--
pull_requests: +17543
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/18157

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39420] Windows: convertenviron() doesn't parse environment variables properly

[Eryk Sun]

Eryk Sun  added the comment:

> The problem is that the _wputenv() function allows to insert variable 
> names containing the "=" character 

No, it does not. Your experiments uncovered bugs elsewhere with the 
implementations of _wgetenv and WINAPI GetEnvironmentVariableW, but this isn't 
one of them. _wputenv(L"victor==secret") updates the OS environment by calling 
SetEnvironmentVariableW(L"victor", L"=secret"). That is the only correct 
result. A variable name is only allowed to contain "=" as the first character, 
where it is unambiguous. The first "=" beyond the first character must be the 
value delimiter.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39420] Windows: convertenviron() doesn't parse environment variables properly

[STINNER Victor]

New submission from STINNER Victor :

os.environ is created by convertenviron() of posixmodule.c. The Windows 
implementation calls _wgetenv(L"") to initialize _wenviron, and then parses the 
_wenviron string.

The _wenviron string is parsed by search for the first "=" character to split 
between the variable name and the variable value. For example, "USER=vstinner" 
is parsed as name="USER" and value="vstinner".

The problem is that the _wputenv() function allows to insert variable names 
containing the "=" character (but reject names starting with "=" character). 
Python can inherit an environment with a name containing "=".

One solution can be to use GetEnvironmentStringsW() which uses null characters 
to separate variable name and variable value. It returns a string like 
"name1\0value1\0name2\0value2\0\0": the string ends with a null character as 
well, to mark the end of the list.

https://docs.microsoft.com/en-us/windows/win32/api/processenv/nf-processenv-getenvironmentstrings?redirectedfrom=MSDN

Python 3.8 *explicitly* rejects variable names containing "=", at least on 
Windows, likely to workaround this issue. But another program can inject such 
variable in the environment.

Example with a Python modified to not reject explicitly "=" in the varaible 
name:
---
import subprocess, os, sys
os.putenv("victor=", "secret")
code = """import os; print(f"victor: {os.getenv('victor')!r}"); 
print(f"victor=: {os.getenv('victor=')!r}")"""
subprocess.run([sys.executable, "-c", code])
---

Output:
---
victor: '=secret'
victor=: None
---

Expected output:
---
victor: None
victor=: '=secret'
---

--
components: Library (Lib)
messages: 360473
nosy: vstinner
priority: normal
severity: normal
status: open
title: Windows: convertenviron() doesn't parse environment variables properly
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue39420>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Steve Dower]

Steve Dower  added the comment:

You're right, the logic to actually launch _base_executable is in there twice 
now, and the second one (that never gets used) is more important.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Eryk Sun]

Eryk Sun  added the comment:

This should revert to setting `_python_exe = sys.executable` in 
Lib/multiprocessing/spawn.py. Then the code in 
Lib/multiprocessing/popen_spawn_win32.py will set __PYVENV_LAUNCHER__ in the 
spawned process to the virtual environment's sys.executable. Otherwise the 
worker process has no connection to the virtual environment, other than how 
sys.path gets manually copied from the parent process. In particular, without 
setting __PYVENV_LAUNCHER__, sys.executable is not the virtual-environment 
executable and sys.prefix is not the virtual-environment directory.

Also, the fix for the parameters that are passed to _winapi.CreateProcess needs 
to be backported to 3.7. Else __PYVENV_LAUNCHER__ won't actually be set in the 
child.

--
nosy: +eryksun
resolution: fixed -> 
stage: resolved -> needs patch
status: closed -> open
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38549] Compiler build paths and related environment variables are ignored for native builds

[STINNER Victor]

STINNER Victor  added the comment:

> However in native builds (not cross-compiling) these variables/directories 
> are ignored which leads to failure of e.g. building the BZip2 extension 
> (among others).

Python does not explicitly ignore these environment variable, so I'm not sure 
how setup.py prevent that.

For example, the _bz2 module is built by setup.py by default in the master 
branch of Python.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38549] Compiler build paths and related environment variables are ignored for native builds

[STINNER Victor]

STINNER Victor  added the comment:

> This did work ages ago, but was broken when it was decided that compiler 
> paths (output of `gcc -E -v`) is no longer used for native builds and that 
> function was later even renamed: 
> https://github.com/python/cpython/commit/32f5fdd7f4213743fe2f6eedd0fe2108f3157021#diff-2eeaed663bd0d25b7e608891384b7298R514

Can you please elaborate?

setup.py of Python 2.7:

if cross_compiling:
self.add_gcc_paths()

setup.py of Python 3.5:

# only change this for cross builds for 3.3, issues on Mageia
if cross_compiling:
self.add_gcc_paths()

In the 2.7, this code was changed in 2013 to fix bpo-17990:

"For Issue17086, 8ee6d96a1019 backported some cross-build patches to the 2.7 
branch.  The changes to setup.py detect_modules differ in the backport from 
those in default. In particular, in default, changes to the library and include 
directory lists used to build extensions modules are only made conditionally 
when cross-compiling.  But the 2.7 backport makes these changes 
unconditionally."

commit 6166709eec557f52075417b19676dccc9f405fda
Author: Ned Deily 
Date:   Wed May 15 18:00:45 2013 -0700

Issue #17990: Only modify include and library search paths when 
cross-compiling.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38549] Compiler build paths and related environment variables are ignored for native builds

[Alex Grund]

New submission from Alex Grund :

In e.g. Linux users can set CPATH and LIBRARY_PATH to a list of paths 
considered by e.g. GCC, Clang, ... as if they were passed to `-I`, `-L`

These paths show up in the verbose compiler output too.

However in native builds (not cross-compiling) these variables/directories are 
ignored which leads to failure of e.g. building the BZip2 extension (among 
others).

Use case: On HPC modules are used, which means e.g. BZip2 is not installed into 
a system directory but into some other folder and CPATH, LIBRARY_PATH, 
LD_LIBRARY_PATH, ... are modified so it can be found by compiler and runtime. 
However the Python build script does not find it as it ignores those "compiler 
paths".

This did work ages ago, but was broken when it was decided that compiler paths 
(output of `gcc -E -v`) is no longer used for native builds and that function 
was later even renamed: 
https://github.com/python/cpython/commit/32f5fdd7f4213743fe2f6eedd0fe2108f3157021#diff-2eeaed663bd0d25b7e608891384b7298R514

I propose to rename the function back to its original name and use it always.

--
components: Build
messages: 355073
nosy: Alex Grund
priority: normal
severity: normal
status: open
title: Compiler build paths and related environment variables are ignored for 
native builds
type: compile error
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue38549>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[miss-islington]

miss-islington  added the comment:


New changeset 436b429ade87b10879b3f944e99a515478e86e5e by Miss Islington (bot) 
in branch '3.8':
bpo-38092: Reduce overhead when using multiprocessing in a Windows virtual 
environment (GH-16098)
https://github.com/python/cpython/commit/436b429ade87b10879b3f944e99a515478e86e5e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Steve Dower]

Steve Dower  added the comment:

Thanks for the report and partial patch!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[miss-islington]

miss-islington  added the comment:


New changeset f2b7556ef851ac85e7cbf189d1b29fdeb9539b88 by Miss Islington (bot) 
(Steve Dower) in branch 'master':
bpo-38092: Reduce overhead when using multiprocessing in a Windows virtual 
environment (GH-16098)
https://github.com/python/cpython/commit/f2b7556ef851ac85e7cbf189d1b29fdeb9539b88


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[miss-islington]

Change by miss-islington :


--
pull_requests: +15731
pull_request: https://github.com/python/cpython/pull/16116

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Steve Dower]

Steve Dower  added the comment:

I posted a second PR with the rest of the change, as it'd be good to get this 
in before the next 3.8 release.

--
versions:  -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Steve Dower]

Change by Steve Dower :


--
pull_requests: +15718
pull_request: https://github.com/python/cpython/pull/16098

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Steve Dower]

Steve Dower  added the comment:

The difference is that launching sys._base_executable *without* 
__PYVENV_LAUNCHER__ set (because env is not being passed) should lose you 
access to anything installed into the venv. You may also need to import 
something from the venv in order to see the issue.

Launching sys.executable will hit the launcher that sets the environment 
variable. Setting the environment correctly and launching sys._base_executable 
will also load correctly. The latter is theoretically required for correct 
handle sharing, but that may depend on which Windows version you're running.

I'd like to see both changes in the PR. Just setting the environment variable 
doesn't really improve the situation at all.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Pierre Glaser]

Change by Pierre Glaser :


--
keywords: +patch
pull_requests: +15524
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/15883

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Pierre Glaser]

Pierre Glaser  added the comment:

> Dropping this into Lib/multiprocessing/spawn.py should cause a repro:

  if WINSERVICE:
  _python_exe = os.path.join(sys.exec_prefix, 'python.exe')
  else:
  _python_exe = getattr(sys, '_base_executable', sys.executable)

In this case, spawn.get_executable() will return (sys._base_executable), and 
`env` will be set to None anyways no? (see these lines: 
https://github.com/python/cpython/blob/9008be303a89bfab8c3314c6a42330b5523adc8b/Lib/multiprocessing/popen_spawn_win32.py#L59-L68)

We need to trigger the if clause of these lines instead, which happens by 
default in a virtual env -- this is why it is so troubling: even though a very 
simple case (launching a new process from within a virtualenv) should trigger a 
bug, it does not.

> And maybe submit a PR with the fix?

Will do.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Steve Dower]

Steve Dower  added the comment:

Yeah, very strange that. I can only assume that it's launching the venv 
redirector directly, rather than the real sys.executable, and we aren't ever 
calling set_executable() with the real one anymore.

Dropping this into Lib/multiprocessing/spawn.py should cause a repro:

if WINSERVICE:
_python_exe = os.path.join(sys.exec_prefix, 'python.exe')
else:
_python_exe = getattr(sys, '_base_executable', sys.executable)

And as you point out, fixing the CreateProcess call should provide a fix.

Could you try that? And maybe submit a PR with the fix?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

[Pierre Glaser]

New submission from Pierre Glaser :

If I am not mistaken, when creating a new process on Python3.7 and later on 
Windows, if using a virtualenv, Python now uses a launcher. The launcher is 
being notified that it must create a virtual-environment Python (and not a 
system Python) program using the __PYVENV_LAUNCHER__ environment variable, 
passed as part of the environment of launcher process created using in 
_winapi.CreateProcess
(see 
https://github.com/python/cpython/blob/9008be303a89bfab8c3314c6a42330b5523adc8b/Lib/multiprocessing/popen_spawn_win32.py#L73-L75)

However, if I am not mistaken `env` is not passed at the right position 
(https://github.com/python/cpython/blob/9008be303a89bfab8c3314c6a42330b5523adc8b/Modules/_winapi.c#L1062-L1068).
 These lines were part of a bugfix patch (see 
https://bugs.python.org/issue35797), solving an issue for multiprocessing-based 
packages. We ended trying to backport to loky 
(https://github.com/tomMoral/loky, a multiprocessing-based package) but the bug 
was not fixed. Passing 'env' correctly fixed the bug.

Two things:
- It is hard to provide a reproducer for this issue as it requires patching the 
CPython source code.
- I don't understand why env being not passed correctly does not manifest 
itself in the test-suite. What is even more troubling is that even with this 
bug, the virtualenv launcher seems to get that a virtualenv is used when 
creating processes in multiprocessing (at least, sys.path includes the 
path/to/venv/lib/python3.x/site-packages). However, I am not familiar with the 
virtualenv launcher for python3.7+ and windows.

--
components: Windows
messages: 351650
nosy: paul.moore, pierreglaser, pitrou, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: environment variables not passed correctly using new virtualenv 
launching in windows and python3.7+
type: crash
versions: Python 3.7, Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue38092>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Glenn Linderman]

Glenn Linderman  added the comment:

That's interesting, Pierre, I hadn't really read the RFC carefully, to realize 
that many of the "missing" variables from Apache are HTTP headers, and that 
section 4.1.18 tell how to convert HTTP headers to meta variables.

The code in server.py 3.6 (Sorry, I should check the master branch) picks 
specific HTTP_ headers to include, rather than including them all per the 
rules. Doing the latter would go a long way toward being more compatible with 
Apache. I don't know if Rémi got his NGINX list from source code (looks like 
it) and if maybe NGINX also defines meta variables from the HTTP_ headers, that 
are not listed in the header file he seems to be quoting.

Unless the code has already been improved for Python 3.7, I think there is 
still some work to do to make server.py conform even to the RFC, if not be 
compatible with Apache.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Pierre Quentel]

Pierre Quentel  added the comment:

The QUERY_STRING value is always set by the code at lines 1135-1137 of 
http.server:

for k in ('QUERY_STRING', 'REMOTE_HOST', 'CONTENT_LENGTH',
  'HTTP_USER_AGENT', 'HTTP_COOKIE', 'HTTP_REFERER'):
env.setdefault(k, "")

The RFC for CGI has not evolved since 2004, probably because the technology is 
stable, and also because other, more efficient protocols have been defined to 
avoid the "CGI overhead" (FastCGI for instance).

I think that http.server should only implement the "meta-variables" defined in 
RFC 3875:
- AUTH_TYPE  CONTENT_LENGTH CONTENT_TYPE  GATEWAY_INTERFACE PATH_INFO  
PATH_TRANSLATED QUERY_STRING  REMOTE_ADDR REMOTE_HOST  REMOTE_IDENT REMOTE_USER 
 REQUEST_METHOD SCRIPT_NAME  SERVER_NAME SERVER_PORT  SERVER_PROTOCOL 
SERVER_SOFTWARE. Some of these must always be set (eg QUERY_STRING, 
REQUEST_METHOD, SERVER_NAME...) but for other ones, there are conditions (for 
instance for CONTENT_LENGTH: "The server MUST set this meta-variable if and 
only if the request is accompanied by a message-body entity")
- "protocol-specific meta variables" : for HTTP, variables determined by the 
HTTP request headers such as HTTP_COOKIE (cf section 4.1.18.  Protocol-Specific 
Meta-Variables)

Other meta variables are probably beyond the scope of a module in the standard 
library.

In short, in my opinion the issue can be closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

Hi Glenn, I'm not aware of a document that defines CGI better than the RFC and 
I don't know it enough to disgress from the published standard (even if it is 
not what isdone today as I don't know the current practices enough).

Here is the variables defined by nginx 1.10.3 on Debian:

fastcgi_param  QUERY_STRING   $query_string;
fastcgi_param  REQUEST_METHOD $request_method;
fastcgi_param  CONTENT_TYPE   $content_type;
fastcgi_param  CONTENT_LENGTH $content_length;

fastcgi_param  SCRIPT_NAME$fastcgi_script_name;
fastcgi_param  REQUEST_URI$request_uri;
fastcgi_param  DOCUMENT_URI   $document_uri;
fastcgi_param  DOCUMENT_ROOT  $document_root;
fastcgi_param  SERVER_PROTOCOL$server_protocol;
fastcgi_param  REQUEST_SCHEME $scheme;
fastcgi_param  HTTPS  $https if_not_empty;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWAREnginx/$nginx_version;

fastcgi_param  REMOTE_ADDR$remote_addr;
fastcgi_param  REMOTE_PORT$remote_port;
fastcgi_param  SERVER_ADDR$server_addr;
fastcgi_param  SERVER_PORT$server_port;
fastcgi_param  SERVER_NAME$server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS200;

Someone that knows CGI better than me may know the way forward

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Glenn Linderman]

Glenn Linderman  added the comment:

Rémi Lapeyre, glad to see your interest here, as this is an old and languishing 
bug.

I would have hoped based on my input, that had there been anyone that was 
maintaining the Python web server code, that they might have done a more 
complete analysis than I did.

I note the document you reference is from 2004 (and I referenced it too), and 
doesn't include mention of the HTTP_COOKIE header, yet that header is 
frequently used in practical web applications. Apache supports it (as noted). 
My point is that it is not clear that conforming to the RFC 3875 from 2004 is 
really sufficient to build a useful web server. While it is true that my 
references to Apache are to a particular implementation, it is a widespread 
implementation, which other implementations attempt to be compatible with, 
indicating that being reasonably compatible with Apache would seem to be a good 
thing for other web server implementations. A few more environment variables 
don't cost a lot, and seem to be useful. I don't know if some or all of the 
additional environment variables implemented by Apache are standardized by RFC 
or other standards, or whether they are common practice, or unique to Apache. 
Nor where such standards might be fonud, but I would hope a maintainer of the 
Python web server would be interested in sorting out such environment variables 
and making that determination, rather than relying on a 14 year old RFC as the 
definitive source, when web technologies have progressed significantly in the 
last 14 years. I would agree that variables that are unique to Apache might not 
want to be implemented, but on the other hand, with other implementations 
following Apache's lead, there may be few that are unique to Apache.

--

___
Python tracker 
<https://bugs.python.org/issue10486>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

AUTH_TYPE, CONTENT_LENGTH, CONTENT_TYPE, REMOTE_USER are present

REMOTE_IDENT is not but I'm not sure it's worth adding.

I can send a PR to add REMOTE_HOST and remove the condition for QUERY_STRING.

Otherwise, I don't think the other environment variables should be added, they 
are implementation dependant and not defined in RFC 3875.

Should we close this issue?

--

___
Python tracker 
<https://bugs.python.org/issue10486>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

The reference given in 
https://github.com/python/cpython/blob/b36b0a3765bcacb4dcdbf12060e9e99711855da8/Lib/http/server.py#L1074
 is not accessible anymore.

I think we should replace it by https://tools.ietf.org/html/rfc3875#section-4.1

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Pierre Quentel]

Change by Pierre Quentel :


--
nosy: +quentel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Rémi Lapeyre]

Change by Rémi Lapeyre :


--
nosy: +remi.lapeyre

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34317] Improve docstring Environment variables in Windows NT

[miss-islington]

miss-islington  added the comment:


New changeset 34f59a7a7843d8e2c1922c830a99b5b7a022f4be by Miss Islington (bot) 
in branch '3.7':
bpo-34317: Fix a dead url to Windows documentation (GH-8622)
https://github.com/python/cpython/commit/34f59a7a7843d8e2c1922c830a99b5b7a022f4be


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34317] Improve docstring Environment variables in Windows NT

[Mariatta Wijaya]

Mariatta Wijaya  added the comment:

Thanks!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34317] Improve docstring Environment variables in Windows NT

[miss-islington]

Change by miss-islington :


--
pull_requests: +8144

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34317] Improve docstring Environment variables in Windows NT

[Mariatta Wijaya]

Mariatta Wijaya  added the comment:


New changeset 46ebe61c7f3511b97268b44d5373a9e9cf0b5cc7 by Mariatta 
(HiyashiChuka) in branch 'master':
bpo-34317: Fix a dead url to Windows documentation (GH-8622)
https://github.com/python/cpython/commit/46ebe61c7f3511b97268b44d5373a9e9cf0b5cc7


--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34317] Improve docstring Environment variables in Windows NT

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch
pull_requests: +8127
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34317] Improve docstring Environment variables in Windows NT

[sakamoto aya]

New submission from sakamoto aya :

Expired link in windows.rst:  
https://support.microsoft.com/en-us/help/100843/environment-variables-in-windows-nt

May I make a suggestion?

The new link:https://www.microsoft.com/en-us/wdsi/help/folder-variables

--
assignee: docs@python
components: Documentation
messages: 322944
nosy: HiyashiChuka, docs@python
priority: normal
severity: normal
status: open
title: Improve docstring   Environment variables in Windows NT
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue34317>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33019] Review usage of environment variables in the stdlib

[Benjamin Peterson]

Benjamin Peterson  added the comment:

It's useful if you want to hide the fact that a command is implemented in 
Python and don't want it to malfunction if the user has PYTHONPATH set for some 
reason.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33019] Review usage of environment variables in the stdlib

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I may be mistaken, what's the use of -E if not for security?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33019] Review usage of environment variables in the stdlib

[Benjamin Peterson]

Benjamin Peterson  added the comment:

I don't think -E is a security feature. Even if the stdlib was fixed, there's 
tons of 3rdparty Python code that consumes os.environ.

It seems like if you really cared about not letting the environment influence a 
Python application, you'd just wrap python in a script that cleans out the 
environment before execing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33019] Review usage of environment variables in the stdlib

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33019] Review usage of environment variables in the stdlib

[Christian Heimes]

Christian Heimes  added the comment:

External libraries like sqlite may also use env vars. I know for sure OpenSSL 
uses SSL_CERT_FILE and SSL_CERT_DIR to override default verify locations.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33019] Review usage of environment variables in the stdlib

[Antoine Pitrou]

New submission from Antoine Pitrou <pit...@free.fr>:

Python supports a mode where the interpreter ignores environment variables such 
as PYTHONPATH, etc.

However, there are places in the stdlib where environment-sensitive decisions 
are made, without regard for the ignore-environment flag.

Examples include:
- ssl.get_default_verify_paths() queries SSL_CERT_FILE and SSL_CERT_DIR
- shutil.which() queries PATH
- the tempfile module queries TMPDIR, TEMP, TMP to select the defaut directory 
for temporary files

Do you think those need to be sanitized?

--
components: Library (Lib)
messages: 313393
nosy: alex, christian.heimes, pitrou
priority: normal
severity: normal
status: open
title: Review usage of environment variables in the stdlib
type: security
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33019>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Meaning of "Add Python to environment variables"

[Thomas Jollans]

On 03/12/17 18:32, John Yeung wrote:
> In the custom installation options for Python 3.6, what *exactly* does "Add
Python to environment variables" mean?
>
> Which environment variables are we talking about? I imagine one of them would
have to be PATH. Are there any others?


If the note on the "PrependPath" option in
https://docs.python.org/3/using/windows.html#installing-without-ui is complete
(which I believe it is, but I couldn't swear to it), it's PATH and PATHEXT.

-- Thomas

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Meaning of "Add Python to environment variables"

[Thomas Jollans]

On 03/12/17 18:32, John Yeung wrote:
> In the custom installation options for Python 3.6, what *exactly* does "Add 
> Python to environment variables" mean?
> 
> Which environment variables are we talking about? I imagine one of them would 
> have to be PATH. Are there any others?


If the note on the "PrependPath" option in
https://docs.python.org/3/using/windows.html#installing-without-ui is
complete (which I believe it is, but I couldn't swear to it), it's PATH
and PATHEXT.

-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Meaning of "Add Python to environment variables"

[John Yeung]

In the custom installation options for Python 3.6, what *exactly* does "Add 
Python to environment variables" mean?

Which environment variables are we talking about? I imagine one of them would 
have to be PATH. Are there any others?

John Y.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue26876] Extend MSVCCompiler class to respect environment variables

[Steve Dower]

Steve Dower added the comment:

Taking another look at the patch, I'm not real keen on the C/LDFLAGS section. I 
think if we want to support setting these, we should fully override the default 
settings (otherwise you can't specify certain options that are in the 
defaults), and avoid splitting them (which probably means changing the spawn() 
call to not quote arguments - right now the .split() logic is just wrong).

I'm sorry this is tough to get together - the distutils compilers are really 
fragile, which is why we would really rather migrate people off them onto some 
other build framework. Right now though, that framework doesn't exist. (One 
potential starting point is my msbuildcompiler at 
https://github.com/zooba/pyfindvs/tree/master/pyfindvs/msbuildcompiler will 
generate a .vcxproj file and build it, which allows much more reliable 
behaviour for options and detecting installs.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

Rohit Jamuar added the comment:

Steve, I've limited the changes to _msvccompiler for the master-branch's patch. 
Is it alright?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

Changes by Rohit Jamuar :


Added file: http://bugs.python.org/file45881/distutils_patch_master.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

Changes by Rohit Jamuar :


Removed file: http://bugs.python.org/file45880/distutils_patch_master.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

Changes by Rohit Jamuar :


Added file: http://bugs.python.org/file45880/distutils_patch_master.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

Changes by Rohit Jamuar :


Removed file: http://bugs.python.org/file42725/distutils_patch_master.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Steve Dower]

Steve Dower added the comment:

I'd rather only take the change to _msvccompiler and not the ones that are 
there for historical interest (a.k.a. backwards compatibility with people who 
never expect internal implementation details to change).

If any tools break because they're using the wrong compiler class, please file 
a bug against them.

I didn't check the 2.7 file, but the one for master is fine.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Zachary Ware]

Zachary Ware added the comment:

Steve, does the latest patch look good to you?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[STINNER Victor]

Changes by STINNER Victor :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

Rohit Jamuar added the comment:

Bump! The changes, that Steve was asking for, have been incorporated. If 
something was missed, please inform. Otherwise, could this be merged?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> More warnings.

Excellent result!

> my flush=True changes may be removed if my latest change fixed the issue

Don't bother. The buffering of standard streams was changed in 3.x and I'm not 
sure it always work as expected. For example the meaning of the -u option seems 
was changed. Maybe my beliefs about this are wrong.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[STINNER Victor]

STINNER Victor added the comment:

More warnings.

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/5058/steps/test/logs/stdio

Warning -- threading._dangling was modified by test_logging
Warning -- files was modified by test_io

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/5148/steps/test/logs/stdio

Warning -- threading._dangling was modified by test_logging
Warning -- threading._dangling was modified by test_asyncio
Warning -- files was modified by test_io

http://buildbot.python.org/all/builders/AMD64%20Windows10%203.x/builds/1573/steps/test/logs/stdio

Warning -- files was modified by test_distutils
Warning -- threading._dangling was modified by test_asyncio

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[STINNER Victor]

STINNER Victor added the comment:

Cool, my change fixed this issue. Examples:

http://buildbot.python.org/all/builders/s390x%20RHEL%203.x/builds/1953/steps/test/logs/stdio
Warning -- threading._dangling was modified by test_logging

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/8590/steps/test/logs/stdio
Warning -- files was modified by test_distutils3

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[STINNER Victor]

STINNER Victor added the comment:

> I tried random changes since I was unable to reproduce the issue and didn't 
> understand why the the warning was not printed.

Ooops, I forgot to say: my flush=True changes may be removed if my latest 
change fixed the issue (if the warning is now displayed).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[STINNER Victor]

STINNER Victor added the comment:

> I thought sys.stderr is line buffered and doesn't need flush=True if prints a 
> complete line.

I tried random changes since I was unable to reproduce the issue and didn't 
understand why the the warning was not printed.

I'm not sure that stderr is always line buffered. Is it the case even if stderr 
is redirected to a pipe?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I thought sys.stderr is line buffered and doesn't need flush=True if prints a 
complete line.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[STINNER Victor]

STINNER Victor added the comment:

I was able to reproduce the missing warning by modifying test_sys.py to modify 
sys.path and then run:
./python -m test -W test_sys

The -W eats stderr even if the environment changed. I modified regrtest to copy 
stderr if the test doesn't pass which includes: fail, interrupted, env changed, 
etc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[Roundup Robot]

Roundup Robot added the comment:

New changeset cbe049647a04 by Victor Stinner in branch '3.5':
Issue #27829: regrtest -W displays stderr if env changed
https://hg.python.org/cpython/rev/cbe049647a04

New changeset 12e3023d19bd by Victor Stinner in branch '3.6':
Merge 3.5 (issue #27829)
https://hg.python.org/cpython/rev/12e3023d19bd

New changeset 8210873bd684 by Victor Stinner in branch 'default':
Merge 3.6 (issue #27829)
https://hg.python.org/cpython/rev/8210873bd684

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[Roundup Robot]

Roundup Robot added the comment:

New changeset 491bbba73bca by Victor Stinner in branch 'default':
Show regrtest env changed warn on Windows buildbot
https://hg.python.org/cpython/rev/491bbba73bca

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2e23f7b9515c by Victor Stinner in branch 'default':
Issue #27829: libregrtest.save_env: flush stderr
https://hg.python.org/cpython/rev/2e23f7b9515c

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[STINNER Victor]

New submission from STINNER Victor:

It's common to get a random "env failed" issue when tests are run by buildbots, 
but changed environment variables are not logged, so the "env changed 
"information cannot be used to fix the issue :-/

Example:
--
...
0:07:32 [160/402] test_io failed (env changed) (51 sec)
0:07:33 [161/402] test_threading_local passed
...
0:13:04 [289/402/2] test_ssl failed (env changed) -- running: 
test_multiprocessing_spawn (119 sec)
0:13:05 [290/402/2] test_wsgiref passed -- running: test_multiprocessing_spawn 
(120 sec)
...
2 tests altered the execution environment:
test_io test_ssl
--
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%203.x/builds/932/steps/test/logs/stdio

I expected "Warning -- xxx was modfied by test_xxx" lines, but no, there is no 
such line. I don't know if it's related to stderr buffering or something else?

This buildbot runs tests in subprocesses.

--
components: Tests
messages: 273364
nosy: haypo
priority: normal
severity: normal
status: open
title: test.regrtest: changed environment variables are not logged
versions: Python 3.6

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue27829>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27829] test.regrtest: changed environment variables are not logged

[STINNER Victor]

Changes by STINNER Victor :


--
keywords: +buildbot

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Steve Dower]

Steve Dower added the comment:

I still want the behavior I described, since there's no value in overriding 
just the executable name but not the rest of the path.

For 2.7 I think this'll help with long term maintainability enough to be the 
Right Thing. For 3.5 I'm not as sure, but it's probably worth keeping the 
implementations consistent as long as we can justify it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Zachary Ware]

Zachary Ware added the comment:

The patch looks fine to me, but frankly I'm a bit scared to commit anything to 
distutils :).  It would be nice to have tests for this.

I'm also not sure if we can backport this to 2.7 and 3.5: this is pretty 
clearly a new feature, but dips its toes into the grey area of "supporting new 
platforms".  Anyone else have any thoughts?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Zachary Ware]

Zachary Ware added the comment:

(The version field is for the whole issue, not per patch.)

--
versions: +Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

Changes by Rohit Jamuar :


--
versions: +Python 2.7 -Python 3.5, Python 3.6
Added file: http://bugs.python.org/file42726/distutils_patch_27.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

Changes by Rohit Jamuar :


--
versions: +Python 3.6 -Python 2.7
Added file: http://bugs.python.org/file42725/distutils_patch_master.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

Changes by Rohit Jamuar :


Removed file: http://bugs.python.org/file42642/msvc_respect_env_flags.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Steve Dower]

Steve Dower added the comment:

I was thinking more like this:

if DISTUTILS_USE_SDK:
assume PATH is configured
execute os.getenv("CC", "cl.exe") directly
else:
find VS in the registry
execute "cl.exe" etc.

The only change here from the current situation is preferring the CC variable 
in the former case when it is set. When we look up VS in the registry, we know 
what executable we are going to find and overriding them via the environment is 
only going to cause issues/vulnerabilities.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

Rohit Jamuar added the comment:

Just so that I understand it clearly - Inside MSVCCompiler class (in 
_msvccompiler.py / msvccompiler.py / msvc9compiler.py ), current implementation 
of find_exe() finds compiler / linker / ... after parsing PATH. Should the 
changes be so, that if DISTUTILS_USE_SDK is set to the environment, the values 
set to CC / AR / LD, etc. are used verbatim? Or did you mean to say, that just 
as CC, LD and AR are being read from the environment, the same way rc.exe, 
mc.exe and mt.exe should be read as well, in case DISTUTILS_USE_SDK is set to 
the environment?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Steve Dower]

Steve Dower added the comment:

I'm neutral about applying it to 2.7, though I can see the value in doing so.

For 3.5 it definitely has to modify _msvccompiler - the other two modules are 
just left behind to distract people apparently :) (but actually in case someone 
accidentally(?) too a direct dependency on an undocumented class).

One thing I'd like to see is an environment variable to switch it on - probably 
DISTUTILS_USE_SDK is fine for this. In the default case, we know exactly where 
we are looking for the tools, and so we know what they are called. If you're 
intending to override this completely, you probably need to override 
everything, and DISTUTILS_USE_SDK is the flag for this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Zachary Ware]

Zachary Ware added the comment:

I can't seem to get the patch to apply, could you please regenerate it against 
a fresh checkout of https://hg.python.org/cpython#default (or 'master' of 
github.com/python/cpython)?

This looks fairly straightforward, but I'm far from an expert on distutils :).  
I can see issues arising from attempts to use a compiler/linker/etc. that 
doesn't recognize options that we assume are going to cl/link/etc., but I'm not 
sure how big a deal that is: I'd be inclined to say that if you're trying to 
tell distutils where cl is, don't get mad when what you told it doesn't work.

Also, I don't think Lib/distutils/msvccompiler.py is used anymore, but rather 
Lib/distutils/_msvccompiler.py in 3.5+.

I'm generally in favor of this, but would feel much better about it with buy-in 
from Steve.

There has also been mention of creating a new ICCCompiler class, but I'm not 
sure whether that has gotten anywhere.

--
components: +Windows
nosy: +paul.moore, steve.dower, tim.golden
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Zachary Ware]

Changes by Zachary Ware :


--
nosy: +christopher.hogan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26876] Extend MSVCCompiler class to respect environment variables

[Rohit Jamuar]

New submission from Rohit Jamuar:

The UnixCompiler class respects flags (CC, LD, AR, CFLAGS and LDFLAGS) set to 
the environment, whereas MSVCCompiler class does not. This change allows 
building CPython and any module that invokes distutils to utilize flags and 
executables that have been set to the environment. Inclusion of this change 
would ensure MSVCCompiler's behavior to be same as that of UnixCompiler and 
would also allow using a different set of compiler / linker / archiver, on 
Windows, without having the necessity for implementing separate compiler 
classes - using environment variables it should be possible to use a separate 
set of build executables - for example icl, clang, etc.

--
components: Distutils
files: msvc_respect_env_flags.patch
keywords: patch
messages: 264439
nosy: dstufft, eric.araujo, r.david.murray, rohitjamuar, zach.ware
priority: normal
severity: normal
status: open
title: Extend MSVCCompiler class to respect environment variables
type: enhancement
versions: Python 2.7, Python 3.5
Added file: http://bugs.python.org/file42642/msvc_respect_env_flags.patch

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26876>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Settig some environment variables from within python.

[Terry Reedy]

On 3/23/2016 5:33 AM, Hongyi Zhao wrote:

Hi all,

I want to submit a qsub job to my hpc cluster from within python.  In
this case, I must set some environment variables specific for this qsub
job and then invoking a bash script from within python.

What python code should be used for this job?


The os module.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Settig some environment variables from within python.

[Hongyi Zhao]

Hi all,

I want to submit a qsub job to my hpc cluster from within python.  In 
this case, I must set some environment variables specific for this qsub 
job and then invoking a bash script from within python.

What python code should be used for this job?

Regards
-- 
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
-- 
https://mail.python.org/mailman/listinfo/python-list