[issue28824] os.environ should preserve the case of the OS keys ?

[Eryk Sun]

Change by Eryk Sun :


--
Removed message: https://bugs.python.org/msg414332

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[Eryk Sun]

Eryk Sun  added the comment:

Putting words into action, here's an example of what a privileged process (e.g. 
running as SYSTEM) can do if a script or application is written to call the 
undocumented NT API function NtMakePermanentObject(). A use case would be a 
script running as a system service that needs a shared-memory section object to 
persist after the service is stopped or terminated, e.g. such that the section 
is still accessible if the service is resumed. Anyway, this is just an example 
of what's possible, from a technical perspective.

By default, named kernel objects are temporary:

>>> import os, _winapi, ctypes
>>> from multiprocessing.shared_memory import SharedMemory
>>> name = f'spam_{os.getpid()}'
>>> m = SharedMemory(name, True, 8192)
>>> m.close()
>>> try: SharedMemory(name)
... except OSError as e: print(e)
...
[WinError 2] The system cannot find the file specified: 'spam_6592'

Global permanent example:

Enable privileges for the current thread:

>>> import win32api; from win32security import *
>>> ImpersonateSelf(SecurityImpersonation)
>>> da = TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES
>>> ht = OpenThreadToken(win32api.GetCurrentThread(), da, False)
>>> ps = [[0, SE_PRIVILEGE_ENABLED], [0, SE_PRIVILEGE_ENABLED]]
>>> ps[0][0] = LookupPrivilegeValue(None, 'SeCreatePermanentPrivilege')
>>> ps[1][0] = LookupPrivilegeValue(None, 'SeCreateGlobalPrivilege')
>>> AdjustTokenPrivileges(ht, False, ps)
((16, 0),)

Create a global section object, and make it permanent:

>>> name = rf'Global\spam_{os.getpid()}'
>>> m = SharedMemory(name, True, 8192)
>>> from win32con import DELETE
>>> h = _winapi.OpenFileMapping(DELETE, False, name)
>>> ntdll = ctypes.WinDLL('ntdll')
>>> ntdll.NtMakePermanentObject(h)
0
>>> _winapi.CloseHandle(h)

A permanent object persists after the last handle is closed:

>>> m.close()
>>> m = SharedMemory(name) # This works now.

Make the section object temporary again:

>>> h = _winapi.OpenFileMapping(DELETE, False, name)
>>> ntdll.NtMakeTemporaryObject(h)
0
>>> _winapi.CloseHandle(h)
>>> m.close()
>>> try: SharedMemory(name)
... except OSError as e: print(e)
...
[WinError 2] The system cannot find the file specified: 'Global\\spam_6592'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[Eryk Sun]

Eryk Sun  added the comment:

> I think there should be a public class like this.

I wrote a basic implementation of _CaseInsensitiveString under the assumption 
that it's hidden behind the __getitem__(), __setitem__(), and __delitem__() 
methods of the _Environ class. I don't want to complicate the implementation.

The problem of case-insensitive names in file/registry paths and environment 
variables should be addressed with ntpath.normcase(). This should be based on 
WinAPI LCMapStringEx() with LOCALE_NAME_INVARIANT and LCMAP_UPPERCASE. The 
current implementation of ntpath.normcase() uses str.lower(), which depends on 
Python's Unicode database. This possibly differs from what Windows considers to 
be lower case in the invariant locale. It's also wrong because Windows uses 
upper case for cases-insensitive comparisons, which matters when sorting names. 
See bpo-42658.

> Right now, it's not a good workaround because it contains the 
> environment at the time the interpreter was started, not the 
> current environment.

Note that in some cases the "current environment" is the process environment, 
which isn't necessarily consistent with os.environ on any platform.

In POSIX, posix.environ is created from C environ, which is kept in sync with 
changes to posix.environ via C putenv() and unsetenv(). However, directly 
calling os.putenv() or os.unsetenv(), or the underlying C functions, modifies 
the process environment without changing posix.environ. If subprocess.Popen() 
is called without overriding env, the child inherits the process environment, 
not posix.environ.

In Windows, os.environ is created from nt.environ, with variables names 
converted to upper case. nt.environ is created from C _wenviron, which is 
created from the process environment, as returned by WinAPI 
GetEnvironmentStringsW(), except with variable names that begin with "=" 
filtered out. os.putenv() and os.unsetenv() are based on C _wputenv(), which 
updates C _wenviron and also updates the process environment via WinAPI 
SetEnvironmentVariableW(). If either os.putenv() or os.unsetenv() is called 
directly, or _wputenv() at a lower level, then the variables in C _wenviron 
(not just the letter case of the names) will be out of sync with os.environ. 
Additionally, if SetEnvironmentVariableW() is called directly to set or unset a 
variable, then both os.environ and C _wenviron will be out of sync with the 
process environment. If subprocess.Popen() is called without overriding env, 
the child inherits the process environment, not os.environ or C _wenviron.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[benrg]

benrg  added the comment:

This issue should be marked dependent on issue 43702 or issue 46862, since 
fixing it could break third-party code unless they're fixed first.


> Given 'nt.environ' is available without case remapping, I think that's the 
> best workaround.

Right now, it's not a good workaround because it contains the environment at 
the time the interpreter was started, not the current environment. On Posix, 
_Environ takes a reference to posix.environ and uses it directly, so it does 
get updated. On Windows, _Environ gets a rewritten dictionary and nt.environ is 
just a space-wasting attractive nuisance. I think it should be replaced with 
getenviron() which builds a dict from the environment block each time it's 
called. But posix.environ is documented (though nt.environ isn't), so maybe not.


> class _CaseInsensitiveString(str):

I think there should be a public class like this. It could be useful to 
email.message.Message and its clients like urllib. They currently store headers 
in a list and every operation is O(n).

The semantics are tricky. As written, it violates the requirement that equal 
objects have equal hashes. To fix that, you'd have to make every CIS compare 
unequal to every str. At that point, it probably shouldn't be a str subclass, 
which also has the advantage that it's not limited to strings. It can be a 
generic compare-by-key wrapper.

--
nosy: +benrg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[Larry Hastings]

Change by Larry Hastings :


--
nosy:  -larry, loewis

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[Eryk Sun]

Change by Eryk Sun :


--
versions: +Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[Eryk Sun]

Eryk Sun  added the comment:

In Windows, maybe the os.environ mapping could use a case-insensitive subclass 
of str for its keys, such as the following:

@total_ordering
class _CaseInsensitiveString(str):
def __eq__(self, other):
if not isinstance(other, str):
return NotImplemented
return self.upper() == other.upper()

def __lt__(self, other):
if not isinstance(other, str):
return NotImplemented
return self.upper() < other.upper()

def __hash__(self):
return hash(self.upper())

Change encodekey() to use this type. For example:

def encodekey(key):
return _CaseInsensitiveString(encode(key))

in which encode() is still check_str().

--
components: +Library (Lib)
type:  -> behavior
versions: +Python 3.10, Python 3.9 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[Steve Dower]

Steve Dower added the comment:

Ah, I see what you mean. In this case, we could change how the 
case-insensitivity is handled here, but it would only be applicable to 3.7. I'm 
not opposed to changing the default behavior here, but it does kind of bring up 
the mapping dict discussion again.

If case is important to your application, environment variables are probably 
the wrong way to go about passing them in anyway. Either use the value of the 
variable rather than the key, or find a different approach. Given 'nt.environ' 
is available without case remapping, I think that's the best workaround.

--
versions: +Python 3.7 -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[tzickel]

tzickel added the comment:

Steve, I've checked in Python 3.5.2, and os.environ.keys() still uppercases 
everything when scanning (for my use case). Has it changed since then ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[Eryk Sun]

Eryk Sun added the comment:

I've come across a few problems when passing a modified os.environ.copy() to a 
child process via subprocess.Popen. Ideally it shouldn't be an issue, as long 
as the OS or C runtime functions are used, but some troublesome programs do 
their own case-sensitive search for environment variables (definitely a bug). 
In such cases the simplest workaround is to use nt.environ.copy(), but this 
doesn't include any changes in the environment since startup.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[tzickel]

tzickel added the comment:

My issue is that somebody wants to pass a few dict like environment variables 
as some prefix_key=value but he wants to preserve the case of the key for usage 
in python so the .keys() space needs to be enumerated.

A workaround for this issue can be importing nt and using nt.environ which 
preserves the cases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[Steve Dower]

Steve Dower added the comment:

This works fine in Python 3, and also Python 2.7 *unless* you call .keys().

PS D:\> py -2.7
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) [MSC v.1500 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> 'path' in os.environ, 'path' in os.environ.keys()
(True, False)

PS D:\> py
Python 3.6.0b4 (default, Nov 22 2016, 05:30:12) [MSC v.1900 64 bit (AMD64)] on 
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> 'path' in os.environ, 'path' in os.environ.keys()
(True, True)

I suspect at this point, we aren't going to change this in Python 2.7 unless 
someone comes up with an incredibly motivating issue.

--
versions:  -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[R. David Murray]

R. David Murray added the comment:

That unfortunately would probably break existing code.  It does seem reasonable 
that case should be ignored on get, though, if the OS does so.  So making your 
'in' statement work might be acceptable, backward compatibility wise.  Probably 
only in 3.7, though.

Is it the OS that ignores case, or just the shell?

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28824] os.environ should preserve the case of the OS keys ?

[tzickel]

New submission from tzickel:

In Windows, python's os.environ currently handles the case sensitivity 
different that the OS. While it's true that the OS is case insensitive, it does 
preserve the case that you first set it as.

For example:
C:\Users\user>set aSD=Blah
C:\Users\user>set asd
aSD=Blah

But in python:
>>> import os
>>> 'aSD' in os.environ.keys()
False

Today as more people pass environment variables to processes, it's better to 
behave as the OS does. Basically I think that os.environ (both in 2.7 and 3) 
should preserve the case as well (for when you need to access / iterate over 
the keys or set a key), but ignore it when you get a key.

https://github.com/python/cpython/blob/b82a5a65caa5b0f0efccaf2bbea94f1eba19a54d/Lib/os.py#L733

--
components: Windows
messages: 281906
nosy: larry, loewis, paul.moore, steve.dower, tim.golden, tzickel, zach.ware
priority: normal
severity: normal
status: open
title: os.environ should preserve the case of the OS keys ?
versions: Python 2.7, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com