[Python-ideas] Re: os.workdir() context manager

[Marc-Andre Lemburg]

Just as an update to this thread:

The workdir context manager got implemented for Python 3.11
and now lives in contextlib as contextlib.chdir():

https://github.com/python/cpython/commit/3592980f9122ab0d9ed93711347742d110b749c2

Thanks to everyone who contributed to the thread, the SC for
approving it, Filipe Laíns who implemented it and others who
are still helping to sort out some smaller issues:

https://bugs.python.org/issue45545

Cheers,
-- 
Marc-Andre Lemburg


On 15.09.2021 21:41, Marc-Andre Lemburg wrote:
> On 15.09.2021 21:02, Guido van Rossum wrote:
>> To make chdir() return a context manager *and* keep it working without 
>> calling
>> `__enter__`, it would have to call `getcwd()`, which I've heard is expensive.
>>
>> So I don't think that would work, alas.
> 
> At least on Linux, the cost for os.getcwd() is similar to the cost
> of os.chdir(), but yes, since we can't have os.chdir() not change
> the dir when called, the logic would need the extra os.getcwd() call:
> 
> # python3 -m timeit -s 'import os' "os.getcwd()"
> 50 loops, best of 5: 619 nsec per loop
> # python3 -m timeit -s 'import os' "os.chdir('.')"
> 50 loops, best of 5: 726 nsec per loop
> 
> Here's simple implementation of the chdir() context manager:
> 
> import os
> import pathlib
> 
> # chdir context manager
> PlatformPath = pathlib.WindowsPath if os.name == 'nt' else pathlib.PosixPath
> class chdir(PlatformPath):
> def __init__(self, dir):
> self.dir = dir
> self.olddir = os.getcwd()
> os.chdir(dir)
> def __enter__(self):
> return self
> def __exit__(self, *exc):
> os.chdir(self.olddir)
> return False
> 
> # Normal chdir()
> path = chdir('abc/')
> print (os.getcwd())
> print (path.olddir)
> 
> # chdir() context manager
> with chdir('def/') as wd:
> print (repr(wd))
> print (os.getcwd())
> print (os.listdir('.'))
> 
> For extra perks, I made os.chdir() return a pathlib Path object
> and you get to see the old directory, so you can backtrack
> if needed, even without a context manager.
> 
> 
>> On Wed, Sep 15, 2021 at 11:55 AM Eric V. Smith > > wrote:
>>
>> On 9/15/2021 2:48 PM, Eric Fahlgren wrote:
>>> On Wed, Sep 15, 2021 at 12:21 AM Eric V. Smith >> > wrote:
>>>
>>> And I'm not crazy about the name "workdir". To me, it sounds like it
>>> returns something, not sets and resets something. But I don't have a
>>> particularly great alternative in mind: in my own code I've used
>>> "change_dir", which isn't awesome either.
>>>
>>>
>>> Our version is named "pushdir", modeled after shell's pushd (even though
>>> pushdir is a context manager and does auto-pop).  Everyone figures out
>>> what it does at a glance.
>>
>> That's a great name!
>>
>> Although I think having os.chdir() return a context manager is a better
>> design (assuming it can work, but at first blush it would seem so).
>>
>> Eric

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 22 2021)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JHN5UOA3DNPI2RLWVKC2BLW5RWCOWZD4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Jeremiah Paige]

On Wed, Sep 15, 2021 at 4:20 AM Steven D'Aprano  wrote:

> On Wed, Sep 15, 2021 at 08:53:21PM +1000, Chris Angelico wrote:
>
> > Sometimes there is no valid path to something,
>
> I don't understand what you mean by that. If there is no valid path to
> something, where exactly are you cd'ing to, and how does it help you
> with the thing that has no valid path to it?
>

No one has mentioned the other case where cd'ing somewhere makes paths
suddenly valid, and that is for long path names. Once you pass PATH_MAX
length your only options are to cd as deep as possible and continue
relatively,
or to pass dir fds. The second is perhaps the better option, but harder to
implement as you have to handle extra fds and remember to clean them up and
I at least never remember how to thread fds through python code
without going
and looking it up.

This also brings up the issue of how to restore the prior working
directory. Just
remembering its path and calling a new chdir will fail, both in the long
path case
and in the case that the old dir was deleted since last visit. We could
hold an
open fd before leaving the old cwd, and that should allow us to return in
both
cases. However I don't know how others feel about holding on to dir fds,
especially
essentially hidden ones; it sounds ripe for FS deadlock.

Regards,
~Jeremiah
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/L37QIXGPPBSQJFOHI5O3QZCKDEHKDVTJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Steven D'Aprano]

On Wed, Sep 15, 2021 at 12:02:04PM -0700, Guido van Rossum wrote:

> To make chdir() return a context manager *and* keep it working without
> calling `__enter__`, it would have to call `getcwd()`, which I've heard is
> expensive.
> 
> So I don't think that would work, alas.

How expensive would it have to be to rule it out?

Calling os.getcwd() is about 80% as costly as chdir on my machine so it 
is clearly not free.

[steve ~]$ python3.9 -m timeit -s "import os" "os.getcwd()"
50 loops, best of 5: 687 nsec per loop

[steve ~]$ python3.9 -m timeit -s "import os" "os.chdir('.')"
50 loops, best of 5: 842 nsec per loop


But I don't think that chdir is likely to be the bottleneck in many 
scripts.


-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/YSWHM6NAF46JLTXXYXGLZJU7UMS7BC6B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Marc-Andre Lemburg]

On 15.09.2021 21:02, Guido van Rossum wrote:
> To make chdir() return a context manager *and* keep it working without calling
> `__enter__`, it would have to call `getcwd()`, which I've heard is expensive.
> 
> So I don't think that would work, alas.

At least on Linux, the cost for os.getcwd() is similar to the cost
of os.chdir(), but yes, since we can't have os.chdir() not change
the dir when called, the logic would need the extra os.getcwd() call:

# python3 -m timeit -s 'import os' "os.getcwd()"
50 loops, best of 5: 619 nsec per loop
# python3 -m timeit -s 'import os' "os.chdir('.')"
50 loops, best of 5: 726 nsec per loop

Here's simple implementation of the chdir() context manager:

import os
import pathlib

# chdir context manager
PlatformPath = pathlib.WindowsPath if os.name == 'nt' else pathlib.PosixPath
class chdir(PlatformPath):
def __init__(self, dir):
self.dir = dir
self.olddir = os.getcwd()
os.chdir(dir)
def __enter__(self):
return self
def __exit__(self, *exc):
os.chdir(self.olddir)
return False

# Normal chdir()
path = chdir('abc/')
print (os.getcwd())
print (path.olddir)

# chdir() context manager
with chdir('def/') as wd:
print (repr(wd))
print (os.getcwd())
print (os.listdir('.'))

For extra perks, I made os.chdir() return a pathlib Path object
and you get to see the old directory, so you can backtrack
if needed, even without a context manager.


> On Wed, Sep 15, 2021 at 11:55 AM Eric V. Smith  > wrote:
> 
> On 9/15/2021 2:48 PM, Eric Fahlgren wrote:
>> On Wed, Sep 15, 2021 at 12:21 AM Eric V. Smith > > wrote:
>>
>> And I'm not crazy about the name "workdir". To me, it sounds like it
>> returns something, not sets and resets something. But I don't have a
>> particularly great alternative in mind: in my own code I've used
>> "change_dir", which isn't awesome either.
>>
>>
>> Our version is named "pushdir", modeled after shell's pushd (even though
>> pushdir is a context manager and does auto-pop).  Everyone figures out
>> what it does at a glance.
> 
> That's a great name!
> 
> Although I think having os.chdir() return a context manager is a better
> design (assuming it can work, but at first blush it would seem so).
> 
> Eric
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Sep 15 2021)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KRMBQYQUCSU4GYN3F4OPUKVQ72VLCBIV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Guido van Rossum]

To make chdir() return a context manager *and* keep it working without
calling `__enter__`, it would have to call `getcwd()`, which I've heard is
expensive.

So I don't think that would work, alas.

On Wed, Sep 15, 2021 at 11:55 AM Eric V. Smith  wrote:

> On 9/15/2021 2:48 PM, Eric Fahlgren wrote:
>
> On Wed, Sep 15, 2021 at 12:21 AM Eric V. Smith  wrote:
>
>> And I'm not crazy about the name "workdir". To me, it sounds like it
>> returns something, not sets and resets something. But I don't have a
>> particularly great alternative in mind: in my own code I've used
>> "change_dir", which isn't awesome either.
>>
>
> Our version is named "pushdir", modeled after shell's pushd (even though
> pushdir is a context manager and does auto-pop).  Everyone figures out what
> it does at a glance.
>
> That's a great name!
>
> Although I think having os.chdir() return a context manager is a better
> design (assuming it can work, but at first blush it would seem so).
>
> Eric
>
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/U5UQFDAJ4KWG2OQ3YMP2THMSJ72JRV6Y/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LZL4YXXRW4TBBSFZVW46PTTUOT36LX62/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Eric V. Smith]

On 9/15/2021 2:48 PM, Eric Fahlgren wrote:
On Wed, Sep 15, 2021 at 12:21 AM Eric V. Smith > wrote:


And I'm not crazy about the name "workdir". To me, it sounds like it
returns something, not sets and resets something. But I don't have a
particularly great alternative in mind: in my own code I've used
"change_dir", which isn't awesome either.


Our version is named "pushdir", modeled after shell's pushd (even 
though pushdir is a context manager and does auto-pop).  Everyone 
figures out what it does at a glance.


That's a great name!

Although I think having os.chdir() return a context manager is a better 
design (assuming it can work, but at first blush it would seem so).


Eric


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/U5UQFDAJ4KWG2OQ3YMP2THMSJ72JRV6Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Eric Fahlgren]

On Wed, Sep 15, 2021 at 12:21 AM Eric V. Smith  wrote:

> And I'm not crazy about the name "workdir". To me, it sounds like it
> returns something, not sets and resets something. But I don't have a
> particularly great alternative in mind: in my own code I've used
> "change_dir", which isn't awesome either.
>

Our version is named "pushdir", modeled after shell's pushd (even though
pushdir is a context manager and does auto-pop).  Everyone figures out what
it does at a glance.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/5ZG2NM2ETPYEX5IBONDHFRB7625H2RY7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Joao S. O. Bueno]

On Tue, 14 Sept 2021 at 22:59, David Mertz, Ph.D. 
wrote:

> I think this would be convenient.
>
> And yes, it's not thread safe. But neither is os.chdir() to start with.
> Someone whose script, or library, wants to chdir can already shoot
> themselves in the foot. This makes that slightly less likely, not more.
>
The problem is someone thinking that by using this they would not be
shooting themselves on the foot
because "wow, a 2021 adition to the stdlib will certainly be thread and
async safe".

I am with Chris Angelico in that this would be dangerous.




>
> In terms of the bikeshed color, I think putting this in `pathlib` is the
> best approach for "modern Python."
>
OTOH, on  Pathlib we could have a proper implementation that could take care
of _all_ in context file operations, and internally use absolute paths
(instead
of calling chdir). Such an implementation would work.
```
with Path("/tmp").workdir as mydir:
with mydir.open("myfile"):
 ...
```
and maybe could be practical enough to stirr people away from "chdir".

>
>
> On Tue, Sep 14, 2021, 7:37 PM Cameron Simpson  wrote:
>
>> On 15Sep2021 07:50, Chris Angelico  wrote:
>> >On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson  wrote:
>> >> I know I'm atypical, but I have quite a lot of multithreaded stuff,
>> >> including command line code. So while it'd be ok to avoid this context
>> >> manager for my own code, I fear library modules, either stdlib or pypi,
>> >> quietly using this in their code, making them unuseable in the general
>> >> case. Unrepairably unuseable, for the user.
>> >
>> >Library code shouldn't be changing the working directory, context
>> >manager or not. That belongs to the application.
>>
>> Entirely agree.
>>
>> I'm concerned that convenient stackable chdir is a bug magnet, and would
>> creep into library code. Maybe not in the stdlib, but there's no point
>> writing such a context manager if it isn't goingg to be used, and
>> therefore it could get used in library code. Imagine when a popular pypi
>> module starts using it internally and breaks a multithreaded app
>> previously relying on it?
>>
>> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/KM6NEMJTUIMRTRTR4IZCW3CAZ264JBY2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6SURJ3IUMBH7IJTHQTZLOYXAY4SLFHIM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Filipe Laíns]

On Tue, 2021-09-14 at 21:43 +0200, Marc-Andre Lemburg wrote:
> Hello all,
> 
> I sometimes write Python scripts which need to work in specific
> work directories.
> 
> When putting such code into functions, the outer function typically
> does not expect the current work dir (CWD) to be changed, so wrap the
> code which need the (possibly) modified CWD using a simply context
> manager along the lines of:
> 
> class workdir:
>     def __init__(self, dir):
>     self.dir = dir
>     def __enter__(self):
>     self.curdir = os.getcwd()
>     os.chdir(self.dir)
>     def __exit__(self, *exc):
>     os.chdir(self.curdir)
>     return False
> 
> Would there be interest in adding something like this to the os module
> as os.workdir() ?
> 
> Example:
> 
> def backup_home_dir(account):
>     with os.workdir(os.path.join('/home', account)):
>     # Create a backup of the account dir, rooted at the account's
>     # home dir
>     restic.backup(repo, '.')
> 
> Notes:
> - The context manager is not thread safe. There's no thread safe model
>   for the current work dir. OTOH, scripts usually don't use threads,
>   so not a big deal.
> - The context manager could be made more elaborate, e.g. adding optional
>   logging for debugging purposes.
> - The same could be added to pathlib's objects as .workdir() method
>   returning a context manager.
> 
> Thoughts ?
> 
> Thanks,

bpo: https://bugs.python.org/issue25625
PR: https://github.com/python/cpython/pull/28271

Filipe Laíns


signature.asc
Description: This is a digitally signed message part
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SRILWIWGNFTL6B2QOLS4KIMLEXEUAHCB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Alex Waygood]

I also really like this idea. Though I wonder if it might be a better fit for 
`contextlib` rather than `os`? Can see arguments both ways (and it's a minor 
detail, anyway).

Alex

> On 14 Sep 2021, at 20:55, Paul Moore  wrote:
> 
> On Tue, 14 Sept 2021 at 20:44, Marc-Andre Lemburg  wrote:
>> 
>> I sometimes write Python scripts which need to work in specific
>> work directories.
>> 
>> When putting such code into functions, the outer function typically
>> does not expect the current work dir (CWD) to be changed, so wrap the
>> code which need the (possibly) modified CWD using a simply context
>> manager along the lines of:
>> 
>> class workdir:
>>def __init__(self, dir):
>>self.dir = dir
>>def __enter__(self):
>>self.curdir = os.getcwd()
>>os.chdir(self.dir)
>>def __exit__(self, *exc):
>>os.chdir(self.curdir)
>>return False
>> 
>> Would there be interest in adding something like this to the os module
>> as os.workdir() ?
> 
> I've needed (and implemented my own version of) this often enough that
> I'd be in favour of it. Of course, as you show, it's not hard to write
> it yourself, so I can live with it not getting implemented, too :-)
> 
> Paul
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/I6YFWYNUAV7G6DNPZTBJ6T3SGJDPGBJL/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/B2DICGY5Q65IFWPSBVFYCYNJBGJ7KNGY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Chris Angelico]

On Wed, Sep 15, 2021 at 9:22 PM Steven D'Aprano  wrote:
>
> On Wed, Sep 15, 2021 at 08:53:21PM +1000, Chris Angelico wrote:
>
> > Sometimes there is no valid path to something,
>
> I don't understand what you mean by that. If there is no valid path to
> something, where exactly are you cd'ing to, and how does it help you
> with the thing that has no valid path to it?

You retain it from when it DID have a path.

> > or you need to be
> > immune to directory renames. The normal solution would be to use
> > openat() etc (or, in Python, os.open() with a dir_fd). I'm not sure
> > what happens on Windows.
>
> Normal for whom?
>
> Let's not forget that there are still a bazillion people who write
> scripts in bash or some other shell, or Python, that just do the
> simplest thing that can possibly work. Namely cd into the directory they
> want to be in.
>
> This is a legitimate use-case. Not everyone needs to care about threads.

When you cd into something (on POSIX, at least), your process retains
a reference to it. Not a file descriptor as such, I think, but it has
some sort of reference. It is entirely possible to be in a directory
that no longer has a valid path name to it, or when the path to that
directory has changed. (Renames are easiest to demo, and can be seen
in bash without even entering Python.) The way I see it, this is
exactly the same use-case, just implemented slightly differently.

> > Additional wrinkle: Can this be made asyncio compatible? Is it
> > possible to make it, instead of thread-local, aware in some way of
> > which task it's running?
>
> Are you suggesting that you might want to run multiple async coroutines
> in different working directories?
>
> I think that designing this for threads or async is over-engineering.
>

If you don't care about going back to the original working directory,
just use os.chdir() as it already is. For a lot of scripts, ANY
context manager is over-engineering. Why is this being proposed?
Because there are use-cases where there needs to be a reset.

So if there needs to be a reset, that means you need to isolate the
action of "change working directory" to a subset of code. And while
it's of some value to be able to do that even if it's not thread-safe
or asyncio-safe, it would be of distinctly more value if it can be
aware of those use-cases. Depending on how that's implemented, it
might not even be that difficult - for instance, if there's a stack of
file descriptors, and all open() calls are rerouted via
os.open(dir_fd=top_of_stack), then it would be only a small amount
more work to make it per-task instead of per-thread.

Having some proper language support may be necessary here, though. I'm
not sure how deeply this would affect things. Of course, it would be
pretty straight-forward if it's all done cooperatively, with a
dedicated "open relative to top of stack" API.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LZ5FUKENL5ZZH5D4DYTSMCV2N67IWQOS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Steven D'Aprano]

On Wed, Sep 15, 2021 at 08:53:21PM +1000, Chris Angelico wrote:

> Sometimes there is no valid path to something,

I don't understand what you mean by that. If there is no valid path to 
something, where exactly are you cd'ing to, and how does it help you 
with the thing that has no valid path to it?


> or you need to be
> immune to directory renames. The normal solution would be to use
> openat() etc (or, in Python, os.open() with a dir_fd). I'm not sure
> what happens on Windows.

Normal for whom?

Let's not forget that there are still a bazillion people who write 
scripts in bash or some other shell, or Python, that just do the 
simplest thing that can possibly work. Namely cd into the directory they 
want to be in.

This is a legitimate use-case. Not everyone needs to care about threads.

> Additional wrinkle: Can this be made asyncio compatible? Is it
> possible to make it, instead of thread-local, aware in some way of
> which task it's running?

Are you suggesting that you might want to run multiple async coroutines 
in different working directories?

I think that designing this for threads or async is over-engineering. 

-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/OBSKQ5K2WVJRNDTW4UPQKFQK32FPKBTH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Eryk Sun]

On 9/15/21, Paul Moore  wrote:
>
> Just a somewhat off-topic note, but dir_fd arguments are only
> supported on Unix, and the functionality only appears to be present at
> the NT Kernel level on Windows, not in the Windows API.

Handle-relative paths are supported by all NT system calls that access
object paths, but NT doesn't support ".." components. Normal user-mode
programs can make system calls directly (e.g. call NtCreateFile
instead of CreateFile), but even if Python bypassed the Windows API to
support dir_fd, the lack of support for ".." components in relative
paths would be an annoying inconsistency with POSIX dir_fd support.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QO253Y4XOMJZC7YQQRZPYME353M7WDDA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Steven D'Aprano]

On Wed, Sep 15, 2021 at 11:56:17AM +0200, Marc-Andre Lemburg wrote:

> > - The context manager could be made more elaborate, e.g. adding optional
> >   logging for debugging purposes.
> 
> We could add a logger parameter to the context manager for
> this to customize the logger, or simply use the default one
> from logging.

That seems unnecessary. We don't put logging in other context managers.

with open(filename, logging=...) as f:

Keep workdir() nice and simple, let it do one thing (change the working 
directory and restore it on exit) and not overload it with "elaborate" 
functionality. If people want logging, they can handle logging 
themselves.

Or maybe they can use a logging context manager?


>   We could even have os.chdir() return the context
>   manager, without introducing any new API in the os module.

That's an interesting suggestion. I think I like it. No, I'm *sure* that 
I really like it. Let's do this!

# The old way:
os.chdir(dirname)
...
os.chdir(prevdir)

# The new way:
with os.chdir(dirname):
...

Of course the old way would still work.



-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/F4HYGBPJIMKBGBVSWF3KS3C3L25NGUVE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Chris Angelico]

On Wed, Sep 15, 2021 at 8:41 PM Christian Heimes  wrote:
>
> On 15/09/2021 11.56, Marc-Andre Lemburg wrote:
> > - Chris mentioned that library code should not be changing the
> >   CWD. In an ideal world, I'd agree, but there are cases where
> >   you don't have an option and you need to change the CWD in order
> >   make certain things work, e.g. you don't control the code you
> >   want to run, but need to make it work in a specific directory
> >   by changing the CWD and then passing relative paths to the
> >   code.
>
> This seems rather hypothetical to me. Can you provide a real-world
> example where you cannot use absolute paths?
>

Sometimes there is no valid path to something, or you need to be
immune to directory renames. The normal solution would be to use
openat() etc (or, in Python, os.open() with a dir_fd). I'm not sure
what happens on Windows.

Does Windows let you open a directory and retain a handle to it? If
not, it probably just won't allow the concept in question, and
absolutification would be a solution. This would be another good
reason to wrap this up into a library - on POSIX systems, it can open
a directory and make everything use dir_fd, but on Windows, it
remembers the drive letter and path, and absolutes out all path names.

Additional wrinkle: Can this be made asyncio compatible? Is it
possible to make it, instead of thread-local, aware in some way of
which task it's running?
ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3TQWWXLKOQIVUHYLI5GHQPX7L36X6QMY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Serhiy Storchaka]

14.09.21 22:43, Marc-Andre Lemburg пише:
> Would there be interest in adding something like this to the os module
> as os.workdir() ?

There is already an open issue and PR for this:

https://bugs.python.org/issue25625
https://github.com/python/cpython/pull/28271

My main concern is that it is not thread-safe (nor generator-safe, nor
coroutine-safe), but on other hand, redirect_stdout() and
redirect_stderr() have same limitations.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KCDF3L7SDPVOSFNPX7GQDWEE5HPZRXMT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Christian Heimes]

On 15/09/2021 11.56, Marc-Andre Lemburg wrote:
> - Chris mentioned that library code should not be changing the
>   CWD. In an ideal world, I'd agree, but there are cases where
>   you don't have an option and you need to change the CWD in order
>   make certain things work, e.g. you don't control the code you
>   want to run, but need to make it work in a specific directory
>   by changing the CWD and then passing relative paths to the
>   code.

This seems rather hypothetical to me. Can you provide a real-world
example where you cannot use absolute paths?

Christian

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UIBLP5GNXIVAE52IUJCCTLOFVQ7LOMYS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Marc-Andre Lemburg]

On 15.09.2021 00:16, Guido van Rossum wrote:
> Here I think we need to drop our perfectionist attitude. When I saw 
> Marc-Andre's
> proposal my first response was also "but what about threads." But really,
> os.chdir() is the culprit here, and since it's a syscall we can't fix it. If 
> we
> can live with that, we can live with the proposed os.workdir(). The docs just
> need a warning about threads.
> 
> Yes, you could create a thread-safe version of this by overriding open() -- 
> and
> several hundred other functions that refer to pathnames natively. Such a 
> project
> will inevitably have serious backwards compatibility concerns (any use of
> pathnames from C extensions will not work right) so it's just not going to be 
> done.
> 
> If we don't offer this in the stdlib, users will just implement this 
> themselves,
> poorly (for example, by not restoring the original when done).

Thanks for all the feedback. I'm just going to reply here in a single
email.

First, let me repeat the notes I had added to the proposal, since these
seem to have gotten lost in the thread. I'll some more context:

> Notes:
> - The context manager is not thread safe. There's no thread safe model
>   for the current work dir. OTOH, scripts usually don't use threads,
>   so not a big deal.

The CWD (current working directory) concept is a very old concept
where the OS manages a pointer into the file system on a per process
basis in order to handle requests for relative paths and map them
to absolute paths. https://en.wikipedia.org/wiki/Working_directory

Since this is per process and managed by the OS, the concept does
not extend to threads (possibly) running in the process.

While Python could theoretically try to emulate parts of this for
threads as well, this is not within the scope of os.workdir().

It's supposed to be a simple convenient way of saying: please
chdir here, do something and then return to the previously
active work dir.

The problem it's trying to solve is that last part... functions
forgetting to restore the previous active work dir can easily
break other parts of the script, since the CWD is used for
resolving relative paths in Python as well.

If thread-safety is seen as a big problem, we could add a concept
of managing a single CWD across all threads using lock protected
globals. Not sure whether this is worth the trouble, but perhaps
making it optional would ease some concerns. In this case, os.chdir()
would have to be adapted as well, in order to update the Python
CWD state in the same way (and possibly block or warn when os.chdir()
is called inside an active with os.workdir() block).

> - The context manager could be made more elaborate, e.g. adding optional
>   logging for debugging purposes.

We could add a logger parameter to the context manager for
this to customize the logger, or simply use the default one
from logging.

> - The same could be added to pathlib's objects as .workdir() method
>   returning a context manager.

Since people are starting to use pathlib more, this would be
a natural thing to do, as others have noted already :-)

The context manager will also accept a path-like object as
argument (simply because it passes this directly to os.chdir()
which already does).

As for having the context manager method on a Path object
return the Path object itself (suggested by David), this could
be made to work by returning a subclass of Path, which then
has the necessary extras needed for CWD processing.

I'll add some more notes:

- Why in the os module ? Because this is where you'd search for a way
  to set the CWD and indeed, we have os.chdir() and os.getcwd() in
  this module. Also see the note on pathlib above.

- There are other global settings which could benefit from such
  context managers as well, such as umask (mentioned by Cameron)
  or the locale settings, which you sometimes want to temporarily
  modify, but those are not used as often.

- Why the lowercase name ? Simply to make it fit the other os module
  APIs. Classes should normally have SnakeCase names, but since we're
  not interested in the class aspect here, but rather the context
  manager API of the resulting object, I think it's fair to stick
  with lower case. Many other context managers use a lower case name
  as well.

- Chris mentioned that library code should not be changing the
  CWD. In an ideal world, I'd agree, but there are cases where
  you don't have an option and you need to change the CWD in order
  make certain things work, e.g. you don't control the code you
  want to run, but need to make it work in a specific directory
  by changing the CWD and then passing relative paths to the
  code.

  If you use subprocesses for this, you can make this work,
  since the module forks for you and then sets the CWD in the
  forked process on Unix, but this still leaves you with a gap
  in Python, since you may well want to code in the same
  block which e.g. relies on os.abspath() working with your
  modified 

[Python-ideas] Re: os.workdir() context manager

[Eric V. Smith]

> On Sep 15, 2021, at 4:11 AM, Christian Heimes  wrote:
> 
> On 15/09/2021 09.21, Eric V. Smith wrote:
>>> On 9/15/2021 3:02 AM, Christian Heimes wrote:
>>> On 15/09/2021 01.55, Guido van Rossum wrote:
 I know where I'd file a bug. :-)
 
 "Bug magnet" is an extremely subjective pejorative term. When the
 *better* way to do things (os.workdir()) is harder than the *easy* way
 to do (os.chdir()), which is the real bug magnet?
>>> The "better way" to handle current working directory is to use the
>>> modern *at() variants of syscalls, e.g. openat() instead open(). The
>>> variants take an additional file descriptor dirfd that is used as the
>>> current working directory for the syscall.
>> 
>> While I generally agree, the only times I've written a context manager
>> like os.workdir() is when running an executable with subprocess.call(),
>> and the executable requires that its current directory be set to some
>> specific directory. So while I don't use this functionality very often,
>> there are times when nothing else will do. I realize I could handle this
>> temporary working directory with yet another executable (including a
>> shell), but using a context manager is just easier, and I only use this
>> in single-threaded programs.
> 
> You don't have to change the current working directory of your process
> in order to run a child process in a different working directory.
> subprocess.call() and other functions in the subprocess module accept a
> "cwd" argument that lets you run a process with a different working
> directory. The "cwd" argument is thread safe.

I learned something today. Thanks!

Eric
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/MTNIK4Q3AQH5Q7IVA44EFPMJODCUWAIN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Christian Heimes]

On 15/09/2021 09.21, Eric V. Smith wrote:
> On 9/15/2021 3:02 AM, Christian Heimes wrote:
>> On 15/09/2021 01.55, Guido van Rossum wrote:
>>> I know where I'd file a bug. :-)
>>>
>>> "Bug magnet" is an extremely subjective pejorative term. When the
>>> *better* way to do things (os.workdir()) is harder than the *easy* way
>>> to do (os.chdir()), which is the real bug magnet?
>> The "better way" to handle current working directory is to use the
>> modern *at() variants of syscalls, e.g. openat() instead open(). The
>> variants take an additional file descriptor dirfd that is used as the
>> current working directory for the syscall.
> 
> While I generally agree, the only times I've written a context manager
> like os.workdir() is when running an executable with subprocess.call(),
> and the executable requires that its current directory be set to some
> specific directory. So while I don't use this functionality very often,
> there are times when nothing else will do. I realize I could handle this
> temporary working directory with yet another executable (including a
> shell), but using a context manager is just easier, and I only use this
> in single-threaded programs.

You don't have to change the current working directory of your process
in order to run a child process in a different working directory.
subprocess.call() and other functions in the subprocess module accept a
"cwd" argument that lets you run a process with a different working
directory. The "cwd" argument is thread safe.


> And I'm not crazy about the name "workdir". To me, it sounds like it
> returns something, not sets and resets something. But I don't have a
> particularly great alternative in mind: in my own code I've used
> "change_dir", which isn't awesome either.

Speaking with almost 25 years experience in Unix system scripting:
Changing the current working directory during the runtime of a process
is problematic and can lead to bugs. In general applications should only
change their working directory once right after start and not rely on
the cwd. It's better to normalize paths, use absolute paths, or *at()
syscalls with dirfd.

In my opinion a workdir() context manager would only be useful for small
quick-n-dirty scripts.

Christian


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DGGZHKSJGVRBRARZF7T63FFR2OOGF4XC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Paul Moore]

On Wed, 15 Sept 2021 at 08:02, Christian Heimes  wrote:

> The "better way" to handle current working directory is to use the
> modern *at() variants of syscalls, e.g. openat() instead open(). The
> variants take an additional file descriptor dirfd that is used as the
> current working directory for the syscall.

Just a somewhat off-topic note, but dir_fd arguments are only
supported on Unix, and the functionality only appears to be present at
the NT Kernel level on Windows, not in the Windows API. So this is
only a "better" approach if you are writing Unix-only code.
Conversely, the most significant cases where I've wanted a "set the
current directory" context manager are to run some code with a
temporary directory as CWD, but then make sure I don't leave that as
the CWD, because Windows won't let me delete a directory that's the
CWD of a running process, so my cleanup will fail if I don't reset the
CWD. So the proposed context manager is probably more beneficial for
Windows users.

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/7DWT26TNJ3HPZ7XAQG3HP74QSQX2ISNC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Eric V. Smith]

On 9/15/2021 3:02 AM, Christian Heimes wrote:

On 15/09/2021 01.55, Guido van Rossum wrote:

I know where I'd file a bug. :-)

"Bug magnet" is an extremely subjective pejorative term. When the
*better* way to do things (os.workdir()) is harder than the *easy* way
to do (os.chdir()), which is the real bug magnet?

The "better way" to handle current working directory is to use the
modern *at() variants of syscalls, e.g. openat() instead open(). The
variants take an additional file descriptor dirfd that is used as the
current working directory for the syscall.


While I generally agree, the only times I've written a context manager 
like os.workdir() is when running an executable with subprocess.call(), 
and the executable requires that its current directory be set to some 
specific directory. So while I don't use this functionality very often, 
there are times when nothing else will do. I realize I could handle this 
temporary working directory with yet another executable (including a 
shell), but using a context manager is just easier, and I only use this 
in single-threaded programs.


And I'm not crazy about the name "workdir". To me, it sounds like it 
returns something, not sets and resets something. But I don't have a 
particularly great alternative in mind: in my own code I've used 
"change_dir", which isn't awesome either.


Eric

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6VGCVAHDHZDMVBU2QHMLXE6IXL3DXYGI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Christian Heimes]

On 15/09/2021 01.55, Guido van Rossum wrote:
> I know where I'd file a bug. :-)
> 
> "Bug magnet" is an extremely subjective pejorative term. When the
> *better* way to do things (os.workdir()) is harder than the *easy* way
> to do (os.chdir()), which is the real bug magnet?

The "better way" to handle current working directory is to use the
modern *at() variants of syscalls, e.g. openat() instead open(). The
variants take an additional file descriptor dirfd that is used as the
current working directory for the syscall.

Christian

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FDCKFPVBBSNLBDBTI577NP55JN2DBWCE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[David Mertz, Ph.D.]

On Tue, Sep 14, 2021, 11:17 PM Finn Mason  wrote:

> I disagree. The way I see it, `pathlib` is home of *Path(). No functions.
> Just *Path(). Then again, I didn't write pathlib. I could be
> misunderstanding the intent.
>

When I wrote "in" I was thinking more of a method than a function. E.g.

with Path("/new/dir/").workdir() as wd:
# ... do stuff ...

>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IQ5J2T26WZPZTV4DOOD724CWRETM2PFS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Finn Mason]

On Tue, Sep 14, 2021, 7:58 PM David Mertz, Ph.D. 
wrote:

> I think this would be convenient.
>
> And yes, it's not thread safe. But neither is os.chdir() to start with.
> Someone whose script, or library, wants to chdir can already shoot
> themselves in the foot. This makes that slightly less likely, not more.
>

I agree.

In terms of the bikeshed color, I think putting this in `pathlib` is the
> best approach for "modern Python."
>

I disagree. The way I see it, `pathlib` is home of *Path(). No functions.
Just *Path(). Then again, I didn't write pathlib. I could be
misunderstanding the intent.
However, workdir() doesn't deal with Paths. Sure, it takes a PathLike as an
argument, but it doesn't perform operations on a Path or return a Path.
I think changing the CWD belongs squarely in the os module.

>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FRCHASRXJLEUCMYBPPRJLLBCCQS22AVR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[David Mertz, Ph.D.]

I think this would be convenient.

And yes, it's not thread safe. But neither is os.chdir() to start with.
Someone whose script, or library, wants to chdir can already shoot
themselves in the foot. This makes that slightly less likely, not more.

In terms of the bikeshed color, I think putting this in `pathlib` is the
best approach for "modern Python."


On Tue, Sep 14, 2021, 7:37 PM Cameron Simpson  wrote:

> On 15Sep2021 07:50, Chris Angelico  wrote:
> >On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson  wrote:
> >> I know I'm atypical, but I have quite a lot of multithreaded stuff,
> >> including command line code. So while it'd be ok to avoid this context
> >> manager for my own code, I fear library modules, either stdlib or pypi,
> >> quietly using this in their code, making them unuseable in the general
> >> case. Unrepairably unuseable, for the user.
> >
> >Library code shouldn't be changing the working directory, context
> >manager or not. That belongs to the application.
>
> Entirely agree.
>
> I'm concerned that convenient stackable chdir is a bug magnet, and would
> creep into library code. Maybe not in the stdlib, but there's no point
> writing such a context manager if it isn't goingg to be used, and
> therefore it could get used in library code. Imagine when a popular pypi
> module starts using it internally and breaks a multithreaded app
> previously relying on it?
>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KM6NEMJTUIMRTRTR4IZCW3CAZ264JBY2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Guido van Rossum]

For the stdlib context managers that I know of, all-lowercase seems the
convention. Would it even be a class? The simplest implementation would use
contextlib.contextmanager (unless that’s a undesirable dependency for
os.py).

—Guido

On Tue, Sep 14, 2021 at 17:53 Finn Mason  wrote:

> BTW, should it be `WorkDir` instead of `workdir` because it's a class, or
> would that be too inconsistent?
>
>
> On Tue, Sep 14, 2021, 6:47 PM Finn Mason  wrote:
>
>>
>> On Tue, Sep 14, 2021, 5:36 PM Cameron Simpson  wrote:
>>
>>> On 15Sep2021 07:50, Chris Angelico  wrote:
>>> >On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson  wrote:
>>> >> I know I'm atypical, but I have quite a lot of multithreaded stuff,
>>> >> including command line code. So while it'd be ok to avoid this context
>>> >> manager for my own code, I fear library modules, either stdlib or
>>> pypi,
>>> >> quietly using this in their code, making them unuseable in the general
>>> >> case. Unrepairably unuseable, for the user.
>>> >
>>> >Library code shouldn't be changing the working directory, context
>>> >manager or not. That belongs to the application.
>>>
>>> Entirely agree.
>>>
>>> I'm concerned that convenient stackable chdir is a bug magnet, and would
>>> creep into library code. Maybe not in the stdlib, but there's no point
>>> writing such a context manager if it isn't going to be used, and
>>> therefore it could get used in library code. Imagine when a popular pypi
>>> module starts using it internally and breaks a multithreaded app
>>> previously relying on it?
>>>
>>
>> I don't think we should worry about it "creeping into library code." The
>> thread-unsafety is not a cause of this context manager. It comes from the
>> preexisting `os.chdir()`. If the library is changing the CWD, it's already
>> thread-unsafe. It's not because of the new context manager. All
>> `os.workdir()` does is make things easier.
>> However, if it's implemented (which I personally support), there should
>> still of course be a warning in the documentation. But I'd like to
>> emphasize that *it is not because of `workdir()` itself, but the
>> underlying `chdir()`!*
>>
>> On 14Sep2021 15:16, Guido van Rossum  wrote:
>> >Here I think we need to drop our perfectionist attitude.
>>
>> I completely agree.
>>
>>> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/J4L3I3QWV65GPKHJ5RZ4NASGX7ZIS774/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
--Guido (mobile)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/X6IGI33OS7534GF7CCFUKJXTG5OMJAXA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Finn Mason]

BTW, should it be `WorkDir` instead of `workdir` because it's a class, or
would that be too inconsistent?

On Tue, Sep 14, 2021, 6:47 PM Finn Mason  wrote:

>
> On Tue, Sep 14, 2021, 5:36 PM Cameron Simpson  wrote:
>
>> On 15Sep2021 07:50, Chris Angelico  wrote:
>> >On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson  wrote:
>> >> I know I'm atypical, but I have quite a lot of multithreaded stuff,
>> >> including command line code. So while it'd be ok to avoid this context
>> >> manager for my own code, I fear library modules, either stdlib or pypi,
>> >> quietly using this in their code, making them unuseable in the general
>> >> case. Unrepairably unuseable, for the user.
>> >
>> >Library code shouldn't be changing the working directory, context
>> >manager or not. That belongs to the application.
>>
>> Entirely agree.
>>
>> I'm concerned that convenient stackable chdir is a bug magnet, and would
>> creep into library code. Maybe not in the stdlib, but there's no point
>> writing such a context manager if it isn't going to be used, and
>> therefore it could get used in library code. Imagine when a popular pypi
>> module starts using it internally and breaks a multithreaded app
>> previously relying on it?
>>
>
> I don't think we should worry about it "creeping into library code." The
> thread-unsafety is not a cause of this context manager. It comes from the
> preexisting `os.chdir()`. If the library is changing the CWD, it's already
> thread-unsafe. It's not because of the new context manager. All
> `os.workdir()` does is make things easier.
> However, if it's implemented (which I personally support), there should
> still of course be a warning in the documentation. But I'd like to
> emphasize that *it is not because of `workdir()` itself, but the
> underlying `chdir()`!*
>
> On 14Sep2021 15:16, Guido van Rossum  wrote:
> >Here I think we need to drop our perfectionist attitude.
>
> I completely agree.
>
>>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/J4L3I3QWV65GPKHJ5RZ4NASGX7ZIS774/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Finn Mason]

On Tue, Sep 14, 2021, 5:36 PM Cameron Simpson  wrote:

> On 15Sep2021 07:50, Chris Angelico  wrote:
> >On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson  wrote:
> >> I know I'm atypical, but I have quite a lot of multithreaded stuff,
> >> including command line code. So while it'd be ok to avoid this context
> >> manager for my own code, I fear library modules, either stdlib or pypi,
> >> quietly using this in their code, making them unuseable in the general
> >> case. Unrepairably unuseable, for the user.
> >
> >Library code shouldn't be changing the working directory, context
> >manager or not. That belongs to the application.
>
> Entirely agree.
>
> I'm concerned that convenient stackable chdir is a bug magnet, and would
> creep into library code. Maybe not in the stdlib, but there's no point
> writing such a context manager if it isn't going to be used, and
> therefore it could get used in library code. Imagine when a popular pypi
> module starts using it internally and breaks a multithreaded app
> previously relying on it?
>

I don't think we should worry about it "creeping into library code." The
thread-unsafety is not a cause of this context manager. It comes from the
preexisting `os.chdir()`. If the library is changing the CWD, it's already
thread-unsafe. It's not because of the new context manager. All
`os.workdir()` does is make things easier.
However, if it's implemented (which I personally support), there should
still of course be a warning in the documentation. But I'd like to
emphasize that *it is not because of `workdir()` itself, but the underlying
`chdir()`!*

On 14Sep2021 15:16, Guido van Rossum  wrote:
>Here I think we need to drop our perfectionist attitude.

I completely agree.

>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DIQXTTKRASFB2DMQZ4THMS4TK2ALK7BA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Guido van Rossum]

On Tue, Sep 14, 2021 at 4:36 PM Cameron Simpson  wrote:

> On 15Sep2021 07:50, Chris Angelico  wrote:
> >On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson  wrote:
> >> I know I'm atypical, but I have quite a lot of multithreaded stuff,
> >> including command line code. So while it'd be ok to avoid this context
> >> manager for my own code, I fear library modules, either stdlib or pypi,
> >> quietly using this in their code, making them unuseable in the general
> >> case. Unrepairably unuseable, for the user.
> >
> >Library code shouldn't be changing the working directory, context
> >manager or not. That belongs to the application.
>
> Entirely agree.
>
> I'm concerned that convenient stackable chdir is a bug magnet, and would
> creep into library code. Maybe not in the stdlib, but there's no point
> writing such a context manager if it isn't goingg to be used, and
> therefore it could get used in library code. Imagine when a popular pypi
> module starts using it internally and breaks a multithreaded app
> previously relying on it?
>

I know where I'd file a bug. :-)

"Bug magnet" is an extremely subjective pejorative term. When the *better*
way to do things (os.workdir()) is harder than the *easy* way to do
(os.chdir()), which is the real bug magnet?

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/66ZPA74HWFPYB7G7ZLHLONX32KSWIW4E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Chris Angelico]

On Wed, Sep 15, 2021 at 9:36 AM Cameron Simpson  wrote:
>
> On 15Sep2021 07:50, Chris Angelico  wrote:
> >On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson  wrote:
> >> I know I'm atypical, but I have quite a lot of multithreaded stuff,
> >> including command line code. So while it'd be ok to avoid this context
> >> manager for my own code, I fear library modules, either stdlib or pypi,
> >> quietly using this in their code, making them unuseable in the general
> >> case. Unrepairably unuseable, for the user.
> >
> >Library code shouldn't be changing the working directory, context
> >manager or not. That belongs to the application.
>
> Entirely agree.
>
> I'm concerned that convenient stackable chdir is a bug magnet, and would
> creep into library code. Maybe not in the stdlib, but there's no point
> writing such a context manager if it isn't goingg to be used, and
> therefore it could get used in library code. Imagine when a popular pypi
> module starts using it internally and breaks a multithreaded app
> previously relying on it?
>
> >> I think what would be more useful is a context manager which worked on a
> >> threading.local which pushed/popped a reference directory path, and had
> >> an open() method which used that (and friends for other pathname based
> >> functions).
> >
> >Hmm. If it's going to push/pop directories, it should probably work
> >with file descriptors rather than path names, and effectively turn all
> >open calls into openat calls.
>
> Yah. I even thought about that during my sketch. But (a) I've possibly
> never used openat() so I'm unfamiliar with it (other than its existence)
> and (b) I don't think that many other OS calls accept a file descriptor.
> Which still leaves one with the os.path.join() approach, and a mix might
> be inconsistent.
>

I believe that, at the OS level, there are *at() versions of
everything relevant. But the question is more whether it would work at
the Python level. For instance, you can use os.open() with a dir_fd
parameter, but I don't think there's a good way to make the builtin
open() use a reference directory.

Making that happen would probably be a good thing, but a decidedly
nontrivial change.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/NPZC52I3V3Q4JMBAY3NFXIPWCEZX4DVJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Cameron Simpson]

On 14Sep2021 15:16, Guido van Rossum  wrote:
>Here I think we need to drop our perfectionist attitude. When I saw
>Marc-Andre's proposal my first response was also "but what about threads."
>But really, os.chdir() is the culprit here, and since it's a syscall we
>can't fix it. If we can live with that, we can live with the proposed
>os.workdir(). The docs just need a warning about threads.
[...]
>If we don't offer this in the stdlib, users will just implement this
>themselves, poorly (for example, by not restoring the original when 
>done).

Agreed here. As long as it has a big clear warning about process global 
state.

That is the beauty of libraries, to me: not just reuse, but a correct 
implementation is correct everywhere, and likewise a bugfix fixes all 
the things.

Cheers,
Cameron Simpson 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/BLFDWLLFOMARR532GL7GRN3GUXL2S4SM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Cameron Simpson]

On 15Sep2021 07:50, Chris Angelico  wrote:
>On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson  wrote:
>> I know I'm atypical, but I have quite a lot of multithreaded stuff,
>> including command line code. So while it'd be ok to avoid this context
>> manager for my own code, I fear library modules, either stdlib or pypi,
>> quietly using this in their code, making them unuseable in the general
>> case. Unrepairably unuseable, for the user.
>
>Library code shouldn't be changing the working directory, context
>manager or not. That belongs to the application.

Entirely agree.

I'm concerned that convenient stackable chdir is a bug magnet, and would 
creep into library code. Maybe not in the stdlib, but there's no point 
writing such a context manager if it isn't goingg to be used, and 
therefore it could get used in library code. Imagine when a popular pypi 
module starts using it internally and breaks a multithreaded app 
previously relying on it?

>> I think what would be more useful is a context manager which worked on a
>> threading.local which pushed/popped a reference directory path, and had
>> an open() method which used that (and friends for other pathname based
>> functions).
>
>Hmm. If it's going to push/pop directories, it should probably work
>with file descriptors rather than path names, and effectively turn all
>open calls into openat calls.

Yah. I even thought about that during my sketch. But (a) I've possibly 
never used openat() so I'm unfamiliar with it (other than its existence) 
and (b) I don't think that many other OS calls accept a file descriptor.  
Which still leaves one with the os.path.join() approach, and a mix might 
be inconsistent.

Cheers,
Cameron Simpson 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VX4O43SOLPSKDP4OVOYZYOTBH7BUPXWZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Guido van Rossum]

Here I think we need to drop our perfectionist attitude. When I saw
Marc-Andre's proposal my first response was also "but what about threads."
But really, os.chdir() is the culprit here, and since it's a syscall we
can't fix it. If we can live with that, we can live with the proposed
os.workdir(). The docs just need a warning about threads.

Yes, you could create a thread-safe version of this by overriding open() --
and several hundred other functions that refer to pathnames natively. Such
a project will inevitably have serious backwards compatibility concerns
(any use of pathnames from C extensions will not work right) so it's just
not going to be done.

If we don't offer this in the stdlib, users will just implement this
themselves, poorly (for example, by not restoring the original when done).

On Tue, Sep 14, 2021 at 2:43 PM Cameron Simpson  wrote:

> On 14Sep2021 21:43, M.-A. Lemburg  wrote:
> >- The context manager is not thread safe. There's no thread safe model
> >  for the current work dir. OTOH, scripts usually don't use threads,
> >  so not a big deal.
>
> This is the source of my concerns. Though of course it applies to any
> process global state. It would need this stated up front in big letters
> (as of course does chdir itself). My real concern is that this can leak
> into other functions whose use then makes whatever uses them to become
> inherently and unrepairably not thread safe.
>
> I know I'm atypical, but I have quite a lot of multithreaded stuff,
> including command line code. So while it'd be ok to avoid this context
> manager for my own code, I fear library modules, either stdlib or pypi,
> quietly using this in their code, making them unuseable in the general
> case. Unrepairably unuseable, for the user.
>
> I think what would be more useful is a context manager which worked on a
> threading.local which pushed/popped a reference directory path, and had
> an open() method which used that (and friends for other pathname based
> functions).
>
> In my own code I'd write this like (sketch, untested):
>
> from cs.threads import State as ThreadState
>
> class RefDir(ThreadState)
>
>   def __init__(self, refpath=None):
> if refpath is None:
>   refpath = os.getcwd()
> self.refpath = abspath(refpath)
>
>   def open(self, path, *a, **kw):
> if not isabs(path):
>   path = os.path.join(self.refpath, path)
> return open(path, *a, **kw) # calls the builtin open()
>
>   ... listdir, mkdir, etc etc ...
>
>   # on reflection, __enter__ and __exit__ would make the below even
>   # more convenient
>   @contextmanager
>   def dirpath(newrefpath):
> ''' Push `newrefpath` for the duration of the context manager.
> '''
> with self(refpath=newrefpath):
>   yield self.refpath
>
> and then use it like this:
>
> R = RefDir()
> 
> with R.dirpath('/some/new/place') as newpath:
>   ...
>   with R.open("something.txt") as f:
>   ... work on /some/new/place/something.txt ...
>
> In the above, cs.threads.State is a threading.lcoal subclass which is
> also a context manager whose operation pushes arbitrary attribute
> values.  Great for thread safe execution scoped state. Like this
> example.
>
> All that said, I wrote pretty much exactly what you describe just the
> other week for umask().
>
> Cheers,
> Cameron Simpson 
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/EJETKXD7PAW3XKD5AW4OM45XKDSNX2QJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KS3KYQA65NS47CMY5LAEMPNINUNRZRY3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Chris Angelico]

On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson  wrote:
>
> On 14Sep2021 21:43, M.-A. Lemburg  wrote:
> >- The context manager is not thread safe. There's no thread safe model
> >  for the current work dir. OTOH, scripts usually don't use threads,
> >  so not a big deal.
>
> This is the source of my concerns. Though of course it applies to any
> process global state. It would need this stated up front in big letters
> (as of course does chdir itself). My real concern is that this can leak
> into other functions whose use then makes whatever uses them to become
> inherently and unrepairably not thread safe.
>
> I know I'm atypical, but I have quite a lot of multithreaded stuff,
> including command line code. So while it'd be ok to avoid this context
> manager for my own code, I fear library modules, either stdlib or pypi,
> quietly using this in their code, making them unuseable in the general
> case. Unrepairably unuseable, for the user.

Library code shouldn't be changing the working directory, context
manager or not. That belongs to the application.

> I think what would be more useful is a context manager which worked on a
> threading.local which pushed/popped a reference directory path, and had
> an open() method which used that (and friends for other pathname based
> functions).

Hmm. If it's going to push/pop directories, it should probably work
with file descriptors rather than path names, and effectively turn all
open calls into openat calls. I'm currently not sure what the best way
to do that in Python is.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VKJDWN6J7NP4GHBBARF44XWTWWD3UCJ4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Cameron Simpson]

On 14Sep2021 21:43, M.-A. Lemburg  wrote:
>- The context manager is not thread safe. There's no thread safe model
>  for the current work dir. OTOH, scripts usually don't use threads,
>  so not a big deal.

This is the source of my concerns. Though of course it applies to any 
process global state. It would need this stated up front in big letters 
(as of course does chdir itself). My real concern is that this can leak 
into other functions whose use then makes whatever uses them to become 
inherently and unrepairably not thread safe.

I know I'm atypical, but I have quite a lot of multithreaded stuff, 
including command line code. So while it'd be ok to avoid this context 
manager for my own code, I fear library modules, either stdlib or pypi, 
quietly using this in their code, making them unuseable in the general 
case. Unrepairably unuseable, for the user.

I think what would be more useful is a context manager which worked on a 
threading.local which pushed/popped a reference directory path, and had 
an open() method which used that (and friends for other pathname based 
functions).

In my own code I'd write this like (sketch, untested):

from cs.threads import State as ThreadState

class RefDir(ThreadState)

  def __init__(self, refpath=None):
if refpath is None:
  refpath = os.getcwd()
self.refpath = abspath(refpath)

  def open(self, path, *a, **kw):
if not isabs(path):
  path = os.path.join(self.refpath, path)
return open(path, *a, **kw) # calls the builtin open()

  ... listdir, mkdir, etc etc ...

  # on reflection, __enter__ and __exit__ would make the below even 
  # more convenient
  @contextmanager
  def dirpath(newrefpath):
''' Push `newrefpath` for the duration of the context manager.
'''
with self(refpath=newrefpath):
  yield self.refpath

and then use it like this:

R = RefDir()

with R.dirpath('/some/new/place') as newpath:
  ...
  with R.open("something.txt") as f:
  ... work on /some/new/place/something.txt ...

In the above, cs.threads.State is a threading.lcoal subclass which is 
also a context manager whose operation pushes arbitrary attribute 
values.  Great for thread safe execution scoped state. Like this 
example.

All that said, I wrote pretty much exactly what you describe just the 
other week for umask().

Cheers,
Cameron Simpson 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/EJETKXD7PAW3XKD5AW4OM45XKDSNX2QJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: os.workdir() context manager

[Paul Moore]

On Tue, 14 Sept 2021 at 20:44, Marc-Andre Lemburg  wrote:
>
> I sometimes write Python scripts which need to work in specific
> work directories.
>
> When putting such code into functions, the outer function typically
> does not expect the current work dir (CWD) to be changed, so wrap the
> code which need the (possibly) modified CWD using a simply context
> manager along the lines of:
>
> class workdir:
> def __init__(self, dir):
> self.dir = dir
> def __enter__(self):
> self.curdir = os.getcwd()
> os.chdir(self.dir)
> def __exit__(self, *exc):
> os.chdir(self.curdir)
> return False
>
> Would there be interest in adding something like this to the os module
> as os.workdir() ?

I've needed (and implemented my own version of) this often enough that
I'd be in favour of it. Of course, as you show, it's not hard to write
it yourself, so I can live with it not getting implemented, too :-)

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/I6YFWYNUAV7G6DNPZTBJ6T3SGJDPGBJL/
Code of Conduct: http://python.org/psf/codeofconduct/