[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Miguel Ángel Prosper]

> It looks like you have a good handle on the code -- do you want to submit a 
> PR to GitHub to add such a parameter?

Thanks, but I'm not really sure how to implement it in the ProcessPoolExecutor, 
I just think the solution is probably related to the code responsible of 
handling a failed initializer (since they do very similar things). On the 
ThreadPoolExecutor maybe I could, but I haven't really checked for side effects 
or weird things.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FGTSBGTXW63SSXFQBASLCTXAT4M6RGGN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Miguel Ángel Prosper]

> But I don’t think “terminate” is the right name. Maybe “cancel”? Or even 
> “shutdown(wait=whatever, cancel=True)?”

"terminate" was definitely not a good name, especially because it doesn't 
actually terminate anything, it just cancels some of the operations. Since it 
also has to cooperate with the shutdown method, executing code after the state 
change but also before the wait, probably the best way to implement it would be 
just to add an additional parameter to shutdown. "cancel" seems perfect, by 
default set to false, to not change the current behavior.

Then maybe it would be a good idea to change the __exit__ method to set cancel 
to true if an exception is being handled, but I'm not sure about that.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XMDBSSMBXFJG2C3DXTNLTCNBGAUQOO2K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Andrew Barnert via Python-ideas]

On Jan 3, 2020, at 10:11, Miguel Ángel Prosper  
wrote:
> 
> 
>> 
>> Having a way to clear the queue and then shutdown once existing jobs are 
>> done is a lot
>> more manageable.
> ...
>> So the only clean way to do this is cooperative: flush the queue, send some 
>> kind of
>> message to all children telling them to finish as quickly as possible, then 
>> wait for them
>> to finish.
> 
> I was personally thinking of an implementation like that, cancel all still in 
> pending and if wait is true the wait for the ones running, for both 
> implementations.

OK, that makes sense. And it seems like it should be implementable; the only 
hard part is identifying all the edge cases and verifying they all do the right 
thing for both threads and processes.

But I don’t think “terminate” is the right name. Maybe “cancel”? Or even 
“shutdown(wait=whatever, cancel=True)?”

I think Java inspired this library, so maybe it’s worth looking at what they 
do. But IIRC, it’s a much more complicated API in general, and for this case 
you’d actually have to do something like this:

x.shutdown() # stop new submissions
x.cancelAll() # cancel all tasks still in the queue
x.purge() # remove and handle all canceled tasks
x.join() # wait for already-started tasks to finish

… which probably isn’t what we want.

> I didn't actually meant terminate literally, I just called it that as that's 
> what multiprocessing.dummy.Pool.terminate (+ join after) does.

IIRC, it only does that by accident, because dummy.Process.terminate is a 
no-op, and that isn’t documented but just happens to be what CPython does.


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LZEKK6EWMPMBNGJSVZ4YGGAW5Y6OTV6G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Guido van Rossum]

On Fri, Jan 3, 2020 at 3:28 PM Miguel Ángel Prosper <
miguelangel.pros...@gmail.com> wrote:

> > gets one item from the queue, runs it, and then checks if the executor
> is being shut down.
>
> That's exactly what I thought at first, but just after that the continue
> statement prevents that check, so all futures always get processed. Only
> when the sentinel is reached, which it's placed at the end, it actually
> does the check.
>

Keen eye!


> > perhaps we can add some functionality where leftover work items are
> explicitly cancelled? I think that would satisfy the OP's requirement.
>
> Yes that would be perfect, that way we could have a controlled but fast
> exit for certain scenarios, such as the ones listed in the first post.
>
> From what I examined the procedure would be very similar to the one
> followed when a initializer fails, both implementations clear all pending
> futures when it happens. On ThreadPoolExecutor specifically, just calling a
> modified version of _initializer_failed at the time discussed would seem to
> be sufficient, changing the call Future.set_exception to Future.cancel.
>

OK, since the current behavior of shutdown() is "disallow submitting new
futures and wait for all currently queued futures to be processed", we
definitely need a new flag.

It looks like you have a good handle on the code -- do you want to submit a
PR to GitHub to add such a parameter?

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/R5CMS6QGZK6CGRPRLML33SFPB3NJH534/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Miguel Ángel Prosper]

> gets one item from the queue, runs it, and then checks if the executor is 
> being shut down.

That's exactly what I thought at first, but just after that the continue 
statement prevents that check, so all futures always get processed. Only when 
the sentinel is reached, which it's placed at the end, it actually does the 
check.

> perhaps we can add some functionality where leftover work items are 
> explicitly cancelled? I think that would satisfy the OP's requirement.

Yes that would be perfect, that way we could have a controlled but fast exit 
for certain scenarios, such as the ones listed in the first post.

>From what I examined the procedure would be very similar to the one followed 
>when a initializer fails, both implementations clear all pending futures when 
>it happens. On ThreadPoolExecutor specifically, just calling a modified 
>version of _initializer_failed at the time discussed would seem to be 
>sufficient, changing the call Future.set_exception to Future.cancel.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RAFIAFGTPYTWQX2ZXLRK3IWYMYYMF7HD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Target-and-expression lists and if-try

[David Mertz]

There have been a number of comments in this thread, and some slight
variations in the idea.

I'm afraid that it all looks awkward to me, so far. I understand trying to
work out the smallest change that would allow the style, but it just feels
"bolted on" in a bad way.

I believe that if Python gets pattern matching, it should either be full
F#-ish syntax *OR* be done as a library with a mini-language (which is not
actual Python syntax)

Would it be so bad to put something like the F# syntax in tripple quotes,
or perhaps in some nested data structure, then process it with a function
call? At least as something that could live on PyPI that could try to gain
users and see how much reaction the style gets.

On Tue, Dec 31, 2019, 5:30 PM Andrew Barnert via Python-ideas <
python-ideas@python.org> wrote:

> Every so often, someone suggests that Python should have a pattern
> matching case statement like Scala, C#, Swift, Haskell, etc. Or they just
> suggest that “Python needs algebraic data types” but end up concluding that
> the biggest part Python is missing is not the types themselves, but a
> practical way to do something useful with them, which basically means
> pattern matching.
>
> Anyway, with the addition of the walrus operator and @dataclass, Python is
> actually a lot closer than it used to be. I believe with just two smallish
> syntax features, it would be possible to do almost anything you’d ever want
> in a library.
>
> The first is to extend unpacking assignment to target-or-expression lists.
> Like this:
>
> x, 0, z = vec
>
> Just like `x, y, z = vec`, this raises a TypeError if vec isn’t Iterable,
> and a ValueError if it has more or fewer than 3 elements, and otherwise, it
> binds x and z to the first and third elements. But it doesn’t bind anything
> to the second element; instead, it checks if that element is 0, and, if
> not, raises a ValueError.
>
> The second is an “if try” statement, which tries an expression and runs
> the body if that doesn’t raise (instead of if it’s truthy), but jumps to
> the next elif/else/statement (swallowing the exception) if it does. The
> actual value of the expression is discarded, but the walrus operator takes
> care of that:
>
> if try 0, y, z := vec:
> # do yz plane stuff
> elif try x, 0, z := vec:
> # do xz plane stuff
> elif try x, y, 0 := vec:
> # do xy plane stuff
> elif x, y, z := vec:
> # do slow/imprecise/whatever 3D stuff
> else:
> raise TypeError(f'{vec} is not a 3-vector!')
>
> Alternatively, this could just be a try expression that can be used
> anywhere: it’s truthy if evaluating doesn’t raise, falsey if it does. But I
> don’t think it’s needed anywhere but if/elif.
>
> Anyway, neither of these features seems very useful on its own. (It should
> be obvious how to rewrite that example as something just as readable that
> just unpacks and then checks the values with normal if.)
>
> But I think the two of them together will allow a pure-library
> implementation of pattern matching syntax that reads nicely and can do
> everything most other languages do—in particular, concisely and readably
> pattern match and deconstruct dataclasses (and other types with an opt-in
> protocol or registry, but dataclasses would work out of the box the way
> Scala case classes, Swift structs, etc. do).
>
> If people think either of these features is too horrible to contemplate no
> matter what the potential benefits, I won’t bother working through the
> details.
>
> But if people do want to see the details (including some good motivating
> examples—I’ll go through tutorials for those other languages to find some
> that don’t involve recursing into cons lists and other stuff you wouldn’t
> want to do in Python…), I’ll be happy to do so, and, if it works out, turn
> this into a more detailed proposal.
>
> And obviously, if people want to bikeshed the spelling before even seeing
> what it’s good for, well, this is Python-ideas. :)
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/W64QFNZVAGU2EMHTLDXHRGJ7V7TYBR5R/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XSKEQK2Z7O463EZ7MPYTDBES3CMFP47B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Target-and-expression lists and if-try

[Neil Girdhar]

This is very cool, well thought-out, and solves an important problem.  I 
would have definitely benefited from Python having better pattern matching.

As an idealist though, if you're going to add pattern matching, why not 
just do it completely?  I agree that your proposal would be useful in some 
cases, but then what happens when someone wants to go just a step farther?  
For example,

def parse(text):
if try "(", *a, ")" = text[i, i+n]:   # how do you set n?  do you try 
them all?
parse(a)  # etc.

and even more complicated:

def parse(text):
   if try "\begin{", *a, "}", *b, "\end{", *a, "}" = ... # how do you 
ensure that the two a's match?  how do you ensure that a can only consist 
of letter characters?

I feel like it would be more idealistic to find a powerful, expressive way 
to specify parsers.  (As far as I know, no Python parser library can 
specify expression yet because they don't execute code on their "rules").  
If we had such a parsing library or extension, then we could compare 
whether it's overkill or reasonable to use that library in situations that 
you're proposing to address with "if try".

Best,

Neil


On Friday, January 3, 2020 at 12:15:42 PM UTC-5, Andrew Barnert via 
Python-ideas wrote:
>
> > On Jan 2, 2020, at 18:34, Dan Sommers <2qdxy4rz...@potatochowder.com 
> > wrote: 
> > 
> > On 1/2/20 3:07 PM, Random832 wrote: 
> >> On Thu, Jan 2, 2020, at 13:22, Dan Sommers wrote: 
> > 
> >>> What about "if except" (any time I can eliminate a "not," that's a 
> >>> good thing): 
> >> ... is this meant to be for the doesn't-throw case or does-throw? ... 
> >>> if except x: 
> >>> y 
> > 
> > That would be the does-throw case.  Read it as "if there's an exception 
> > evaluating x, then do the following." 
>
> But then you have the opposite of what you need. You want “if it matches 
> this, do that with these bindings”, not “if it doesn’t match this, do that 
> (with these failed bindings)”. 
>
> Also, you can’t chain these up and keep trying patterns until one 
> succeeds, or default if all of them fail; instead you can only chain them 
> up and keep trying patterns until one fails, or default if they all 
> succeed. 
>
> So, this syntax might be useful for other things, but not for pattern 
> matching. 
>
> >> ... how are we eliminating the not? 
> > 
> > I changed "not try" to "except," thus eliminating the "not." 
>
> But there’s no “not try”, just a “try”. We want to run code if the try 
> condition succeeds, which is positive, not negative. 
>
> Or, if you prefer, it’s a double negative, and you’ve removed one of the 
> negatives: 
>
>if try (x, 0, z) := vec: 
>xz_stuff(x, z) 
>
> … would be roughly equivalent to: 
>
>try: 
>x, 0, z = vec 
>except ValueError. 
>pass 
>else: 
>xz_stuff(x, z) 
>
> What you’ve given is a way to put code in the one place where we didn’t 
> want any (but had to put a “pass” just for syntactic reasons) without 
> putting code in the place we did. 
>
>
> ___
> Python-ideas mailing list -- python...@python.org 
> To unsubscribe send an email to python-id...@python.org 
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/5HOJJFQIQ25JZLPPGBOZTEMDDTAMGBSJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DC2OKRB6GKQHIIY65XEMHWGK4GKGEJ6L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Guido van Rossum]

Looking at the implementation in concurrent/futures/thread.py, it looks
like each of the worker threads repeatedly gets one item from the queue,
runs it, and then checks if the executor is being shut down. Worker threads
get added dynamically until the executor's max thread count is reached. New
futures cannot be submitted when the executor is being shut down. The
shutdown() method waits until all workers have cleanly exited.

So it looks like the only time when submitted futures are neither executed
nor cancelled is when there are more items in the work queue than there are
worker threads. In this situation the worker threads just exit, and the
unprocessed items will stay pending forever.

If I analyzed this correctly, perhaps we can add some functionality where
leftover work items are explicitly cancelled? I think that would satisfy
the OP's requirement. I *think* it would be safe to do this in shutdown()
after it has set self._shutdown but before it waits for the worker threads.

--Guido

On Fri, Jan 3, 2020 at 10:10 AM Miguel Ángel Prosper <
miguelangel.pros...@gmail.com> wrote:

> > Having a way to clear the queue and then shutdown once existing jobs are
> done is a lot
> > more manageable.
> ...
> > So the only clean way to do this is cooperative: flush the queue, send
> some kind of
> > message to all children telling them to finish as quickly as possible,
> then wait for them
> > to finish.
>
> I was personally thinking of an implementation like that, cancel all still
> in pending and if wait is true the wait for the ones running, for both
> implementations. I didn't actually meant terminate literally, I just called
> it that as that's what multiprocessing.dummy.Pool.terminate (+ join after)
> does.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/M62FHIZGAPCUSULLVV63NEWKQ2HWH6OY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3YK2TPYQF6VB4K73NLG2C3NCGV6VC422/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Miguel Ángel Prosper]

> Having a way to clear the queue and then shutdown once existing jobs are done 
> is a lot
> more manageable.
...
> So the only clean way to do this is cooperative: flush the queue, send some 
> kind of
> message to all children telling them to finish as quickly as possible, then 
> wait for them
> to finish.

I was personally thinking of an implementation like that, cancel all still in 
pending and if wait is true the wait for the ones running, for both 
implementations. I didn't actually meant terminate literally, I just called it 
that as that's what multiprocessing.dummy.Pool.terminate (+ join after) does.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/M62FHIZGAPCUSULLVV63NEWKQ2HWH6OY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Python, Be Bold!

[Andrew Barnert via Python-ideas]

Where’s the initial email you’re replying to here? I don’t have it in my inbox, 
and it isn’t on the Mailman archive either, and since you snipped it down to a 
single line I have no idea what that snippet was referring to.

Meanwhile:

> On Jan 2, 2020, at 18:14, James Lu  wrote:
> 
> 
> Ideally, we'd have a functional package manager: one that can delete binaries 
> when disk space is running low, and recompiles from sources when the binary 
> is needed again. It could store Python 2 as a series of diffs against Python 
> 3.

I’m not sure why you want to store Python 2 as diffs against Python 3, but that 
is exactly what a version control system does, not what a package manager does; 
it’s easy if you use the right tool. For example:

* Create an empty git repo in ~/python 
* Create a branch called 2.7
* Build Python 2.7 and install it to that directory, and also install any 
site-packages you want
* Commit
* Create a new branch off master called 3.8
* Build Python 3.8 and install it to that directory, and also install any 
site-packages you want
* Commit

Git (and its various plugins for diffing files) takes care of what’s a diff 
against what and so on; you just have to know there are at least two useful 
leaves on the graph of diffs and they can be accessed with the names 2.7 and 
3.8.

Now you have to checkout the appropriate branch every time you want to run 
python or python2, or activate a non-isolated virtual environments built from 
either, etc. Which seems like a huge pain, but it seems like you’re asking for 
that pain.

Would this actually save any space? I don’t know. I think 2.7 and 3.8 have so 
little binary code in common (bad enough for just the executable—things like 
stdlib .pyc files end up with different names in different locations that git 
probably isn’t smart enough to link up in the first place) that you might well 
waste more on repo overhead than you’d gain in deduplicating. (If you wanted 
3.8.0 and 3.8.1 and 3.8.2, on the other hand, that would probably save a lot 
more, but how often do you want an earlier bug fix build of the same minor 
version?)

But it’s simple enough to do that rather than guessing you can just try it.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/PQVDFDVH7NEQZK7YRAXTGBZ6NWDRCNSZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Andrew Barnert via Python-ideas]

On Jan 2, 2020, at 20:40, Miguel Ángel Prosper  
wrote:
> 
> I think it would be very helpful to have an additional argument (cancel for 
> example) added to Executor.shutdown that cancels all pending futures 
> submitted to the executor.

> Then context manager would gain the ability to abort all futures incase of a 
> exception, additionally this would also implement the missing cuterpart of 
> the multiprocessing module terminate, currently we only have close.

But terminate kills the process abruptly—skipping with/finally and atexit and 
internal interpreter cleanup, orphaning any children, etc. This is rarely what 
you want, especially not if you’re sharing a queue and a lock with the process.

And I think you can’t cleanly abort existing futures once you’ve done that; you 
have to mark them as being in an unknown state, not call any callbacks, and 
probably raise if the parent does anything but discard them uninspected.

Also, there’s no way to do the same thing for threads—and if you could, it 
would leave the process in a corrupted state.

Having a way to clear the queue and then shutdown once existing jobs are done 
is a lot more manageable. But it’s not terminate; there’s still a probably 
short but potentially unbounded wait for shutdown to finish. (Or, if you 
haven’t designed your jobs to be appropriately small, a probably _long_ wait.)

So the only clean way to do this is cooperative: flush the queue, send some 
kind of message to all children telling them to finish as quickly as possible, 
then wait for them to finish.

If you’re looking for a way to do a “graceful shutdown if possible but after N 
seconds just go down hard” the way, e.g., Apache or nginx does, I don’t think 
that could be safely done killing child threads in Python. (In C, you can write 
code to handle any emergency situation up to even having a trashed heap and 
stack if you’re careful, but in Python, you can’t do that; you can’t interpret 
a single Python instruction inside such an emergency.) But since you’re 
shutting down as quickly as possible after the N seconds are up, you can just 
daemonize the whole pool, do your emergency shutdown code, and exit or abort, 
after which the daemon threads get safely killed immediately.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SZSQS63IGHBATXBCS6L6GZQ56YN3GNZP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Python, Be Bold!

[Chris Angelico]

On Sat, Jan 4, 2020 at 1:16 AM Stephen J. Turnbull
 wrote:
> If you really want to save disk space measured in MB, I guess you'd
> probably want LRU semantics, and possibly a blacklist of modules that
> are only used interactively and at most once a day or so, so the time
> to import doesn't really matter.  It would be easy enough to write a
> script to do the cleaning (the stat module can get you the access
> times), and Python will take care of the recompilation since it only
> ever executes compiled code.
>

Be aware that access times aren't always accurate, since they depend
on the system performing a write every time you request a read. That's
an inefficiency that can easily be disabled (perhaps selectively - the
"relatime" mount option in Linux means to update the access time only
if the file's been changed since it was last accessed), at the price
of breaking the stats of something like you propose.

But honestly, if you're worried about a MB here and a MB there, you
probably will do better to zip up the stdlib than to fiddle around
with deleting stale modules.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/EK7MG22LF2ONRNUTDXNCD3XQEDMEMRBP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Python, Be Bold!

[Stephen J. Turnbull]

James Lu writes:

 > Ideally, we'd have a functional package manager: one that can
 > delete binaries when disk space is running low, and recompiles from
 > sources when the binary is needed again.

First, that's not what package managers do.  Package managers manage
dependencies, not disk space.

Second, my "where I get almost all my work done" Python 3.7
installation is only 571MB; I need about 20GB free to update XCode, so
disk space is not high on my reasons for screwing with a Python
installation.  Much more practical to search for old .isos and .dmgs
to delete.  If I really thought a few hundred MB would help, I'd just
delete the whole thing and hope nothing depended on it until I could
resolve the issue requiring more space than I have.

If you really want to save disk space measured in MB, I guess you'd
probably want LRU semantics, and possibly a blacklist of modules that
are only used interactively and at most once a day or so, so the time
to import doesn't really matter.  It would be easy enough to write a
script to do the cleaning (the stat module can get you the access
times), and Python will take care of the recompilation since it only
ever executes compiled code.

Although speaking practically, you might want to compile with -O and
delete all the sources if space is such a problem.  Teach help() to
call out to doc.python.org or/and pip the sources.

 > It could store Python 2 as a series of diffs against Python 3.

Even if you used a binary diff such as xdelta or bsdiff, I doubt this
would save space.  Feel free to try it and tell me I'm wrong, though.

Steve

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6SRPMEWVNAK7DQE7BDTDL7KFKN6CLWMS/
Code of Conduct: http://python.org/psf/codeofconduct/