[Python-ideas] Re: C API for converting Python integers to/from bytes sequences

[Kyle Stanley]

I lack the relevant experience to have an opinion on most of this, but FWIW
"PyLong_FromBytes/PyLong_ToBytes' seems clearest to me out of the options
proposed.

On Sat, Aug 7, 2021 at 2:23 PM Serhiy Storchaka  wrote:

> Python integers have arbitrary precision. For serialization and
> interpolation with other programs and libraries we need to represent
> them as fixed-width integers (little- and big-endian, signed and
> unsigned). In Python, we can use struct, array, memoryview and ctypes
> use for some standard sizes and int methods int.to_bytes and
> int.from_bytes for non-standard sizes. In C, there is the C API for
> converting to/from C types long, unsigned long, long long and unsigned
> long long. For other C types (signed and unsigned char, short, int) we
> need to use the C API for converting to long, and then truncate to the
> destination type with checking for overflow. For integers type aliases
> like pid_t we need to determine their size and signess and use
> corresponding C API or wrapper. For non-standard integers (e.g. 24-bit),
> integers wider than long long, and arbitrary precision integers all is
> much more complicated. There are private C API functions
> _PyLong_AsByteArray and _PyLong_FromByteArray, but they are for internal
> use only.
>
> I am planning to add public analogs of these private functions, but more
> powerful and convenient.
>
> PyObject *PyLong_FromBytes(const void *buf, Py_ssize_t size,
>int byteorder, int signed)
>
> Py_ssize_t PyLong_AsBytes(PyObject *o, void *buf, Py_ssize_t n,
>   int byteorder, int signed, int *overflow)
>
> PyLong_FromBytes() returns the int object. It only fails in case of
> memory error or incorrect arguments (e.g. buf is NULL).
>
> PyLong_AsBytes() writes bytes to the specified buffer, it does not
> allocate memory. If buf is NULL it returns the minimum size of the
> buffer for representing the integer. -1 is returned on error. if
> overflow is NULL, then OverfowError is raised, otherwise *overflow is
> set to +1 for overflowing the upper limit, -1 for overflowing the lower
> limit, and 0 for no overflow.
>
> Now I have some design questions.
>
> 1. How to encode the byte order?
>
> a) 1 -- little endian, 0 -- big endian
> b) 0 -- little endian, 1 -- big endian
> c) -1 -- little endian, +1 -- big endian, 0 -- native endian.
>
> Do we need to reserve some values for mixed endians?
>
> 2. How to specify the reduction modulo 2**(8*size) (like in
> PyLong_AsUnsignedLongMask)?
>
> Add yet one flag in PyLong_AsBytes()? Use special value for the signed
> argument? 0 -- unsigned, 1 -- signed, 2 (or -1) -- modulo. Or use some
> combination of signed and overflow?
>
> 3. How to specify saturation (like in PyNumber_AsSsize_t())? I.e. values
> less than the lower limit are replaced with the lower limit, values
> greater than the upper limit are replaced with the upper limit.
>
> Same options as for (2): separate flag, encode in signed (but we need
> two values here) or combination of other parameters.
>
> 4. What exact names to use?
>
> PyLong_FromByteArray/PyLong_AsByteArray,
> PyLong_FromBytes/PyLong_AsBytes, PyLong_FromBytes/PyLong_ToBytes?
>
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/V2EKXMKSQV25BMRPMDH47IM2OYCLY2TF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VVEKHEHP27I453YRW46T3HPJMTZFXLQT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures.ProcessPoolExecutor(restart_workers=True)

[Kyle Stanley]

 On Sat, Jun 26, 2021 at 11:59 AM Ram Rachum  wrote:

> Do you think it's worthwhile to add this feature to `ProcessPoolExecutor`?
>

*restart_workers* certainly seems worth further investigation and
consideration. However, you may also be able to find a workaround using the
*initialzer* argument for ProcessPoolExecutor. A detailed description is
found in the docs for ThreadPoolExecutor:

*initializer* is an optional callable that is called at the start of each
worker thread; *initargs* is a tuple of arguments passed to the
initializer.

Not certain and could be decently involved, but figured it may be a lead
worth sharing. :)

With loving-kindness,

On Sat, Jun 26, 2021 at 11:59 AM Ram Rachum  wrote:

> Hi guys,
>
> I want to have a version of `concurrent.futures.ProcessPoolExecutor` in
> which every worker process shuts down after each task. I want this because
> I believe I have a memory leak in my program that I wasn't able to fix.
> (Possibly it's in C extensions that I use.)
>
> Since I'm going to use a process pool anyway, I'm hoping to sidestep this
> memory leak by launching every task on a worker process that shuts down
> after it's done.
>
> A few years back when I was working at Dell, they did something similar.
> They had a Python server that had a memory leak, and they worked around it
> by having a process that can be easily restarted, and then they restarted
> it often.
>
> The plan is that each new work item will run on a fresh process that has
> no memory leaks from previous work items.
>
> I looked at the `futures/process.py` file to see whether I could subclass
> `ProcessPoolExecutor` and add this functionality, and boy oh boy this is
> not easy.
>
> Do you think it's worthwhile to add this feature to `ProcessPoolExecutor`?
>
>
> Thanks,
> Ram.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/UNDY7YKBJ4DOB6H45N7T6QK7PVJBIKZV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Kyle R. Stanley, Python Core Developer (what is a core dev?
)
*Pronouns: they/them **(why is my pronoun here?*

)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/S5EFXCG5XQJ7Y6GWB6ZCCHHNQA6SYQIM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Naming Accepted PEPs as PAPs

[Kyle Stanley]

On Mon, Sep 21, 2020 at 4:04 PM Abdur-Rahmaan Janhangeer <
arj.pyt...@gmail.com> wrote:

> This list is about discussing the idea,
> not only stating an opinion. I know
> i am not endearing people to me but i
> prefer threads to remain short and on
> topic which helps projects like these:
> https://abdur-rahmaanj.github.io/pyfaq
> The lists make great finds but it's more
> difficult when many mails which don't
> bring value to the discussion are present
>

Having some number of +1/-1s are useful in python-ideas threads, especially
coming from those with significant relevant experience or have some
influence in whether the proposal will be eventually accepted (such as core
devs, who could potentially sponsor the eventual PEP). In-depth feedback
contributes more to the topic, but without those +1/-1s, it's harder to
assess if the idea has gained enough traction to justify turning it into a
PEP (or simply implementing a feature).

Sorry if that adds to the volume of messages to sort through, but that's
just simply how python-ideas works.

FWIW, I'm personally against the proposal because it's trivial to look up
the PEP to figure out those additional details, which you would likely have
to do anyways if you weren't familiar with that PEP. It's not worth a
significant change in convention that would require a significant mental
shift from practically the entire Python community. It would require a
conscious effort to check if the PEP was accepted prior to deciding whether
to use "PEP" or "PAP", as well as the initial learning to figure out "WTF
is a PAP?". For an individual it would be a small cost, but if you were to
consider that small cost replicated for all of the millions of Python users
and thousands of reference materials that would need to be updated to
address "PAP", it doesn't seem worth the marginal at best benefit.

Regards,
Kyle Stanley
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/J77F3DVSVQPSLTNLNXSCQOYBJRVMQZNH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Else assert statement

[Kyle Stanley]

-1. Assertions are ignored when Python is run with -O, so we definitely
don't want to encourage relying on asserts to ensure users pass the correct
value. But even if that wasn't an issue, I consider the `else:
ValueError()` to be significantly more readable.

Otherwise, the alternative offered by Steve Barnes of using a dict for the
input options works well when you have a large number of possible inputs
with entirely different behaviors. Although I'd say for anything with
around 5 or less options, the `else: ValueError()` makes more sense for its
simplicity.

On Thu, Jul 2, 2020 at 7:43 PM Artemis  wrote:

> Often, I know that a variable should have one of a set of values, and I
> want to determine which one, with an if/elif/else clause. This looks
> something like this:
> ```
> if foo == 1:
> # do a
> elif foo == 2:
> # do b
> elif foo == 3:
> # do c
> else:
> raise ValueError('foo must be 1, 2 or 3')
> ```
> Sometimes, I just do
> ```
> if foo == 1:
> # do a
> elif foo == 2:
> # do b
> else:
> # do c
> ```
> But this is less readable and allows errors to slip past. My proposal is
> to allow the following syntax:
> ```
> if foo == 1:
> # do a
> elif foo == 2:
> # do b
> else foo == 3:
> # do c
> ```
> Or perhaps the more readable version:
> ```
> if foo == 1:
> # do a
> elif foo == 2:
> # do b
> else assert foo == 3:
> # do c
> ```
> Both of these options are backward compatible, since currently nothing is
> allowed between the `else` and the `:`.
> This would be roughly equivalent to
> ```
> if foo == 1:
> # do a
> elif foo == 2:
> # do b
> else:
> assert foo == 3
> # do c
> ```
> But shorter and more readable, since it puts the assertion at the same
> levels as the others. Thoughts?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/ZKAQK7YTOR2GVVFZFVO3U22WYJHVC3KE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3DFZ24QV36BG644KGTQZWPFNYDK7CJRO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Amend PEP-8 to require clear, understandable comments instead of Strunk & White Standard English comments

[Kyle Stanley]

I can't say that I've ever assessed the quality of a code comment
based on how well it adheres to Strunk & White, nor have I ever been
involved with an environment that tries to strongly enforce that
specific style of writing. So FWIW, I agree that "clear and concise
English" is more relevant to most.

On Fri, Jun 26, 2020 at 9:49 PM Guido van Rossum  wrote:
>
> Steven just likes an argument. Nobody has ever taken the idea of a standard 
> for language in comments seriously. It Just doen come up.
>
> On Fri, Jun 26, 2020 at 18:35 Bernardo Sulzbach 
>  wrote:
>>
>> On Fri, Jun 26, 2020 at 9:43 PM Steven D'Aprano  wrote:
>>>
>>> I dislike Strunk and White, and don't follow it myself (except by
>>> accident, as it were) but I've worked with neuro-atypical programmers
>>> who found it really useful to have a common standard that they could
>>> follow and reduce the uncertainty of what makes for good writing.
>>
>>
>> I agree.
>>
>> I would like to reinforce that many software engineers today are _not_ 
>> native English speakers but work with English codebases and write 
>> documentation in English.
>>
>> Any standard is better than just asking for "clear and easily 
>> understandable" and hoping that for some reason every person, no matter 
>> where they were born and how they learned English, will share your 
>> definition of "clear and easily understandable" by accident.
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-ideas@python.org/message/3MOO4LSWEAOAULDPN4LBHQWIHIRR7U5K/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
> --Guido (mobile)
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/XG4J5U4OVV752TXC66KS7OCVF7XD6TP2/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/Q6YTS77EORYS4MLDB22O4XREO7ZGWMR3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: asyncio: return from multiple coroutines

[Kyle Stanley]

I believe asyncio.wait() with "return_when=FIRST_COMPLETED" would
perform the functionality you're looking for with the
"asyncio.on_first_return()". For details on the functionality of
asyncio.wait(), see
https://docs.python.org/3/library/asyncio-task.html#asyncio.wait.

> I understand that I can create two coroutines that call the same function, 
> but it would be much cleaner (because of implementation issues) if I can 
> simply create a coroutine that yields the result of whichever connection 
> arrives first.

You can use an asynchronous generator that will continuously yield the
result of the first recv() that finishes (I'm assuming you mean
"yields" literally and want multiple results from a generator, but I
might be misinterpreting that part).

Here's a brief example, using the recv() coroutine function from the
pastebin linked:

```
import asyncio
import random

async def recv(message: str, max_sleep: int):
sleep_time = max_sleep * random.random()
await asyncio.sleep(sleep_time)
return f'{message} awaited for {sleep_time:.2f}s'

async def _start():
while True:
msgs = [
asyncio.create_task(recv("Messager 1", max_sleep=1)),
asyncio.create_task(recv("Messager 2", max_sleep=1))
]
done, _ = await asyncio.wait(msgs,
return_when=asyncio.FIRST_COMPLETED)
result = done.pop()
yield await result

async def main():
async for result in _start():
print(result)

asyncio.run(main())
```

Note that in the above example, in "msgs", you can technically pass
the coroutine objects directly to asyncio.wait(), as they will be
implicitly converted to tasks. However, we decided to deprecate that
functionality in Python 3.8 since it can be rather confusing. So
creating and passing the tasks is a better practice.

> Again, it's quite likely I am not seeing something obvious, but I didn't know 
> where else to ask.

If you're not mostly certain or relatively inexperienced with the
specific area that the question pertains to, I'd recommend asking on
python-list first (or another Python user community). python-ideas is
primarily intended for new feature proposals/suggestions. Although if
you've tried other resources and haven't found an answer, it's
perfectly fine to ask a question as part of the suggestion post.


On Mon, Jun 22, 2020 at 6:24 PM Pablo Alcain  wrote:
>
> Hey everyone. I have been looking into asyncio lately, and even though I have 
> had my fair share of work, I still have some of it very shaky, so first of 
> all forgive me if what I am saying here is already implemented and I totally 
> missed it (so far, it looks *really* likely).
>
> Basically this is the situation: I have an application that listens on two 
> websockets through the async library https://websockets.readthedocs.io/ and I 
> have to perform the same function on the result, no matter where the message 
> came from. I understand that I can create two coroutines that call the same 
> function, but it would be much cleaner (because of implementation issues) if 
> I can simply create a coroutine that yields the result of whichever 
> connection arrives first.
>
> I have implemented a rather cumbersome solution with async Queues (as I would 
> do in threading), in which each connection puts its message in a queue and an 
> adapter class awaits the first element of the queue on "receive". Here I 
> attach a pastebin with the minimal working example: 
> https://pastebin.com/BzaxRbtF
>
> However, it looks like a more async-friendly solution should exist, something 
> like
>
> ```
> async def _start():
> msg1 = recv("Messager 1", sleep_time=1)
> msg2 = recv("Messager 2", sleep_time=2)
> while True:
> result = await asyncio.on_first_return(msg1, msg2)
> print(result)
> ```
>
> (I understand that this implementation would not work because the event loop 
> doesn't know that it is "the same task repeated", but it's just to tell you 
> the general idea)
>
> Again, it's quite likely I am not seeing something obvious, but I didn't know 
> where else to ask.
>
> Thank you very much,
> Pablo
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/XBR5QPXRBCCJELDVEWMKRBPTNG4SJM64/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TFLM7AZWUWZKZIVJBSTQOIDFW7V3VSCV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: EVENTFD(2) support

[Kyle Stanley]

> Perfect. What's the procedure now for getting that PR accepted?

Next, it will go through a PR review process prior to being included for
Python 3.10. Since it's being implemented as a wrapper for an existing OS
API, a PR and bpo issue should be adequate (as opposed to major change
proposals that typically require a PEP and/or in-depth discussion.)

For more details on our GitHub PR process, see the following section from
the devguide: https://devguide.python.org/pullrequest/.

On Wed, Jun 17, 2020 at 6:58 AM  wrote:

> > I opened a BPO and created https://github.com/python/cpython/pull/20930
> > to implement a low-level interface to glibc's eventfd() function.
>
> Perfect. What's the procedure now for getting that PR accepted?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/ZLHYEXUGRD5BOPYPFOKKW7JB6W2IREAU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IVVGIMNXFZGXGDGLOMN7R3MAEUG4SAOI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: EVENTFD(2) support

[Kyle Stanley]

> Instead of trying to fix the PyPi version, I can write a proof of concept
of what I described above?

That would definitely be useful, and would help us get a more solid idea of
what exactly is being proposed. I also agree that a potential version
contained in the select module should be as simple as possible and rather
low-level (more so than the linked PyPI project), as it's likely going to
be there primarily for building other tools with. There's no guarantee of
course that it will ultimately be accepted, but a proof of concept
implementation would be a highly productive first step, IMO.

On Tue, Jun 16, 2020 at 2:58 PM  wrote:

> I've looked over the PyPi version of `eventfd`, and I feel that it is
> trying to be more than just a thin wrapper. The attempt to make it cross
> platform has given it more fields than it requires, as has the attempt to
> wrap it as a semaphore, and with that comes added complexity.
>
> The `eventfd` that I was suggesting be added would in my mind look
> something more like this:
>
> `eventfd`:
> - `__init__(...)`
> - `read(...)`
> - `write(...)`
> - `close(...)`
> - `closed(...)`
> - `fileno(...)`
> + some extra dunder methods
>
> We could possibly add a method to get the flags that were passed to
> `__init__`, and I think pickling should be disabled (while it does get
> inherited through `fork(2)` and likely through `execve(2)` depending on
> flags, I think the fd's integer value in each process can be different,
> though I could be wrong here).
>
> From that, one could then build more complex classes, but the base
> primitive should be as simple as possible.
> Instead of trying to fix the PyPi version, I can write a proof of concept
> of what I described above?
>
> doodspav
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/62XBNINQ5HCUQVGJ6UZX4PM67QRYECUA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WNHAP6DCPDAJVVNO3FHXQZDDLWQ45ACT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: EVENTFD(2) support

[Kyle Stanley]

> It would only be available on the Linux platform, and one of the benefits
would be the ability to create synchronisation primitives on said platform
that can block on normal threads, and be awaited on in coroutines (without
busy looping inside said coroutine).

+1. Although it already exists as a package on PyPI, I've seen a
non-negligible amount of user demand for a synchronization primitive that
works as described above for asyncio. Adding eventfd() to the stdlib seems
like a step towards eventually implementing something
like that in an effective manner, and I don't think the maintenance burden
would be especially high. I could also see a significant potential use case
for multiprocessing (CC'd Antoine in case he has anything to add there).

> Currently the best place I can think of to put it would be in one of the
Networking and Interprocess Communication modules (possibly `select` or
`socket`?).

The select module seems like it would be a reasonable location.

On Tue, Jun 16, 2020 at 8:42 AM  wrote:

> Can we implement eventfd(2) as documented here <
> https://man7.org/linux/man-pages/man2/eventfd.2.html>?
>
> It would only be available on the Linux platform, and one of the benefits
> would be the ability to create synchronisation primitives on said platform
> that can block on normal threads, and be awaited on in coroutines (without
> busy looping inside said coroutine).
>
> Currently the best place I can think of to put it would be in one of the
> Networking and Interprocess Communication modules (possibly `select` or
> `socket`?). The fact that it's Linux only shouldn't be an issue, since much
> of the contents of `select` is OS dependent.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/2BX6V4NB24QFXAPXAYQVKGG72SKAF3YF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/NEPEG5UQ2ZRQNODQM6NKXW77E7XWOYZI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Python WebAssembly Support

[Kyle Stanley]

> But my question is why not to apply all this patches to `CPython` to be
able to compile `CPython` on the Web ?

> Patches to CPython seems pretty straightforward ...

In order for an existing project to even be considered as a patch for
CPython as an addition to the standard library, it typically requires
sufficient maturity, a robust API, and thorough tests. There's also a
formal PEP process that most major changes have to go through, along with
taking on experts that are willing to maintain the new code and
documentation for many years. That doesn't even include the discussion of
whether something like that fits in the stdlib in general, or other
considerations such as security (which has been an ongoing issue for WASM).

So, simply because an implementation exists does not mean it should be
immediately included in CPython. There are a multitude of important factors
to consider, especially for something this broad.

On Mon, Jun 15, 2020 at 5:04 PM  wrote:

> Thanks, but I know about this implementation ...
>
> But my question is why not to apply all this patches to `CPython` to be
> able to compile `CPython` on the Web ?
> https://github.com/iodide-project/pyodide
> https://github.com/dgym/cpython-emscripten
>
> Patches to CPython seems pretty straightforward ...
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/GIQ7RREZAAHO2S6XWVEPN6SDIVXP7GMB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LMYMNPCPJ4BFXIXJWF6XQUZR6M53T3FQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: await by default

[Kyle Stanley]

> for any IO-bound call with a variable time where async isn't an option
(either because it's not available, standardized, widespread, etc.), I'd
advise using loop.run_in_executor()/to_thread() preemptively.

Clarification: this pretty much applies to any non-async IO-bound call that
can block the event loop. You can definitely get away with ignoring some
that have a consistently negligible duration, but I would not *directly*
call any of them that could vary significantly in time (or are consistently
long running) within a coroutine. Otherwise, it's a complete gamble as to
how long it stalls the rest of the program, which is generally not
desirable to say the least.

On Sun, Jun 14, 2020 at 1:42 AM Kyle Stanley  wrote:

> > IOW the solution to the problem is to use threads. You can see here
> why I said what I did: threads specifically avoid this problem and the
> only way for asyncio to avoid it is to use threads.
>
> In the case of the above example, I'd say it's more so "use coroutines by
> default and threads as needed" rather than just using threads, but fair
> enough. I'll concede that point.
>
> > For instance, maybe during testing (with debug=True), your
> DNS lookups are always reasonably fast, but then some time after
> deployment, you find that they're stalling you out. How much effort is
> it to change this over? How many other things are going to be slow,
> and can you find them all?
>
> That's very situationally dependent, but for any IO-bound call with a
> variable time where async isn't an option (either because it's not
> available, standardized, widespread, etc.), I'd advise using
> loop.run_in_executor()/to_thread() preemptively. This is easier said than
> done of course and it's very possible for some to be glossed over. If it's
> missed though, I don't think it's too much effort to change it over; IMO
> the main challenge is more so with locating all of them in production for a
> large, existing codebase.
>
> > 3) Steven D'Aprano is terrified of them and will rail on you for using
> threads.
>
> Haha, I've somehow completely missed that. I CC'd Steven in the response,
> since I'm curious as to what he has to say about that.
>
> > Take your pick. Figure out what your task needs. Both exist for good
> reasons.
>
> Completely agreed, threads and coroutines are two completely different
> approaches, with neither one being clearly superior for all situations.
> Even as someone who's invested a significant amount of time in helping to
> improve asyncio recently, I'll admit that I decently often encounter users
> that would be better off using threads. Particularly for code that isn't
> performance or resource critical, or when it involves a reasonably small
> number of concurrent operations that aren't expected to scale in volume
> significantly. The fine-grained control over context switching (which can
> be a pro or a con), shorter switch delay, and lower resource usage from
> coroutines isn't always worth the added code complexity.
>
>
>
> On Sun, Jun 14, 2020 at 12:43 AM Chris Angelico  wrote:
>
>> On Sun, Jun 14, 2020 at 2:16 PM Kyle Stanley  wrote:
>> >
>> > > If
>> > you're fine with invisible context switches, you're probably better
>> > off with threads, because they're not vulnerable to unexpectedly
>> > blocking actions (a common culprit being name lookups before network
>> > transactions - you can connect sockets asynchronously, but
>> > gethostbyname will block the current thread).
>> >
>> > These "unexpectedly blocking actions" can be identified in asyncio's
>> debug mode. Specifically, any callback or task step that has a duration
>> greater than 100ms will be logged. Then, the user can take a closer look at
>> the offending long running step. If it's like socket.gethostbyname() and is
>> a blocking IO-bound function call, it can be executed in a thread pool
>> using loop.run_in_executor(None, socket.gethostbyname, hostname) to avoid
>> blocking the event loop. In 3.9, there's also a roughly equivalent
>> higher-level function that doesn't require access to the event loop:
>> asyncio.to_thread(socket.gethostbyname, hostname).
>> >
>> > With the default duration of 100ms, it likely wouldn't pick up on
>> socket.gethostbyname(), but it can rather easily be adjusted via the
>> modifiable loop.slow_callback_duration attribute.
>> >
>> > Here's a quick, trivial example:
>> > ```
>> > import asyncio
>> > import socket
>> >
>> > async def main():
>> > loop = asyncio.get_running_loop()
>> > loop.slow_callback_duration = .01 # 10ms
>> > socket.

[Python-ideas] Re: await by default

[Kyle Stanley]

> IOW the solution to the problem is to use threads. You can see here
why I said what I did: threads specifically avoid this problem and the
only way for asyncio to avoid it is to use threads.

In the case of the above example, I'd say it's more so "use coroutines by
default and threads as needed" rather than just using threads, but fair
enough. I'll concede that point.

> For instance, maybe during testing (with debug=True), your
DNS lookups are always reasonably fast, but then some time after
deployment, you find that they're stalling you out. How much effort is
it to change this over? How many other things are going to be slow,
and can you find them all?

That's very situationally dependent, but for any IO-bound call with a
variable time where async isn't an option (either because it's not
available, standardized, widespread, etc.), I'd advise using
loop.run_in_executor()/to_thread() preemptively. This is easier said than
done of course and it's very possible for some to be glossed over. If it's
missed though, I don't think it's too much effort to change it over; IMO
the main challenge is more so with locating all of them in production for a
large, existing codebase.

> 3) Steven D'Aprano is terrified of them and will rail on you for using
threads.

Haha, I've somehow completely missed that. I CC'd Steven in the response,
since I'm curious as to what he has to say about that.

> Take your pick. Figure out what your task needs. Both exist for good
reasons.

Completely agreed, threads and coroutines are two completely different
approaches, with neither one being clearly superior for all situations.
Even as someone who's invested a significant amount of time in helping to
improve asyncio recently, I'll admit that I decently often encounter users
that would be better off using threads. Particularly for code that isn't
performance or resource critical, or when it involves a reasonably small
number of concurrent operations that aren't expected to scale in volume
significantly. The fine-grained control over context switching (which can
be a pro or a con), shorter switch delay, and lower resource usage from
coroutines isn't always worth the added code complexity.



On Sun, Jun 14, 2020 at 12:43 AM Chris Angelico  wrote:

> On Sun, Jun 14, 2020 at 2:16 PM Kyle Stanley  wrote:
> >
> > > If
> > you're fine with invisible context switches, you're probably better
> > off with threads, because they're not vulnerable to unexpectedly
> > blocking actions (a common culprit being name lookups before network
> > transactions - you can connect sockets asynchronously, but
> > gethostbyname will block the current thread).
> >
> > These "unexpectedly blocking actions" can be identified in asyncio's
> debug mode. Specifically, any callback or task step that has a duration
> greater than 100ms will be logged. Then, the user can take a closer look at
> the offending long running step. If it's like socket.gethostbyname() and is
> a blocking IO-bound function call, it can be executed in a thread pool
> using loop.run_in_executor(None, socket.gethostbyname, hostname) to avoid
> blocking the event loop. In 3.9, there's also a roughly equivalent
> higher-level function that doesn't require access to the event loop:
> asyncio.to_thread(socket.gethostbyname, hostname).
> >
> > With the default duration of 100ms, it likely wouldn't pick up on
> socket.gethostbyname(), but it can rather easily be adjusted via the
> modifiable loop.slow_callback_duration attribute.
> >
> > Here's a quick, trivial example:
> > ```
> > import asyncio
> > import socket
> >
> > async def main():
> > loop = asyncio.get_running_loop()
> > loop.slow_callback_duration = .01 # 10ms
> > socket.gethostbyname("python.org")
> >
> > asyncio.run(main(), debug=True)
> > # If asyncio.run() is not an option, it can also be enabled via:
> > # loop.set_debug()
> > # using -X dev
> > # PYTHONASYNCIODEBUG env var
> > ```
> > Output (3.8.3):
> > Executing  asyncio_debug_ex.py:5> result=None created at
> /usr/lib/python3.8/asyncio/base_events.py:595> took 0.039 seconds
> >
> > This is a bit more involved than it is for working with threads; I just
> wanted to demonstrate one method of addressing the problem, as it's a
> decently common issue. For more details about asyncio's debug mode, see
> https://docs.python.org/3/library/asyncio-dev.html#debug-mode.
> >
>
> IOW the solution to the problem is to use threads. You can see here
> why I said what I did: threads specifically avoid this problem and the
> only way for asyncio to avoid it is to use threads. (Yes, you can
> asynchronously do a DNS lookup rather than using gethostbyname, but
> the sema

[Python-ideas] Re: await by default

[Kyle Stanley]

> If
you're fine with invisible context switches, you're probably better
off with threads, because they're not vulnerable to unexpectedly
blocking actions (a common culprit being name lookups before network
transactions - you can connect sockets asynchronously, but
gethostbyname will block the current thread).

These "unexpectedly blocking actions" can be identified in asyncio's debug
mode. Specifically, any callback or task step that has a duration greater
than 100ms will be logged. Then, the user can take a closer look at the
offending long running step. If it's like socket.gethostbyname() and is a
blocking IO-bound function call, it can be executed in a thread pool using
loop.run_in_executor(None, socket.gethostbyname, hostname) to avoid
blocking the event loop. In 3.9, there's also a roughly equivalent
higher-level function that doesn't require access to the event loop:
asyncio.to_thread(socket.gethostbyname, hostname).

With the default duration of 100ms, it likely wouldn't pick up on
socket.gethostbyname(), but it can rather easily be adjusted via the
modifiable loop.slow_callback_duration attribute.

Here's a quick, trivial example:
```
import asyncio
import socket

async def main():
loop = asyncio.get_running_loop()
loop.slow_callback_duration = .01 # 10ms
socket.gethostbyname("python.org")

asyncio.run(main(), debug=True)
# If asyncio.run() is not an option, it can also be enabled via:
# loop.set_debug()
# using -X dev
# PYTHONASYNCIODEBUG env var
```
Output (3.8.3):
Executing  result=None created at
/usr/lib/python3.8/asyncio/base_events.py:595> took 0.039 seconds

This is a bit more involved than it is for working with threads; I just
wanted to demonstrate one method of addressing the problem, as it's a
decently common issue. For more details about asyncio's debug mode, see
https://docs.python.org/3/library/asyncio-dev.html#debug-mode.


On Sat, Jun 13, 2020 at 9:44 PM Chris Angelico  wrote:

> On Sun, Jun 14, 2020 at 11:29 AM Greg Ewing 
> wrote:
> >
> > On 14/06/20 12:05 pm, Chris Angelico wrote:
> > > On Sun, Jun 14, 2020 at 9:54 AM Greg Ewing <
> greg.ew...@canterbury.ac.nz> wrote:
> > >> Likewise, it's legitimate to create an awaitable
> > >> object and then await it later.
> > >>
> > >> (Personally I think it *shouldn't* be legitimate to do that in
> > >> the case of await, but Guido thinks otherwise, so it is the way
> > >> it is.)
> > >
> > > If it isn't, then how do you start multiple tasks in parallel?
> >
> > There would need to be a primitive that takes an async def function
> > and creates an awaitable from it. The API for spawning tasks would
> > then take an async function and use this primitive to get things
> > rolling. So it wouldn't be impossible to separate the two, but you
> > would have to go out of your way to do it. It wouldn't be the usual
> > way to do things.
> >
> > (For more on this, look up the discussions about my "cofunctions"
> > idea. It was very similar to async/await, except that the operations
> > of calling an async function and awaiting the result were fused into
> > a single syntactic entity.)
> >
>
> Hmm, I think I see what you mean. So awaiting it would be "spam(123)"
> and getting an awaitable for later would be "spam.defer(123)"? That
> would make reasonable sense.
>
> Still, I don't think it's of value, since part of the point of
> coroutines is knowing exactly where a context switch could happen. If
> you're fine with invisible context switches, you're probably better
> off with threads, because they're not vulnerable to unexpectedly
> blocking actions (a common culprit being name lookups before network
> transactions - you can connect sockets asynchronously, but
> gethostbyname will block the current thread).
>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/WDYCTCHG6NH36W5QS46NYEBOQVGUPOSR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UAF47XYNL6GBPSDKQYHJUZZ2DM6QYHQB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: await outside async function (was: Re: await outside async function could map to asyncio.run ?)

[Kyle Stanley]

> An await outside async def, would map to awaitable.run().

> A call to baz() would thus create a coroutine foo(), and then run() it.

-1. The `await` expression already has a well-defined purpose of suspending
the coroutine it's used within until the awaitable is completed. I think
this change in behavior would be rather confusing, and results in minimal
(if any) practical benefit since you can already accomplish the main
intended behavior of effectively running coroutines as blocking subroutines
(see next section).

> In particular, this could allow one to turn any arbitrary async coroutine
into a blocking subroutine, as long as it supports this run() protocol.
This would be great for refactoring existing blocking code into async -
just use stuff as if it's sync, then when you're done, switch up your
function with async def, and change relevant calls into it into "await"
calls. Rinse and repeat until you're ready to add a top-level event loop.

This can already be effectively accomplished by gradually converting the
synchronous code into coroutines and running them individually with
asyncio's loop.run_until_complete(coro()) (which blocks until finished)
from a subroutine while the other parts of the code are still transitioning
to async. However, the bulk of the effort is considering the architecture
of the program, specifically analyzing the parts that need to be completed
sequentially vs those that can be completed concurrently, and then figuring
out exactly when/where the results are needed. So, I don't think that
making an await expression outside of a coroutine function an alias to an
awaitable.run() and effectively doing the same thing as
loop.run_until_complete() would save substantial development time, and it
adds additional unneeded complexity to the await expression for little to
no practical benefit.

Also, I very much appreciate knowing that when I see "await" used, I'm
certain it's being called within a coroutine and more importantly, the
expected behavior. Having to check whether the function is awaited in is a
coroutine or a subroutine to determine the behavior could prove to be quite
cumbersome in non-trivial programs. A significant part of why await was
added in the first place was to avoid the ambiguity present with "yield
from". In general, I think there's a significant advantage in having a
singular purpose for a keyword, rather than changing based on where it's
used.

> Optionally, this would also allow us to deprecate blocking APIs (like
open(), etc

Huge -1 for deprecating the existing blocking APIs. Although converting
some IO-bound functions to asynchronous equivalents can yield significant
performance benefits (when structured optimally), it's not always
worthwhile to do so. Particularly in situations where the underlying OS
call doesn't have asynchronous support (frequently the case for async file
system operations), or when the intended use case for the program wouldn't
benefit from the performance improvements enough to justify the added
complexity and development time. So blocking IO-bound functions still have
a significant purpose, and should definitely __not__ be deprecated. Even if
there was a working automatic conversion process to place an "await" before
each of the calls, it would still cause a lot of breakage and add
additional noise to the code that wouldn't provide additional useful
context.

Also, consider the amount of times that various Python tutorials and
textbooks use open() that would all have to be rewritten because it no
longer works without "await" before it. Realistically speaking, I don't
think there's any chance of something as commonly used as open() being
deprecated, regardless how far out the removal would be. Even if/when async
file system support at the OS level becomes the mainstream (which it
currently isn't), I still don't think it would be worth breaking open().

> and asyncio, and merge them into one TBD API. Perhaps as part of Python 4.

Even if I agreed with the main proposal, I would still be against this
part. Stdlib module deprecation is a very long process that has to be
strongly justified, considering the amount of breakage that would occur
from its eventual removal. So even with the assumption that the proposal
would provide some real benefit in the transition from sync to async (which
I disagree with), I don't think this would come even close to justifying a
full deprecation of asyncio into a separate "TBD API".

Thanks for sharing the idea Soni, but I would personally be opposed to
every component of the proposal.

On Fri, Jun 12, 2020 at 7:20 PM Soni L.  wrote:

>
>
> On 2020-06-12 5:47 p.m., J. Pic wrote:
>
> Hi all,
>
> Currently, you can not use await outside an async function, the following
> code:
>
>   async def lol():
> return 'bar'
>
>   def test():
> return await lol()
>
>   print(test())
>
> Will fail with: SyntaxError: 'await' outside async function
>
> Of course, you can use asyncio.run and then it works fine:

[Python-ideas] Re: await by default

[Kyle Stanley]

While I agree that we should avoid telling people to "just use threads", I
think Chris was instead informing the OP that their desired behavior is
already present in threads, and if they don't want to be concerned at all
about context switching, OS threads should be considered as an alternative.
Of course, this doesn't take into account the lower memory usage of
coroutines, the shorter context switching delay, etc. But that's not
necessary for all use cases.

On Fri, Jun 12, 2020 at 10:11 PM Greg Ewing 
wrote:

> On 13/06/20 9:37 am, Chris Angelico wrote:
> > If you don't care where the context
> > switches happen and just want everything to behave sanely by default,
> > use threads, not coroutines.
>
> There are other reasons for using coroutines, such as the fact that
> they're very lightweight compared to threads. Telling people to
> "just use threads" without knowing more about their use case is
> not helpful.
>
> --
> Greg
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/QWUWDA7L6LOZR2VW7JIM2RYI7L7N5W2B/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3K54K7B5FKOHJX266EAYD6NN27E7OPLQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Making asyncio more thread-friendly

[Kyle Stanley]

> Could you be more specific about these tools?  They sound like they
may be what Celelibi is looking for.  Just a list of APIs (even just a
few core items) would likely point them at the right parts of the
doc.

Sorry if it wasn't clear, but by "interoperability tools", I was referring
to the API members mentioned in the second paragraph of my previous message:

loop.run_in_executor() (or the higher-level upcoming asyncio.to_thread() in
3.9)
loop.call_soon_threadsafe()
asyncio.run_coroutine_threadsafe()

I consider these to be interoperability tools, as they can be used to
bridge code that was previously built using an OS thread model with an
async/await coroutine model, without having to entirely rewrite it. E.g. if
some IO-bound function "func" was previously passed to the *target*
argument of threading.Thread(), you could pass it to
loop.run_in_executor(None, func, *args) or the upcoming
asyncio.to_thread(func, *args) instead. I'm using loop.run_in_executor() as
an example because that's the one I've used the most in my own code.

Of course, it would likely result in improved performance if it were to be
entirely rewritten instead of still using a thread, but this isn't always
possible. Especially in cases where the underlying OS call doesn't have
async support. Also, it can require a significant amount of time investment
when working with a large existing library, so using something like
loop.run_in_executor() allows it to be used and tested with the rest of the
async program while working on a larger rewrite.

I believe we're also going to be working on an asyncio.from_thread() in the
near future (3.10?), which would presumably be for executing a function in
a *specific* thread/event loop. AFAIK, there are still some details to work
out though, such as how the context will be specified. I'm not certain if
it would make more sense to use a unique token (Trio does this), or use an
event loop argument like we've done for other parts of asyncio. Personally,
I'm somewhat inclined towards something like a token or other option since
we're removing the event loop arg from most of the high-level API.

The OPs issue highlights a difficult balance from an API design
perspective. Having constant access to the event loop by using it to
call/schedule everything and being able to pass it around as an argument
makes it easier to access when working with multiple event loops (loops
outside of the main thread, specifically), but it also adds a significant
amount of additional boilerplate and room for error (such as attempting to
share async objects between multiple event loops).

While I think eventually removing the event loop arg is a good thing, we
may need to consider delaying the removal until we're certain that users
have access to a full array of tools for working with multiple event loops.
It was deprecated in 3.8 and scheduled for removal in 3.10, but that might
be a bit too soon (even more so with the release cadence being shortened
recently).

On Tue, Jun 9, 2020 at 11:09 PM Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Kyle Stanley writes:
>
>  > Fundamentally, OS threads and coroutines are two entirely different
>  > models of concurrency; though, we do have interoperability tools in
>  > place and are actively working on making them easier to
>  > utilize. Also, with adequate arguments for specific real-world use
>  > cases, those interoperability tools can be expanded upon as needed.
>
> Could you be more specific about these tools?  They sound like they
> may be what Celelibi is looking for.  Just a list of APIs (even just a
> few core items) would likely point them at the right parts of the
> doc. I'll try if you can't respond, but this is well outside my
> experience, and my ocean of round tuits is looking like the Sahara Sea
> lately.
>
> Steve
>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/MY6QQOQ7KWUZUCCHYMB7C7RBOQQQ2SAI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [Python-Dev] Making asyncio more thread-friendly

[Kyle Stanley]

I would like to clarify that asyncio *does* most certainly provide support
for programs that use multiple event loops and interact with a number of
different threads. So, asyncio can definitely be used outside of a purely
"single-process, single-threaded, single-event-loop
approach". However, a key point that should be followed is that __async
objects__ created in a single event loop are intended to be used only in
that event loop. If you try to go outside of the bounds of that, you'll
likely find yourself in the territory of undefined behavior (which is a
part of why we're deprecating the loop argument throughout the high-level
API, to discourage that).

The asyncio part of your program can use a thread to run a function and get
the result without blocking the event loop (using
loop.run_in_executor(None, func, *args) or the upcoming asyncio.to_thread()
in 3.9), schedule callbacks from another thread with
loop.call_soon_threadsafe(), or submit a coroutine to be run in a specific
event loop with asyncio.run_coroutine_threadsafe() from a different thread.
But it does require some thinking about the architecture of your program,
and can't just be used to replace an existing threaded program without any
considerations.

Fundamentally, OS threads and coroutines are two entirely different models
of concurrency; though, we do have interoperability tools in place and are
actively working on making them easier to utilize. Also, with adequate
arguments for specific real-world use cases, those interoperability tools
can be expanded upon as needed. However, these should be separate from
existing parts of asyncio rather than changing methods like
loop.run_until_complete(), IMO. There's something to be said about having
too much purpose crammed into a single API member -- it tends to make the
behavior needlessly complex and hard to follow.

On Tue, Jun 9, 2020 at 2:37 AM Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Fortunately, Guido has jumped in.  He's the authority on asyncio.  I
> recommend you consider his advice very carefully.  Here are a couple
> more commments that I had already written when I saw his post.
>
> Celelibi writes:
>
>  > But that doesn't mean we can't try to design asyncio to be more
>  > thread-friendly.
>
> True, lack of an example to emulate doesn't mean we can't try ... but
> the fact that asyncio and threads are based on completely different
> models of concurrency likely does, in combination with that lack.
> Strong suggestion that "here be dragons".
>
>  > I'm working on switching out a thread-based client protocol lib for
>  > an asyncio-based lib instead.
>
> Have you tried to do this with an alternative to asyncio such as curio
> (by David Beazley, IIRC) or trio (by Nathaniel Smith)?  They provide
> simpler APIs, and Nathaniel claims that they also provide a
> concurrency model that's far easier to understand.  They're still
> async def/await-based, but they have a property Nathaniel calls
> "respect for causality" that seems very useful in understanding
> programs.
>
>  > But if it did, I'd end up calling call_soon_threadsafe from the thread
>  > that runs the event loop and wait for the result. Which would just lock
>  > up.
>
> If I understand correctly, this is the kind of thing that "respect for
> causality" helps to identify and to some extent prevent.
>
>  > Another issue I found is that the client protocol lib class isn't always
>  > instanciated by the same thread, and its constructor indirectly
>  > instanciate asyncio objects (like Event).
>
> The relations among the threading library, the asyncio library, and
> the client protocol library in your software are quite unclear to me.
> But this sounds very bad for your "divide and conquer" strategy.
> Asyncio is fundamentally a single-process, single-threaded,
> single-event-loop approach to handling concurrency.  If the asyncio
> part of your program can ignore what other threads are doing, you
> might be OK, but this kind of coupling means you can't, and will need
> to do things like:
>
>  > In the end, its start method will start the event loop in its own
>  > thread.  With the deprecation of the loop argument, Event() finds
>  > the loop it should use by calling get_event_loop() in its
>  > constructor. I therefore had to add a call set_event_loop(), even
>  > though this event loop will never run in this thread. That's a
>  > pretty ugly hack.
>
> But it's intended to be a temporary one, no?  This is the kind of
> thing you need to do when you have multiple incompatible models of
> computation running in the same system.
>
>  > The deprecation of the loop arguments looks like an incentive to create
>  > the asyncio objects in the threads that use them.
>
> It is.  A major point (perhaps the whole point) of asyncio is to
> provide that "single-process, single-threaded, single-event-loop
> approach" which nevertheless is extremely fast and scalable for
> I/O-bound programs.
>
> If you can't get to the 

[Python-ideas] Re: An HTTP API to list versions of Python

[Kyle Stanley]

Assuming the release managers are on-board with the idea (since it would
impact them the most), I'm +1. Overall, it seems like it would help
significantly to reduce the total number of different locations to update
during the release process once we get it fully configured.

> It could become more detailed about each minor versions, git tag, links
to changelogs, links to the repositories, to the docs, download links,
and so on.

I don't know that it needs to be said, but for now, I think we should start
with a minimalist approach by keeping the API focused on reducing the
number of *existing* locations to update, rather than predicting what might
sort of fields might be useful to include. Otherwise, it could very well
end up becoming more work to maintain compared to what it actually saves.

On Tue, May 26, 2020 at 6:38 PM Julien Palard via Python-ideas <
python-ideas@python.org> wrote:

> We do not have a standard way to get (programatically) informations
> about Python versions, this leads to having to commit to numerous places
> during each releases, and some other places never get the information
> (like old docs, see [1]).
>
>
> Maybe we should provide an HTTP API listing Python versions, like:
>
> {
>  "current_stable": "3.8.3",
>  "versions": [
>  {
>  "branch": "master",
>  "status": "dev",
>  "version": "3.10",
>  "release_manager": "Łukasz Langa",
>  "doc_languages": ["en", "es", "fr", "ja", "ko", "pt-br",
> "zh-cn"]
>  },
>  {
>  "branch": "3.9",
>  "latest": "3.9.0b1",
>  "status": "features",
>  "pep": 596,
>  "version": "3.9",
>  "release_manager": "Łukasz Langa",
>  "doc_languages": ["en", "fr", "ja", "ko", "pt-br", "zh-cn"]
>  },
>  {
>  "branch": "3.8",
>  "latest": "3.8.3",
>  "status": "bugfix",
>  "pep": 569,
>  "version": "3.8",
>  "release_manager": "Łukasz Langa",
>  "first_release": "2019-10-14",
>  "end_life": "2024-10",
>  "doc_languages": ["en", "fr", "ja", "ko", "pt-br", "zh-cn"]
>  },
>  {
>  "branch": "3.7",
>  "latest": "3.7.7",
>  "status": "bugfix",
>  "pep": 537,
>  "version": "3.8",
>  "release_manager": "Ned Deily",
>  "first_release": "2018-06-27",
>  "end_life": "2023-06-27",
>  "doc_languages": ["en", "fr", "ja", "ko", "pt-br"]
>  }
>  ...
>
> (Don't really mind this particular example, it's a random first though,
> the doc_languages should probably be a dict instead, with links to the
> actual docs, ...)
>
> It could be used to (and maybe avoid a commit) to:
> - build the documentation version switcher and languages switchers
> - toggle the red deprecated banner (to be seen on [2] if you missed it)
> - generate the versions table at https://devguide.python.org/
> - generate the Docs by version menu
> - generate the Download menu and download page of python.org
> - list the documentations versions to be built by docsbuild-script
> - probably many scripts could use it too, I can think of [3]
> - ...
>
> It could become more detailed about each minor versions, git tag, links
> to changelogs, links to the repositories, to the docs, download links,
> and so on.
>
> Maybe this idea will just add a commit to do between the numerous
> commits to already do during a release... (à la https://xkcd.com/927/),
> and a SPOF.
>
> What do you think?
>
> [1]:
>
> https://mail.python.org/archives/list/d...@python.org/thread/L724YZXXHQZQP4HJYXCDCLYLBJB726D3/
> [2]: https://docs.python.org/2.7/
> [3]: https://gitlab.com/python-devs/ci-images/-/blob/master/Dockerfile#L19
>
> --
> [Julien Palard](https://mdk.fr)
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/Y35KK55BW7PTUIQKSNROLZGYVHBMHE7O/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4PVCVYI6U7OIFQNBOPK4HUCF2NZS6WLN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: len(path) == len(str(path))

[Kyle Stanley]

While I'm -1 on the original proposal, I think the idea of PurePath.__len__
returning the number of components in the path may be worth some further
consideration. Also, I'm not convinced that having indexing is a necessary
prerequisite to pursue it further.

On Sun, May 24, 2020 at 8:14 AM Steven D'Aprano  wrote:

> On Sun, May 24, 2020 at 02:27:00PM +0300, Ram Rachum wrote:
>
> > Today I wrote a script and did this:
> >
> > sorted(paths, key=lambda path: len(str(path)), reverse=True)
> >
> > But it would have been nicer if I could do this:
> >
> > sorted(paths, key=len, reverse=True)
>
> It would have been even nicer if we could compose functions:
>
> sorted(paths, key=len∘str, reverse=True)
>
> *semi-wink*
>
>
> > So I implemented `PurePath.__len__` as `str(len(path))`.
> >
> > Serhiy and Remi objected, because it might not be obvious that the length
> > of the path would be the length of string.
>
> It isn't obvious to me that the length of a path is the length of the
> string representation of that path.
>
> I thought that the length of the path would be the number of path
> components:
>
> /tmp/file  # length 2
>
> I wouldn't have predicted that was the length of the string
> representation of the path.
>
>
> --
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/RHOPEZHLIPN5OVHNQGGT55VF5OBG4RJX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JMFSMAYRZ5QXVDJY44EYXS5RJH6WOGYY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: `if` condition with `as` keyword

[Kyle Stanley]

Also, just to clearly demonstrate how the walrus operator could be
used for the OP's examples:

original:
```
if myDict.get('spam'):
print(myDict.get('spam'))
```
walrus:
```
if (x := d.get("spam")):
print(x)
```

original:
```
x = myDict.get('spam')
if x is not None:
print(x)
```
walrus:
```
if (x := d.get("spam")) is not None:
print(x)
```

original:
```
days = seconds // 86400
if days > 1:
print(days)
```
walrus:
```
if (days := seconds // 86400) > 1:
print(days)
```

On Sun, Apr 26, 2020 at 9:04 AM Piotr Duda  wrote:
>
> Use := operator (python 3.8+)
> https://www.python.org/dev/peps/pep-0572/
>
> niedz., 26 kwi 2020 o 14:37 Николай Мостовенко  
> napisał(a):
> >
> > Hi to everybody. I was looking for any resolution for my pain in PEP and 
> > Stack Overflow and can't find an answer. As well the similar ideas here are 
> > 13 years old and completely missed.
> > I feel that I need to create a garbage code doing things like:
> > ```
> > if myDict.get('spam'):
> > print(myDict.get('spam'))
> > ```
> > or
> > ```
> > x = myDict.get('spam')
> > if x is not None:
> > print(x)
> > ```
> > or
> > ```
> > days = seconds // 86400
> > if days > 1:
> > print(days)
> > ```
> >
> > and I want to use the `as` keyword instead i.e.
> > ```
> > if myDict.get('spam') as x: print(x)
> > ```
> > ```
> > if myDict.get('spam') as x is None: print(x)
> > ```
> > ```
> > if (seconds // 86400) as days > 1: print(days)
> > ```
> > ___
> > Python-ideas mailing list -- python-ideas@python.org
> > To unsubscribe send an email to python-ideas-le...@python.org
> > https://mail.python.org/mailman3/lists/python-ideas.python.org/
> > Message archived at 
> > https://mail.python.org/archives/list/python-ideas@python.org/message/T3DV7XL4ZTEYOGPI5V3ZHVJIBVNNJYBS/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> 闇に隠れた黒い力
> 弱い心を操る
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/ZJPMMG7CSWCMV25CHQRWZZIVGBYOFGS2/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CQ73M2VGFGLEIH3ZDGDKG3W632XTVGSF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Adding a "once" function to functools

[Kyle Stanley]

Christopher Barker wrote:
> Is there a  core developer with a particular interest in functools to be 
> consulted?

Yep, Raymond Hettinger is the designated expert for the functools
module. See https://devguide.python.org/experts/#stdlib. I'm not sure
as to whether he regularly checks python-ideas, but I suspect he would
definitely notice a python-dev thread. Other core developers with some
experience in this domain would also likely chime in.

On Mon, Apr 27, 2020 at 4:46 PM Christopher Barker  wrote:
>
> On Mon, Apr 27, 2020 at 12:26 PM Antoine Pitrou  wrote:
>>
>> My own opinion is that it's limited enough in scope (and probably
>> uncontroversial implementation-wise) that it doesn't need a PEP.  Of
>> course other core developers may disagree, so perhaps wait a few days
>> before submitting a PR :-)
>
>
> Is there a  core developer with a particular interest in functools to be 
> consulted?
>
> -CHB
>
>
>
>
>
>>
>>
>> Regards
>>
>> Antoine.
>>
>>
>> On Mon, 27 Apr 2020 20:16:53 +0100
>> Tom Forbes  wrote:
>> > I would think that would be a great name, it’s more explicit and fits more 
>> > in with its siblings “lru_cache” and “cached_property”.
>> >
>> > Not to imply that there is a consensus to move forward, I would be 
>> > interested in knowing what the next steps would be if there was. Would 
>> > this require a PEP to be submitted? Or is it similar enough to existing 
>> > functionality that it might not require one?
>> >
>> > > On 27 Apr 2020, at 19:37, Antoine Pitrou 
>> > >  wrote:
>> > >
>> > > On Mon, 27 Apr 2020 09:26:53 -0700
>> > > Ethan Furman  wrote:
>> > >>
>> > >>> Or are you saying that once could be interpreted to mean I cannot do 
>> > >>> that 2nd def?
>> > >>
>> > >> That is what I was saying -- that `once`, all by itself, could mean 
>> > >> multiple things.
>> > >
>> > > That's a good point.  A more explicit spelling would be `@call_once`,
>> > > what do you think?
>> > >
>> > > (that's also the C++ spelling, incidentally, but I suspect most
>> > > beginners won't care about that ;-))
>> > >
>> > > Regards
>> > >
>> > > Antoine.
>> > >
>> > > ___
>> > > Python-ideas mailing list -- 
>> > > python-ideas-+zn9apsxkcednm+yrof...@public.gmane.org
>> > > To unsubscribe send an email to 
>> > > python-ideas-leave-+zn9apsxkcednm+yrof...@public.gmane.org
>> > > https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> > > Message archived at 
>> > > https://mail.python.org/archives/list/python-ideas@python.org/message/NYOB46XXHLPDK3V7IPQL6IJ6FNE2ZK7S/
>> > > Code of Conduct: http://python.org/psf/codeofconduct/
>> >
>>
>>
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-ideas@python.org/message/BJ7MPRTG7EL3LD4FUWG63XPRLCWEPQH7/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Christopher Barker, PhD
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/KVXXHFTLQN76BD2V74SBUVKO5QOZUNRL/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4WJ42AQE5NCUQK6BW66R6OQNMWPWXTXD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Adding a "once" function to functools

[Kyle Stanley]

I'd like to give my explicit +1 on this proposal, and for the
suggested alternative name "call_once". To me, this seems to be the
case of a common small startup optimization that's very easy to get
wrong, and to many users may be too much of a hassle to write their
own implementation to be worth using. Thus, it makes sense to include
something like this in the standard library.

The alternative name "call_once" will make it's purpose significantly
more clear. IMO, the parallel to a similar functionality in C++ is a
nice added bonus. Although we shouldn't restrict our APIs based on the
naming choice of other languages, it's very convenient for those who
are already familiar with the other one.

Also, I'm in agreement that the addition of this to functools is
similar enough to the existing members and small enough that it
shouldn't require a PEP. A bpo issue and PR should be sufficient, but
I would consider potentially cross-posting this to python-dev to get
additional attention from other core developers.

On Mon, Apr 27, 2020 at 3:25 PM Antoine Pitrou  wrote:
>
>
> My own opinion is that it's limited enough in scope (and probably
> uncontroversial implementation-wise) that it doesn't need a PEP.  Of
> course other core developers may disagree, so perhaps wait a few days
> before submitting a PR :-)
>
> Regards
>
> Antoine.
>
>
> On Mon, 27 Apr 2020 20:16:53 +0100
> Tom Forbes  wrote:
> > I would think that would be a great name, it’s more explicit and fits more 
> > in with its siblings “lru_cache” and “cached_property”.
> >
> > Not to imply that there is a consensus to move forward, I would be 
> > interested in knowing what the next steps would be if there was. Would this 
> > require a PEP to be submitted? Or is it similar enough to existing 
> > functionality that it might not require one?
> >
> > > On 27 Apr 2020, at 19:37, Antoine Pitrou 
> > >  wrote:
> > >
> > > On Mon, 27 Apr 2020 09:26:53 -0700
> > > Ethan Furman  wrote:
> > >>
> > >>> Or are you saying that once could be interpreted to mean I cannot do 
> > >>> that 2nd def?
> > >>
> > >> That is what I was saying -- that `once`, all by itself, could mean 
> > >> multiple things.
> > >
> > > That's a good point.  A more explicit spelling would be `@call_once`,
> > > what do you think?
> > >
> > > (that's also the C++ spelling, incidentally, but I suspect most
> > > beginners won't care about that ;-))
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > > ___
> > > Python-ideas mailing list -- 
> > > python-ideas-+zn9apsxkcednm+yrof...@public.gmane.org
> > > To unsubscribe send an email to 
> > > python-ideas-leave-+zn9apsxkcednm+yrof...@public.gmane.org
> > > https://mail.python.org/mailman3/lists/python-ideas.python.org/
> > > Message archived at 
> > > https://mail.python.org/archives/list/python-ideas@python.org/message/NYOB46XXHLPDK3V7IPQL6IJ6FNE2ZK7S/
> > > Code of Conduct: http://python.org/psf/codeofconduct/
> >
>
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/BJ7MPRTG7EL3LD4FUWG63XPRLCWEPQH7/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/BWICEGCLHTJRYOCWSP4VFPL6CMZZVFWO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Make type(None) the no-op function

[Kyle Stanley]

Soni L. wrote:
> it'd be nice if it did the same regardless of args.

Sorry, but "it'd be nice" is not a real world use case.

> inspired by PEP 559: https://www.python.org/dev/peps/pep-0559/

That PEP was rejected. It's okay to use previously rejected proposals
for inspiration, but your proposal would have to include additional
real world use cases and arguments that weren't provided in the
original PEP. I'm not seeing that in the post above.

Many of your previous proposals have led to some interesting
discussions, but this one seems exceptionally under-developed. There's
nothing wrong with a well developed proposal that gets rejected (as it
can lead to highly constructive discussions), or a newcomer to the
list that doesn't realize how high the bar is; however, that does not
seem to be the case here.

Soni, I hope this doesn't come across as harsh and I really don't want
to discourage you from making proposals in general, but it's rather
inconsiderate of the time of the subscribers to the list to repeatedly
post under-developed proposals that lack any real use cases or strong
arguments that weren't previously rejected. Remember, everyone taking
the time to read your proposal is doing so entirely voluntarily.

If you continue to make these types of proposals, you'll soon find
that they will not be seriously considered or given as much attention
as they otherwise would; making it even more difficult for the
proposal to make it into the language (which I assume is a goal). In
order for any major/fundamental change proposal to be accepted, it has
be sponsored by a core developer, converted into a formal PEP, and
then eventually approved by the SC if there's adequate consensus. So,
if your proposals stop receiving attention because of a large volume
of recent prior proposals that weren't adequately developed, it will
be much more difficult, if not practically impossible to find a
willing sponsor.

Moving forward, I would highly recommend looking at some of the prior
proposals on this list that were accepted, and then try to frame yours
based upon them. Before posting it to the list, think: "How would this
change improve the experience of users? What specific real-world use
cases does it have? Is the potential benefit worth the implementation,
review, and indefinite maintenance cost? Has a similar proposal been
made before? If so, are my counter arguments for the previous
rejection reasons strong enough to justify another proposal?". If
those questions can't be answered, the proposal is most likely not
worth posting in the first place.

On Fri, Apr 24, 2020 at 11:19 AM Soni L.  wrote:
>
> Currently type(None) returns None if you call it with no args:
>
>  >>> print(type(None)())
> None
>
> it'd be nice if it did the same regardless of args. currently it raises:
>
>  >>> print(type(None)(1))
> Traceback (most recent call last):
>File "", line 1, in 
> TypeError: NoneType takes no arguments
>
> inspired by PEP 559: https://www.python.org/dev/peps/pep-0559/
>
>  >>> NoneType
> Traceback (most recent call last):
>File "", line 1, in 
> NameError: name 'NoneType' is not defined
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/7NZM6XGGX2OUZVG6HWJU43BXCCX5QYLX/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QUCZSA4JFGQZRNCKYZXTISPU2GAOKON4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: collections.UpdateDict, collections.InsertDict, collections.InsertOrIgnoreDict

[Kyle Stanley]

Steven D'Aprano wrote:
> Do nothing?

> I don't have to suggest a better idea, since its not me proposing a
> change. I don't think any change is needed. It is up to you to firstly
> justify that a change is needed, and only then justify a specific
> response to that need.

[snip]

> If you personally would use them, I'm happy for you. But unless there is
> likely to be either moderate or widespread use among the community, or
> at least some uses in the stdlib, those functions belong in your own
> personal toolbox, not the stdlib.

+1 to doing nothing here, at least based on the current proposal.

Soni, I would strongly recommend spending some more time to come up
with specific, concrete use cases for feature proposals in the future
(or in this thread). Preferably with some detailed "before and after"
examples using *non-trivial* real world code, and some explanations as
to why you think it should be included in the standard library instead
of just something implemented locally or in a 3rd party package.

Without doing the above, it's very likely going to end up being an
infinite loop of seeing the same repeated counter-arguments on your
proposals. If those points can't be answered and defended, it's highly
unlikely the proposal would be accepted in the first place; regardless
of how interesting the ideas may seem.

On Thu, Apr 16, 2020 at 10:37 PM Steven D'Aprano  wrote:
>
> On Thu, Apr 16, 2020 at 01:11:46PM -0300, Soni L. wrote:
>
> > we can't break setdefault (particularly for tuple keys), do you have a
> > better idea?
>
> Do nothing?
>
> I don't have to suggest a better idea, since its not me proposing a
> change. I don't think any change is needed. It is up to you to firstly
> justify that a change is needed, and only then justify a specific
> response to that need.
>
> An anecdote... when I first discovered Python back in 1.5 days, and
> specifically the dict.update method, practically the first thing I did
> was wonder why update replaced existing keys. What if I wanted it to
> keep the existing key, or raise an exception? So I immediately wrote a
> pair of functions to do both those things.
>
> It's been close to 20 years now and so long as I can remember, I have
> never, not once, used either of those variants I don't even know
> whether I even still have them (not that it would be hard to recreate
> them).
>
> If you personally would use them, I'm happy for you. But unless there is
> likely to be either moderate or widespread use among the community, or
> at least some uses in the stdlib, those functions belong in your own
> personal toolbox, not the stdlib.
>
>
> --
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/SPY2DRKUZ3LUX56AYOA2QKEMT4KAZLNP/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RVBUMROZZIDR3DCMDPBG2RTGUSQKSQPN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Keyword arguments self-assignment

[Kyle Stanley]

Dominik Vilsmeier wrote:
> I'm not sure if this is doable from the compiler perspective, but what
> about allowing tuples after `**` unpacking:
>
>  requests.post(url, **(data, params))
>
>  # similar to
>  requests.post(url, data=data, params=params)

+1. I can see the practical utility of the feature, but was strongly against the
other syntax proposals so far. IMO, the above alternative does a great job of
using an existing feature, and I think it would be rather easy to explain how
it works.


On Thu, Apr 16, 2020 at 3:06 PM Dominik Vilsmeier
 wrote:
>
> I'm not sure if this is doable from the compiler perspective, but what
> about allowing tuples after `**` unpacking:
>
>  requests.post(url, **(data, params))
>
>  # similar to
>  requests.post(url, data=data, params=params)
>
> Probably some magic would need to happen in order to merge the names
> with their values, but the same is true for the `post(url, data=,
> params=)` syntax.
>
>
> On 16.04.20 18:57, oliveira.rodrig...@gmail.com wrote:
> > @StevenDAprano and this goes for @RhodriJames , thank you for sharing your 
> > point of view. Indeed the proposed syntax is obscure and would not be that 
> > readable for beginners.
> >
> > Couldn't we work around this so? The concept is still good for me just the 
> > syntax that is obscure, maybe something like this would work:
> >
> > ```python
> > # '*' character delimits that subsequent passed parameters will be passed
> > # as keyword arguments with the same name of the variables used
> > self.do_something(positional, *, keyword)
> > # this would be equivalent to:
> > self.do_something(positional, keyword=keyword)
> > ```
> > I believe this is readable even if you don't know Python: `positional` and 
> > `keyword` are being passed as parameters, the `*` character is mysterious 
> > at first but so it is in `def foo(a, *, b)` and it doesn't get into the way 
> > of basic readability.
> >
> > Rodrigo Martins de Oliveira
> > ___
> > Python-ideas mailing list -- python-ideas@python.org
> > To unsubscribe send an email to python-ideas-le...@python.org
> > https://mail.python.org/mailman3/lists/python-ideas.python.org/
> > Message archived at 
> > https://mail.python.org/archives/list/python-ideas@python.org/message/N2ZY5NQ5T2OJRSUGZJOANEQOGEQIYYIK/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/S44VDMZ4AFBSGIQEVWMKKWOW4P6WRVXY/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZJ5FROVGX7WM5OPNAWPSGSUSBDWW33L5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Explicitly defining a string buffer object (aka StringIO += operator)

[Kyle Stanley]

Paul Sokolovsky wrote:
> Roughly speaking, the answer would be about the same in idea as answers
> to the following questions:
> [snip]

I would say the difference between this proposal so far and the ones listed
are that they emphasized concrete, real-world examples from existing code
either in the stdlib or "out in the wild", showing clear before and after
benefits of the proposed syntax. It may not seem necessary to the person
proposing the feature and it does take some time to research, but it
creates a drastically stronger argument for the new feature. The code
examples I've seen so far in the proposal have been mostly abstract or
simple toy examples. To get a general idea, I'd recommend looking over the
examples in their respective PEPs, and then try to do something similar in
your own arguments.

> The scholasm of "there's only one way to do it" is getting old for this
> language. Have you already finished explaining everyone why we needed
> assignment expressions, and why Python originally had % as a formatting
> operator, and some people swear to keep not needing anything else?

While I agree that it's sometimes okay to go outside the strict bounds of
"only one way to do it", there needs to be adequate justification for doing
so which provides a demonstrable benefit in real-world code. So the default
should be just having one way, unless we have a very strong reason to
consider adding an alternative. This was the case for the features you
mentioned above.

> Please let people learn
> computer science inside Python, not learn bag of tricks to then escape
> in awe and make up haikus along the lines of:
>
> A language, originally for kids,
> Now for grown-up noobs.

Considering the current widespread usage of Python in the software
development industry and others, characterizing it as a language for
"grown-up noobs" seems rather disingenuous (even if partially in jest). We
emphasize readability and beginner-friendliness, but Python is very far
from beginner-only and I don't think it's even reasonable to say that it's
going in that direction. In some ways, it simplifies operations that would
otherwise be more complicated, but that's largely the point of a high-level
language: abstracting the complex and low-level parts to focus more on the
core business logic.

Also, while I can see that blindly relying on "str += part" can be
sidestepping the underlying computer science to some degree, I find that
appending the parts to a list and joining the elements is very conceptually
similar to using a string buffer/builder; even if the syntax differs
significantly from how other languages do it.

Regarding the proposal in general though, I actually like the main idea of
having "StringBuffer/StringBuilder"-like behavior, *assuming* it provides
substantial benefits to alternative Python implementations compared to
``""join()``. As someone who regularly uses other languages with something
similar, I find the syntax to be appealing, but not strong enough on its
own to justify a stdlib version (mainly since a wrapper would be very
trivial to implement).

But, I'm against the idea of adding this to the existing StringIO class,
largely for the reasons cited above, of it being outside of the scope of
its intended use case. There's also a significant discoverability factor to
consider. Based on the name and its use case in existing versions of
Python, I don't think a substantial number of users will even consider
using it for the purpose of building strings. As it stands, the only people
who could end up benefiting from it would be the alternative
implementations and their users, assuming they spend time *actively
searching* for a way to build strings with reduced memory usage. So I would
greatly prefer to see it as a separate class with a more informative name,
even if it ends up being effectively implemented as a subset of StringIO
with much of the same logic.

For example:

buf = StringBuilder() # feel free to bikeshed over the name
for part in parts:
buf += part # in the __iadd__, it would presumably call something like
buf.append() or buf.write()
return str(buf)

This would be highly similar to existing string building classes in other
popular languages, such as Java and C#.

Also, on the point of memory usage: I'd very much like to see some real
side-by-side comparisons of the ``''.join(parts)`` memory usage across
Python implementations compared to ``StringIO.write()``. I some earlier in
the thread, but the results were inaccurate since they relied entirely on
``sys.getsizeof()``, as mentioned earlier. IMO, having accurate memory
benchmarks is critical to this proposal. As Chris Angelico mentioned, this
can be observed through monitoring the before and after RSS (or equivalent
on platforms without it). On Linux, I typically use something like this:

```
def show_rss():
os.system(f"grep ^VmRSS /proc/{os.getpid()}/status")
```

With the above in mind, I'm currently +0 on the proposal. It seems like it

[Python-ideas] Re: PEP 9999: Retire animal-unfriendly language

[Kyle Stanley]

Chris Angelico wrote:
> We will need a
> migration plan. Unfortunately, African swallows are non-migratory,
> severely impacting your proposal.

Well, aren't European swallows migratory? They may not be able to carry a
coconut, but certainly a migratory plan would be under their weight limit.
:-)

On Tue, Mar 31, 2020 at 2:24 PM Chris Angelico  wrote:

> On Wed, Apr 1, 2020 at 4:19 AM Gerrit Holl  wrote:
> > Abstract
> > 
> >
> > Python has long used metasyntactic variables that are based on the
> > consumption of meat and dairy products, such as "spam", "ham", and
> > "eggs".
> > This language is not considerate to pigs or chicken and violates the
> > spirit of the Code of Conduct.  This PEP proposes to retire the use
> > of those names in official Python documentation and source code and to
> > recommend users of Python to do the same.
> >
>
> While I wholeheartedly agree with your intention here, I believe the
> backward incompatibilities would make this untenable. We will need a
> migration plan. Unfortunately, African swallows are non-migratory,
> severely impacting your proposal.
>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/FRNE2HZWLBSONIHYHK2Z3OVXCRT4MMY7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZI4D5CEBDIVIWHMLAVOUI3BRNXQ5DSY3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: About python3 on windows

[Kyle Stanley]

> The Microsoft Store version does not offer any options full stop!

Good to know, thanks for clarifying. I haven't personally installed it from
the Microsoft app store, but it was my assumption that it did not allow for
custom installation options.

On Tue, Mar 24, 2020 at 3:18 PM Steve Barnes  wrote:

> https://docs.python.org/3.8/using/windows.html#installing-without-ui
> gives details of all of the options. The Microsoft Store version does not
> offer any options full stop!
>
>
>
> *From:* Kyle Stanley 
> *Sent:* 24 March 2020 18:10
> *To:* Frédéric De Jaeger 
> *Cc:* Python-ideas < >
> *Subject:* [Python-ideas] Re: About python3 on windows
>
>
>
> > That would be nice.  Does it apply to the _windows store version_,  the
> _traditional installer_, both ?
>
>
>
> I believe it only applies to the traditional installer from python.org.
> You will also have to verify it, as it has been a decent while since I've
> had to install Python on Windows.
>
>
>
> On Tue, Mar 24, 2020 at 7:05 AM Frédéric De Jaeger <
> fdejae...@novaquark.com> wrote:
>
> > Windows devices, they might want to consider using the PrependPath option
> > (which adds Python to PATH from the command line). I.E.
> > python-3.8.0.exe /quiet InstallAllUsers=1 PrependPath=1
> > (IIRC, the above installation should allow usage of "python3" on
> > Windows for all users on the device)
>
> That would be nice.  Does it apply to the _windows store version_,  the
> _traditional installer_, both ?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/WQFJ4YGNUCKBITTHZQCME3GQRXPVPYT2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZEY7LXEGPKBRCO6RBSSFSQ2W2F7ZLOI6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: About python3 on windows

[Kyle Stanley]

> That would be nice.  Does it apply to the _windows store version_,  the
_traditional installer_, both ?

I believe it only applies to the traditional installer from python.org. You
will also have to verify it, as it has been a decent while since I've had
to install Python on Windows.

On Tue, Mar 24, 2020 at 7:05 AM Frédéric De Jaeger 
wrote:

> > Windows devices, they might want to consider using the PrependPath option
> > (which adds Python to PATH from the command line). I.E.
> > python-3.8.0.exe /quiet InstallAllUsers=1 PrependPath=1
> > (IIRC, the above installation should allow usage of "python3" on
> > Windows for all users on the device)
>
> That would be nice.  Does it apply to the _windows store version_,  the
> _traditional installer_, both ?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/WQFJ4YGNUCKBITTHZQCME3GQRXPVPYT2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VHVGN6UWENOKG2LT37C4APGRFXCKOKTL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: About python3 on windows

[Kyle Stanley]

Brett Cannon wrote:
> There is an option to install Python to PATH on Windows if you check the
appropriate box during installation, but that's not really the
> way Windows apps typically work.

In this case of the OP, if they're mass installing Python on company
Windows devices, they might want to consider using the PrependPath option
(which adds Python to PATH from the command line). I.E.

python-3.8.0.exe /quiet InstallAllUsers=1 PrependPath=1

(IIRC, the above installation should allow usage of "python3" on
Windows for all users on the device)


On Mon, Mar 23, 2020 at 7:52 PM Brett Cannon  wrote:

>
>
> On Mon, Mar 23, 2020 at 3:47 PM Mike Miller 
> wrote:
>
>>
>> On 2020-03-23 10:59, Frédéric De Jaeger wrote:
>> > Hi all,
>> >
>> > There is a recurring problem
>>
>> Yep, that's a problem.  I've built up a habit on Ubuntu where I type
>> python3 a
>> number of times a day, honed over several years.  So far so good.
>>
>
> Also note that Python doesn't control the OS vendors and this is very much
> a vendor decision. We tried to provide guidance, but that doesn't mean we
> are listened to. :)
>
>
>>
>> Recently I've had to use a Windows VM for some stuff at work, where I
>> installed
>> Python 3 as well.  Every time I type python3 at the command-line (instead
>> of
>> python) to use the repl, it tries to load the Microsoft App Store!
>>
>
> There is an option to install Python to PATH on Windows if you check the
> appropriate box during installation, but that's not really the way Windows
> apps typically work.
>
>
>>
>> I realize why that was done, to help out newbies, but it is not what I
>> want at
>> all.  Not the end of the world, but some consistency would be nice.
>>
>
> As I said, you can install Python to PATH on Windows if you choose.
>
> I'm also slowly working on a Python Launcher for UNIX so the other option
> some day might be `py` no matter what OS you're on.
>
> -Brett
>
>
>>
>> -Mike
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/4XXUECWBIFLK6IRW7FA6TXLUIYQ7CQHS/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/NZ3J3Y6GEQDBU7U2ID4L5J4CHRG4LWHS/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RR5MWZD6DB6KAY2FSSY5CIMH5PQYB2UL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: New explicit methods to trim strings

[Kyle Stanley]

Stephen J. Turnbull wrote:
> The only cases I can remember are files named things like
> "thesis.doc.doc" in GUI environments. ;-)

For edge cases like that, something like
`"thesis.doc.doc".removesuffix(".doc").removesuffix(".doc")` should
suffice, no? It may not be the cleanest looking solution, but IMO, it's
better than over-complicating the method for something that would be used
rarely at best.

On Mon, Mar 23, 2020 at 1:34 AM Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Paul Moore writes:
>
>  > I've needed to remove one prefix/suffix. I've never needed to remove
>  > more than one.
>
> The only cases I can remember are files named things like
> "thesis.doc.doc" in GUI environments. ;-)
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/NT3ORR6E54EW3CKJGXGCN7VJWPAG5PBX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3ALBB4MJG3ENCNCZ3VWHYYCOHP75O7WS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Instance method to test equivalence between set and iterable

[Kyle Stanley]

Steve Jorgensen wrote:
> Basically, it is for a sense of completeness. It feels weird that there
is a way to check whether an iterable is a subset of a set or a superset
> of a set but no way to directly ask whether it is equivalent to the set.

I can't say this has never happened historically, but I've yet to ever see
a change proposal accepted simply on the basis of "completeness". There
typically has to be some form of concrete, practical use case for the
feature. Otherwise, it's quite simply not going to be worth the review and
long-term maintenance cost to the limited core CPython development
resources.

> Even though the need for it might not be common, I think that the
collection of methods makes more sense if a method like this is present.

It's okay if the need isn't very common, but it still typically requires
some real use case to be clearly defined. Even if it might be fairly niche,
are there any?

(Note that if it's too niche, it's likely more suitable as a 3rd party
package.)

On Sun, Mar 22, 2020 at 8:11 PM Steve Jorgensen  wrote:

> Paul Moore wrote:
> > On Sun, 22 Mar 2020 at 20:01, Steve Jorgensen ste...@stevej.name wrote:
> > >
> > > Currently, the issubset and
> > > issuperset methods of set objects accept arbitrary iterables as
> arguments. An
> > > iterable that is both a subset and superset is, in a sense, "equal" to
> the set. It would
> > > be inappropriate for == to return True for such a comparison,
> > > however, since that would break the Hashable contract.
> > > Should sets have an additional method, something like like(other),
> > > issimilar(other), or isequivalent(other), that returns
> > > True for any iterable that contains the all of the items in the set
> and no
> > > items that are not in the set? It would therefore be true in the same
> cases where
> > >  = set(other) or .issubset(other) and
> > > .issuperset(other) is true.
> > > What is the practical use case for this? It seems like it would be a
> > pretty rare need, at best.
> > Paul
>
> Basically, it is for a sense of completeness. It feels weird that there is
> a way to check whether an iterable is a subset of a set or a superset of a
> set but no way to directly ask whether it is equivalent to the set.
>
> Even though the need for it might not be common, I think that the
> collection of methods makes more sense if a method like this is present.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/MRCHHRVCXEUAB3HBV4WRMZ56O3HUJQYL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/F63RTRE56JNDUJ3P2ZKIIE5VISKACV3N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: dunder methods for encoding & prettiness aware formal & informal representations

[Kyle Stanley]

I ended up opening an issue for it at https://bugs.python.org/issue40045.
<http://bugs.python.org>

On Sun, Mar 22, 2020 at 8:33 PM Kyle Stanley  wrote:

> Chistopher Barker wrote:
> > I'd suggest you make a PR on the docs.
>
> Yeah I was planning on either doing that, or opening it as a "newcomer
> friendly"/"easy" issue on bugs.python.org. IMO, it could make for a
> decent first PR.
>
> On Sun, Mar 22, 2020 at 1:28 PM Christopher Barker 
> wrote:
>
>> On Fri, Mar 20, 2020 at 8:52 PM Kyle Stanley  wrote:
>>
>>> >
>>> https://docs.python.org/3/reference/lexical_analysis.html#reserved-classes-of-identifiers
>>>
>>> I remembered the existence of this rule and tried to locate it recently
>>> (prior to this discussion), but was unable to because it doesn't explicitly
>>> mention "dunder". IMO, it would make that section much easier to find if it
>>> were to say either of:
>>>
>>> 1) System-defined names, also known as "dunder" names.
>>> 2) System-defined names, informally known as "dunder" names.
>>> 3) System-defined "dunder" names.
>>>
>>
>> The word "dunder" was coined (relatively) recently. It was not in common
>> use when those docs were written. Another common nickname is "magic
>> methods".
>>
>> So that explains why the word "dunder" isn't in those docs. And those
>> docs are pretty "formal" / "technical" anyway, not designed for the casual
>> user to read.
>>
>> But yes, it would be good to add a bit of that text to make it more
>> findable.
>>
>> I'd suggest you make a PR on the docs.
>>
>> -CHB
>>
>>
>> Python Language Consulting
>>   - Teaching
>>   - Scientific Software Development
>>   - Desktop GUI and Web Development
>>   - wxPython, numpy, scipy, Cython
>>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/I4GRJAO37DYY5ZJDBU4XWIEZQT4WRXR4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: dunder methods for encoding & prettiness aware formal & informal representations

[Kyle Stanley]

Chistopher Barker wrote:
> I'd suggest you make a PR on the docs.

Yeah I was planning on either doing that, or opening it as a "newcomer
friendly"/"easy" issue on bugs.python.org. IMO, it could make for a decent
first PR.

On Sun, Mar 22, 2020 at 1:28 PM Christopher Barker 
wrote:

> On Fri, Mar 20, 2020 at 8:52 PM Kyle Stanley  wrote:
>
>> >
>> https://docs.python.org/3/reference/lexical_analysis.html#reserved-classes-of-identifiers
>>
>> I remembered the existence of this rule and tried to locate it recently
>> (prior to this discussion), but was unable to because it doesn't explicitly
>> mention "dunder". IMO, it would make that section much easier to find if it
>> were to say either of:
>>
>> 1) System-defined names, also known as "dunder" names.
>> 2) System-defined names, informally known as "dunder" names.
>> 3) System-defined "dunder" names.
>>
>
> The word "dunder" was coined (relatively) recently. It was not in common
> use when those docs were written. Another common nickname is "magic
> methods".
>
> So that explains why the word "dunder" isn't in those docs. And those docs
> are pretty "formal" / "technical" anyway, not designed for the casual user
> to read.
>
> But yes, it would be good to add a bit of that text to make it more
> findable.
>
> I'd suggest you make a PR on the docs.
>
> -CHB
>
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SAD4FIMQHP5PQI2X6X6KTZCF2L5AZMXG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: dunder methods for encoding & prettiness aware formal & informal representations

[Kyle Stanley]

Steven D'Aprano wrote:
> Yes, really.
>
>
https://docs.python.org/3/reference/lexical_analysis.html#reserved-classes-of-identifiers

I remembered the existence of this rule and tried to locate it recently
(prior to this discussion), but was unable to because it doesn't explicitly
mention "dunder". IMO, it would make that section much easier to find if it
were to say either of:

1) System-defined names, also known as "dunder" names.
2) System-defined names, informally known as "dunder" names.
3) System-defined "dunder" names.

Instead of the current:

System-defined names.

(or any mention of "dunder" for that matter)

It would be much easier to locate. Realistically speaking, I can't imagine
that the majority of Python users and library maintainers would read
through the entire "Lexical Analysis" page of the docs if they're not
specifically interested in parsing or something similar. :-)

Regardless though, thanks for providing the link. I'll try to remember its
location for future reference.


On Fri, Mar 20, 2020 at 12:29 AM Steven D'Aprano 
wrote:

> On Fri, Mar 20, 2020 at 03:01:16PM +1100, Chris Angelico wrote:
> > On Fri, Mar 20, 2020 at 2:59 PM Steven D'Aprano 
> wrote:
> > >
> > > On Thu, Mar 19, 2020 at 11:38:28PM +, Samuel Colvin wrote:
> > >
> > > > But it also looks for a "__pretty__" method on objects, and if found
> uses
> > > > that to display the object.
> > >
> > > Are you aware that dunder names are reserved for Python's use?
> > >
> >
> > Really?
>
> Yes, really.
>
>
> https://docs.python.org/3/reference/lexical_analysis.html#reserved-classes-of-identifiers
>
>
> > Somebody better tell SQLAlchemy that they're breaking rules, then.
>
> Lots of people break the rules all the time. Doesn't mean they should.
>
>
>
>
> --
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/EE63SP2N6DKZ3ISY3UFAC6HMBGAKPZCA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/W7QI6WG5Z2FIFT4K34MQOBNLAUODW3KK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Syntax for loop invariants

[Kyle Stanley]

I'd have to agree with Steven here, I'm -0 on the proposal since it's not
convincing (at least at the moment).

haael wrote:
> Python has more and more optional tools for formal correctness cheking.
> Why not loop invariants?

A better question would be "why add loop invariants?" rather than "why not
add loop invariants?". The OP provides the basic syntax and a simple
overview of how it would work, but doesn't provide any convincing arguments
or real-world examples of why the feature should be added in the first
place. That bar is already quite high for any new feature, but it's even
higher for a new keyword for the language.

I'd recommend finding some existing code and demonstrating how it could be
significantly improved with the new feature. Concrete "before-and-after"
examples can go a long way in making a strong argument for a feature
proposal.

> I am developing a library for formal proofs and such a feature would be
> handy.

This point could be worth further elaborating on:

1) Specifically, how would this feature help your library?
2) Could it similarly benefit other libraries?
3) How much does this benefit from being implemented at the language level?




On Tue, Mar 17, 2020 at 11:00 PM Steven D'Aprano 
wrote:

> On Tue, Mar 17, 2020 at 06:12:56PM +, haael wrote:
> >
> > Python has more and more optional tools for formal correctness cheking.
>
> Are you talking about third-party applications?
>
>
> > Why not loop invariants?
>
> Wrong question :-)
>
> Adding new language features to Python isn't "default allow", adding
> every new feature that somebody suggests unless there is a reason not to
> add it. It is "default deny" -- new features have to be justified before
> they will be added, and even if they are justified, they still might not
> be added if:
>
> - the costs of the new feature outweigh the benefit;
> - there is sufficient opposition and insufficient support;
> - or nobody is interested enough to do the work.
>
> So the better question is:
>
> *Why add loop invariants as a language feature requiring a new keyword?*
>
>
> > Loop invariant is such a statement that if it was true before the loop
> > iteration, it will also be true after the iteration. It can be
> > implemented as an assertion of an implication.
> >
> >
> > now_value = False
> > while running_condition(...):
> > prev_value = now_value
> > now_value = invariant_condition(...)
> > assert now_value if prev_value else True
> >
> >
> > Here for ellipsis we can substitute any values and variables.
>
> An actual real-world example would be more convincing.
>
>
> > I propose the new syntax:
> >
> > while running_condition(...):
> > invariant invariant_condition(...)
> >
> >
> > The keyword 'invariant' is allowed only inside a loop.
>
> There are other kinds of invariants than just loop invariants. Can you
> justify why loop invariants are more important than other invariants?
>
> Can you point to other languages that offer loop invariants, but not
> other invariants or contracts such as preconditions and postconditions?
>
> For comparisons sake, this is how Eiffel handles loop invariants:
>
>
> https://www.eiffel.org/doc/eiffel/ET-_Instructions#Loop_invariants_and_variants
>
> --
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/SJ7PHT3SZPQL3DZTNBANA5K4XKG6N3OK/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/63R3IIL4ULJECQKZAJ5XTDIL3TJ6X4GV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: dunder methods for encoding & prettiness aware formal & informal representations

[Kyle Stanley]

Christopher Barker wrote:
> Is there any guarantee (or even string expectation) that the __str__ for
an object won't change?
> [snip]
> *reality check*: I imagine a LOT of code out there (doctests, who know
what else) does in fact expect the str() of builtins not to change -- so
this is probably dead in the water.

Regardless of whether we can improve/modify the __str__ for existing
builtin objects, I think it is definitely worthwhile to clearly establish
whether they are a part of the stable API in the first place. Users should
be able to know if they can rely on the __str__ remaining the same across
versions.

My personal preference would be for the __repr__ to be stable (since it's
already denoted as the "official" string representation, often can be used
to replicate the object and/or uniquely identify it) and for the __str__ to
be unstable/subject to change.

As for whether the __str__ of any builtin object *should* be modified is
another question entirely. I'm of the opinion that it should require
sufficient justification: the benefit in readability/utility gained from
the change by the majority of its users should outweigh the cost of
potential breakage.

If we've not provided any guarantee for the __str__ to remain the same
across versions (which is true, AFAIK [1]), the fact that some users have
chosen to rely on it doesn't seem like it should prevent any changes from
being considered at all. It just means that the changes have to be
adequately worthwhile and evaluated on a case-by-case basis.

---

[1] - I have not extensively reviewed all of the documentation for all
__str__ stability guarantees or lack thereof, but there are none in the
places where I'd expect it would be present. In particular:
https://docs.python.org/3.8/reference/datamodel.html#object.__str__ or
https://docs.python.org/3.8/library/stdtypes.html#str.


On Sun, Mar 15, 2020 at 9:30 PM Christopher Barker 
wrote:

> On Sun, Mar 15, 2020 at 6:12 PM Stephen J. Turnbull <
> turnbull.stephen...@u.tsukuba.ac.jp> wrote:
>
>> Also, I'd like to note that "inf" and "-inf" (and "nan" for that
>> matter) are not "informal".  They are readable input, part of Python
>> syntax:
>>
>> % python3.8
>> Python 3.8.2 (default, Feb 27 2020, 19:58:50)
>> >>> float("inf")
>> inf
>> >>> float("-inf")
>> -inf
>>
>
> which, of course is what __repr__ is supposed to do, though in this case
> it doesn't quite:
> In [11]: fp = float("inf")
>
>
> In [12]: eval(repr(fp))
>
> ---
> NameError Traceback (most recent call last)
>  in 
> > 1 eval(repr(fp))
>
>  in 
>
> NameError: name 'inf' is not defined
>
> So they are not "readable input, part of Python syntax", but they are part
> of the float API.
>
> Anyway, Python now has two different ways to "turn this into a string":,
> __str__ and __repr__. I think the OP is suggesting a third, for "pretty
> version". But then maybe folks would want a fourth or fifth, or .
>
> maybe we should, instead, think about updating the __str__ for standard
> types.
>
> Is there any guarantee (or even string expectation) that the __str__ for
> an object won't change?
>
> I've always wondered why the standard str(some object) wasn't pretty to
> begin with.
>
> As for using unicode symbols for things like float("inf") -- that would be
> an inappropriate for the __repr__, but why not __str__ ?
>
> *reality check*: I imagine a LOT of code out there (doctests, who know
> what else) does in fact expect the str() of builtins not to change -- so
> this is probably dead in the water.
>
> -CHB
>
> --
> Christopher Barker, PhD
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/LETMD2YBFFA5MAI6H2RXPOEUKA6U77KY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/J6Q24KWPQQLA35VPORGB5HXPOQORC5QL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: SQL string prefix idea

[Kyle Stanley]

Rob Cliffe wrote:
> Your docs link states "... they allow you to pass on the string quote
> character by escaping it with a backslash."
>
> I don't have access to a Python 3 version right now, but that is not
> true in Python 2:
> [snip]

The behavior is the same on Python 3.8.2:

Python 3.8.2 (default, Feb 26 2020, 22:21:03)
[GCC 9.2.1 20200130] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> r'x\'y'
"x\\'y"
>>> list(_)
['x', '\\', "'", 'y']

What is unclear to me (regarding the quoted line from the Design FAQ) is
that backslashes can be used in normal strings
to properly escape the same quote type without the above issue:

>>> 'x\'y'
"x'y"
>>> list(_)
['x', "'", 'y']

Perhaps the "they" is intended to mean something other than "raw strings".
Based on the context, I think "they" might be instead referring to the
previously mentioned text processors (such as regex engines). As in "In
return, [those processors] allow you to pass on the string quote character
by escaping it with a backslash". Or perhaps the above behavior wasn't
always the case. Either way, I think that part of the Design FAQ could
benefit from an update to make that more clear, I can see how the current
wording could be a bit ambiguous.

Unless I'm mistaken about the intended meaning, the above fix could make
for a decent "newcomer friendly"/easy issue. I can open an issue on
bugs.python.org for someone else to work on as a first PR; it would be a
very straightforward (but still helpful) trivial fix.

On Sun, Mar 15, 2020 at 6:09 PM Rob Cliffe via Python-ideas <
python-ideas@python.org> wrote:

>
> On 22/02/2020 06:26, Steven D'Aprano wrote:
> >
> > Actually, in Python, regexes are the primary reason raw strings were
> > added!
> >
> > Raw strings aren't quite fully raw, which is why you can't use raw
> > strings for Windows paths:
> >
> >  path = r'somewhere\some\folder\'
> >
> > doesn't work. The reason is that "raw" (semi-cooked?) strings are
> s/are/were/
> >
> > intended for regexes, not as a general mechanism for disabling string
> > escape codes, and regexes aren't allow to end with a bare backslash.
> >
> >
> >
> https://docs.python.org/3/faq/design.html#why-can-t-raw-strings-r-strings-end-with-a-backslash
>
> So maybe it's time to make raw strings really raw?  They do have uses
> other than regexes, as your path example shows.
>
> I've been bitten by this gotcha a few times.
>
> Your docs link states "... they allow you to pass on the string quote
> character by escaping it with a backslash."
>
> I don't have access to a Python 3 version right now, but that is not
> true in Python 2:
>
> Python 2.7.10 (default, May 23 2015, 09:44:00) [MSC v.1500 64 bit
> (AMD64)] on win32
>
> Type "help", "copyright", "credits" or "licence" for more information.
>
>  >>> r'x\'y'
>
> "x\\'y"
>
>  >>> list(_)
>
> ['x', '\\', "'", 'y']
>
> (Apologies for the double vertical spacing, I'm wrestling with my email
> server.)
>
> Rob Cliffe
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/IABDEOMCX4UDXATLW5LCPKOLISE2GFPV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/W23X3LT2FWCIVEPWU64EA4XKG4XJIQ62/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Non-blocking concurrent.futures ThreadPoolExecutor

[Kyle Stanley]

Remy NOEL wrote:
> This means that if a ThreadExecutor is called from both daemon and
non-daemon threads (which is a very bad idea though), the kind of spawned
threads might be unpredictable.

IIUC, I don't think there's a way that we could reasonably support
stability for provide any guarantees for spawning or interacting with the
Executors from a daemon thread. It might even be worth considering to
explicitly warn or prevent that in some capacity.

Antoine Pitrou wrote:
> Daemon threads are already a bad idea, actually.  It is really worth
> the effort to ensure your threads exit cleanly rather than expect the
> interpreter to kill them at shutdown.

Agreed. Furthermore, if there's a thread in a program that can potentially
hang indefinitely, converting them into daemon threads (allowing it to be
killed on interpreter shutdown) just obscures the problem in most cases,
IMO. There should be some consideration placed into preventing a thread
from hanging indefinitely in the first place, such as with timeouts or some
form of outside signaling (such as threading.Event).

That's not to say that daemon threads aren't sometimes useful. Essentially,
if you're okay with the thread being abruptly killed without allowing it's
resources to be cleaned up gracefully, making it a daemon thread shouldn't
be a significant issue. But, the core problem here is that
ThreadPoolExecutor is an example of a resource that should be cleaned up
gracefully, so spawning it in a daemon thread or trying to interact with it
through one can interfere with that process.

On Tue, Mar 10, 2020 at 3:40 PM Antoine Pitrou  wrote:

> On Tue, 10 Mar 2020 15:16:06 +0100
> Remy NOEL  wrote:
> > On Mon, Mar 9, 2020 at 11:18 PM Kyle Stanley  wrote:
> >
> > > We're currently looking into adjusting the current Executor
> implementation
> > > (for both TPE and PPE) to not use daemon threads at all for
> subinterpreter
> > > compatibility, see https://bugs.python.org/issue39812. Instead of
> using
> > > an atexit handler, there's some consideration for using a similar
> > > threading-specific "exit handler". Instead of being upon program exit,
> > > registered functions would be called just before all of the non-daemon
> > > threads are joined in `threading._shutdown()`.
> > >
> > > As a result, I suspect that we are unlikely to implement something like
> > > the above. But, as a convenient side effect, it should somewhat
> address the
> > > current potentially surprising behavior.
> > >
> >
> > There might still be some fun involved given that current implementation
> > *might* lazy-spawn threads upon submit.
> > This means that if a ThreadExecutor is called from both daemon and
> > non-daemon threads (which is a very bad idea though), the kind of spawned
> > threads might be unpredictable.
>
> Daemon threads are already a bad idea, actually.  It is really worth
> the effort to ensure your threads exit cleanly rather than expect the
> interpreter to kill them at shutdown.
>
> Regards
>
> Antoine.
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/CUY3K4UYD45NAFECRKGP2KU6EMXJHNY3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6S6SOC76YY2LB44OZSHJKBI57U6LGFMC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Fwd: Possible Addition to Python Language: Marked Sub-condition

[Kyle Stanley]

> So I am sorry for the spam. Will be careful the next time.

I wouldn't worry about it too much. :)

In future, if you can establish to some degree that:

1) The feature isn't already available (if you're unsure, try python-list
as suggested above)
2) It has some realistic use cases
3) It hasn't been already proposed (if so, prepare counterarguments using
past discussions)
4) It doesn't involve major breaking changes (if it does, prepare to
strongly justify why it's worth the breakage)

If you have the above points covered, then invest some time into thinking
about and writing up the post, you're pretty much good to go. Even if the
idea is something that will never be implemented, a rejected proposal can
often yield a substantial wealth of information or provide a spark of
inspiration for a similar idea.

On Mon, Mar 9, 2020 at 10:00 AM Shrinivas Kulkarni 
wrote:

> Hi Rajeev and others
>
> Thanks for your responses. I completely overlooked the simple fact
> that the conditions are evaluated to boolean values and as such can be
> stored in variables.
> In the hindsight there may perhaps be a couple of (far-fetched) merits
> to my proposal (like useful in conditions with dynamic function calls
> or sparing users the effort of maintaining rather ephemeral variables)
> but they are of course not worth the effort of making a change to the
> language itself.
>
> So I am sorry for the spam. Will be careful the next time.
>
> Best regards
> Shrinivas Kulkarni
>
> On Mon, Mar 9, 2020 at 10:21 AM Rajeev J Sebastian
>  wrote:
> >
> > Hi Shrinivas,
> >
> > Wouldn’t these just be variables?
> >
> > Cond1 = color == Blue and count == 20
> > Cond2 = color == Red
> >
> > if cond1 or cond2:
> >   Do something
> > elif cond3:
> >Do something else
> >
> > Regards
> > Rajeev J Sebastian
> > On 8 Mar 2020, 20:42 +0530, Shrinivas Kulkarni ,
> wrote:
> >
> > Hello Everyone
> >
> > While writing python code, I frequently come across the need to do
> > certain tasks based on combined conditions.
> >
> > Much of the task for all the sub-conditions are common but some are
> > specific to one or more of these sub-conditions.
> >
> > A simplified example:
> >
> > ## Code ##
> > if (color == BLUE and count == 20) or (color == RED and count % 5 == 0):
> > rotate_the_wheel() # Common to the two sub-conditions
> > if(color == BLUE and count == 20): # First sub-condition
> > set_signal()
> > if(color == RED and count % 5 == 0): # Second sub-condition
> > clear_signal()
> > proc_post_rotate() # Common to the two sub-conditions
> >
> > I am not sure if there is a better way to do this. If not, maybe there
> > can be an extension to the language, which would allow marking a
> > sub-condition just like we mark a sub-expression in a regular
> > expression.
> >
> > Tentative syntax for this could be ({} marks the sub-condition and
> > \number refers back to it):
> >
> > ## Code ##
> > if {(color == BLUE and count == 20)} or {(color == RED and count % 5 ==
> 0)}:
> > rotate_the_wheel()
> > if(\1): # First marked condition
> > set_signal()
> > if(\2): # Second marked condition
> > unset_signal()
> > proc_post_rotate()
> >
> > And like sub-expressions, the nesting of marked sub-condions should
> > also be possible:
> >
> > ## Code ##
> > if {{(color == BLUE and count == 20)} and {value == 20}} or {(color ==
> > RED and count % 5 == 0)}:
> > if(\1):# Refers to the entire subcondition {{(color == BLUE and
> > count == 20)} and {value = 20}}
> > proc1()
> > if(\2):# Refers to sub-subcondition {value == 20}
> >
> > This will not only avoid the repetition of sub-conditions, but make
> > code readable since something like \1 will give an immediate
> > indication of a sub-condition that's defined earlier.
> >
> > Please let me know something similar is already implemented.
> > Even otherwise, all your thoughts, inputs and criticism are welcome.
> >
> > Thanks and best regards
> > Shrinivas Kulkarni
> > ___
> > Python-ideas mailing list -- python-ideas@python.org
> > To unsubscribe send an email to python-ideas-le...@python.org
> > https://mail.python.org/mailman3/lists/python-ideas.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/M5OUL4RC4N4LNVRHFAWB75QXNLN7G7TP/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/7I3K725PUUDNIY4NDSLQGTRDTR32JQVV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas 

[Python-ideas] Re: Non-blocking concurrent.futures ThreadPoolExecutor

[Kyle Stanley]

> It would seems reasonable to have a separate _daemon_thread_queues dict
(in contrast to _threads_queues) that would not get joined upon exit.
> A "daemon" (or any better name) parameter of ThreadPoolExecutor
initialisation would allow to let it add its workers in this dict rather
than the default one.

We're currently looking into adjusting the current Executor implementation
(for both TPE and PPE) to not use daemon threads at all for subinterpreter
compatibility, see https://bugs.python.org/issue39812. Instead of using an
atexit handler, there's some consideration for using a similar
threading-specific "exit handler". Instead of being upon program exit,
registered functions would be called just before all of the non-daemon
threads are joined in `threading._shutdown()`.

As a result, I suspect that we are unlikely to implement something like the
above. But, as a convenient side effect, it should somewhat address the
current potentially surprising behavior.

(Note that this is currently in early draft stages, so it could end up
being entirely different)

On Mon, Mar 9, 2020 at 5:39 PM Remy NOEL  wrote:

> I ran into a problem which, according to a google search is not that
> uncommon:
> concurrent.futures ThreadPoolExecutor, installs an exit handler
> preventing program exit if any of the jobs it is running is blocked. (Which
> might be a surprising behaviour to someone using those).
>
> It is possible to workaround this problem by either unregistering the exit
> handler concurrent.futures.thread._python_exit or by subclassing the
> threadpoolExecutor and overriding the _adjust_thread_count method with
> one that does not register its threads queues.
>
> Both seems kinda ugly though.
> It would seems reasonable to have a separate _daemon_thread_queues dict
> (in contrast to _threads_queues) that would not get joined upon exit.
> A "daemon" (or any better name) parameter of ThreadPoolExecutor
> initialisation would allow to let it add its workers in this dict rather
> than the default one.
>
> Does this seems feasible ?
>
> Thanks !
>
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/FH44JOI73IOCJ42QSI3YFU7S7RDWYYZI/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WK5AOBFVWTUHTGGO5HJMUPISTKPABHXG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Helper to show code to hide warnings

[Kyle Stanley]

> 2. Looking it up is hard. If I Google "python ignore warnings" the top
result is a Stack Overflow question where neither the accepted answer nor
the most upvoted answer mention specifying a module. The second Google
result is the Python docs which are not easy to read through.

Hmm, I think we may benefit from focusing the efforts on this point (at
least at first), particularly with regards to making the official
documentation for the warnings module [1] easier to understand with some
more examples, or perhaps writing a warnings HOWTO guide [2], which could
include the correct way to add filters for specific warning types by module
for a significant number of different filters.

>From my perspective, this seems to be more of an issue with a lack of
understanding and clear guides for using the existing tools, rather than a
direct indication that additional tools need to be added.

> 3. There's lots of similarly named methods and it's hard to find the
right one. Is it catch_warnings, simplefilter, or filterwarnings?

While I agree that the names might be a bit similar, I personally find that
the names accurately describe what each of them do, particularly
`filterwarnings()` and `catch_warnings()`. The purpose of `simplefilter()`
in relation to `filterwarnings()` might not be as obvious without reading
the documentation, but I don't think there's much that can reasonably be
done to address it. I also don't find it to be an especially significant
concern.

> 4. When warnings are displayed, they show the filename, but filters
require a module name, and converting between the two is tedious.

I could see it being useful to have a utility for filtering warnings by
filename rather than just module, if it's reasonably possible to implement.

> 5. The more warnings there are, and thus the more serious the need to
filter them properly, the more work it is to write all the filters.

This could be potentially addressed by adding an example in the warning
docs or a new warnings HOWTO that includes something along the lines of the
following:

```
modules_to_filter = {"foo": ResourceWarning, "bar": DeprecationWarning}
for mod, cat in modules_to_filter.items():
warnings.filterwarnings("ignore", category=cat, module=mod)
```

(I suspect the actual documented example would be much more fleshed out and
explained, the above was very quickly thrown together as a rough example)

Alternatively, we could consider adding a helper function to the warnings
module for adding a filter with the same action and warning category across
multiple modules at once; by accepting a collection of a strings instead of
a single string (like `filterwarnings()` does for it's *module* parameter).
I'm not certain if the implementation would be too trivial to justify a new
helper function for it, but the use case may be common enough for something
like that to be reasonable.

> I propose we add a built in mechanism that is easy to use and remember
which displays Python code that can be copy pasted.

I really don't like the idea of a builtin that specifically revolves around
the expectation of its output being copied and pasted as code elsewhere,
particularly from an API design perspective. Also, *if* we were to consider
something like this, it seems more appropriate to be included in the
warnings module rather than as a builtin.

> Does this seem like a good general idea?

-1 on the addition of the proposed builtin, but +1 on the general idea of
making the process of correctly adding a decent volume of warning filters
more clear.

---

[1] - https://docs.python.org/3/library/warnings.html
[2] - https://docs.python.org/3/howto/index.html

On Tue, Feb 25, 2020 at 4:04 PM Alex Hall  wrote:

> This is inspired by the [discussion on iterable strings](
> https://mail.python.org/archives/list/python-ideas@python.org/thread/WKEFHT4JYCL2PMZ5LB6HJRLVP3OGZI56/),
> although I think it has applications elsewhere.
>
> Sometimes programs emit warnings that need to be filtered out. Maybe
> they're coming from dependencies that we don't control, or maybe it's our
> own code but it's a problem that can't be fixed or isn't worth the trouble.
>
> But filtering warnings correctly isn't very easy. You generally shouldn't
> ignore all warnings, as this would hide important ones. Similarly, you
> probably shouldn't even filter an entire category, which is what many end
> up doing. A good compromise is to specify both the category and the module.
> In particular this lets you ignore warnings from libraries while still
> being able to see warnings from your own code. But there are obstacles on
> the way:
>
> 1. Remembering the syntax is hard. I've never been able to.
> 2. Looking it up is hard. If I Google "python ignore warnings" the top
> result is a Stack Overflow question where neither the accepted answer nor
> the most upvoted answer mention specifying a module. The second Google
> result is the Python docs which are not easy to read through.
> 3. There's lots of 

[Python-ideas] Re: Incremental step on road to improving situation around iterable strings

[Kyle Stanley]

In order for this proposal to be seriously considered, I think it's
necessary to cite many realistic examples where the current behavior is
problematic enough to justify changing the current behavior, and that
adding a str.chars() and eventually removing the ability to iterate over
strings would provide a better fix than any existing solutions. Simply
stating that much of the Python community considers the current behavior to
be "somewhat problematic" based on previous suggestions is certainly not
going to be adequately convincing, at least not to me.

But, even in the case that we are able to conclude that the current
behavior causes significant issues that can't be addressed as well with
existing solutions, I strongly suspect that this is going to be the case of
change that is far too fundamental to Python to be changed at this point,
even with a distant deprecation warning and many years of advanced notice
regarding the removal/change after that.

I could be wrong, but the level of potential breakage that would occur
seems to be on the level of Python 2 => 3 bytes changes (if not worse in
some ways); which I suspect very few people want to repeat. Not to mention
that pretty much all of the textbooks, documentation, and tutorials that
cover Python fundamentals or have examples that iterate over strings (which
is very common) would have to be updated. The argument in favor of changing
the current behavior at any point in time, regardless of how far away,
would have to be incredibly convincing.

For the time being, I'm -1 on both adding a chars() and an eventual
deprecation => removal of iterating over strings directly. At the present
moment, I don't think it's been clearly established that the current
behavior is even problematic or wrong in the first place, not to mention
problematic enough to justify a massive breaking change.

On Sat, Feb 22, 2020 at 7:28 PM Steve Jorgensen  wrote:

> From reading many of the suggestions in this list (including some of my
> own, of course) there seem to be a lot of us in the Python coder community
> who find the current behavior of strings being iterable collections of
> single-character strings to be somewhat problematic. At the same time,
> trying to change that would obviously have vast consequences, so not
> something that can simply be changed, even with at a major version boundary.
>
> I propose that we start by making a small, safe step in the right
> direction so we can maybe reach the bigger goal in a far future release
> (e.g. 5.x or 6.x).
>
> The step I propose is to add a `chars` method that returns a sequence
> (read-only view) of single-character strings that behaves exactly the same
> as `str` currently does when treated as a collection. Encourage developers
> to use that for iterating over characters in a string instead of iterating
> over the string directly. In maybe Python 4 or 5, directly using an `str`
> instance as a collection could become a warning.
>
> BTW, while adding `chars`, it might also be nice to have `ords` which
> would be a view of the string's character sequence as `ord` integer values.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/WKEFHT4JYCL2PMZ5LB6HJRLVP3OGZI56/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DNRCKE6XFUODNPLA57R7EFJOLB6WMVJT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Proposal: Complex comprehensions containing statements

[Kyle Stanley]

> One of the points of using a function is to not define it in the local
scope, but in some other namespace, so it can be reused and tested.

I consider the "tested" part to be of particular importance. Especially if
it were to become multiple layers deep, this can become a real issue.

This might be a bit subjective, but I've always considered list
comprehensions to have a similar purpose as lambda expressions: to be used
for small and easy to read operations, such as for single mapping,
substitution and/or filtering. I.E. "[f(item) if condition else 0 for item
in sequence]", "[func(item) for item in sequence if conditional]", or
"[item for index, item in enumerate(sequence) if not index % 10]". As such,
I tend to avoid and discourage the usage of complex nested comprehensions
[1].

The proposed syntax helps to address the readability issue of nested
comprehensions, in addition to allowing the usage of compound statements
such as "try-except" within them. But that doesn't address the *core issue*
of using "anonymous function"-like features [2] for complex operations: *they
are difficult to debug and have to be transformed in order to be directly
testable* [3]. This is especially problematic when they contain compound
statements.

Also, to provide an alternative perspective on the "existing syntax"
referenced in the OP:

new_matrix = []
for row in matrix:
def new_row():
for cell in row:
try:
yield f(cell)
except ValueError:
yield 0

new_matrix.append(list(new_row()))

I personally find this much cleaner:

def create_matrix(matrix):
# Emphasis on not having the function definition inside of the for loop.
# gen_row() can be defined inside of the function if you're certain you
won't
# need it again, but this makes it easier to move gen_row() to its own
definition
# later if it becomes more complex and you want to test create_matrix()
and
# gen_row() separately.
def gen_row(row):
for cell in row:
try:
yield f(cell)
except ValueError:
yield 0

new_matrix = []
for row in matrix:
new_row = list(gen_row(row))
new_matrix.append(new_row)
return new_matrix

But the above could also be made more succinct using the existing list comp
syntax. Something like this is probably along the lines of what I'd use in
my own code:

def create_matrix(matrix):
def gen_row(row):
for cell in row:
try:
yield f(cell)
except ValueError:
yield 0

# "tuple(gen_row(row))" would be a bit better if the rows have a fixed
# of cells, which is common.
return [list(gen_row(row)) for row in matrix]

This is a much more fair comparison, and I personally prefer the above over
the proposed syntax:

new_matrix = [
for row in matrix: [
for cell in row:
try:
yield f(cell)
except ValueError:
yield 0
]
]

As for the the argument of making complex comprehensions less confusing, *I
would consider the potentially confusing usage of complex comprehensions
referenced in the OP to be a misuse of the feature in the first place*. If
a comprehension is starting to look like it might confuse readers, it's a
sign that it should be refactored into it's own dedicated function. Of
course, this isn't to say that any code which isn't immediately clear to
Python users of all experience should be refactored. But, if the current
version might be confusing to many readers and it can made more clear
without any loss in functionality, performance, or design quality,
refactoring is often the right move. Something like this:

[
f(x)
for y in z
for x in y
if g(x)
]

Can easily be refactored into something like this:

def gen_foo(z):
for y in z:
for x in y:
if g(x):
yield f(x)

return list(gen_foo(z))

Similar to the previous set of examples, I think the above is a much more
fair comparison to the existing syntax, and I personally prefer it over the
proposed syntax.

IMO, there's pretty much zero practical benefit in using the former version
compared to the above. As far as development time goes, you may spend a
tiny bit more time thinking of a useful name for "gen_foo", but a good name
can help greatly in making the purpose of the code more clear. So, I can't
really see that as a downside.

With all of the above in mind, I'm -1 on the proposal; I don't see a
substantial enough benefit from using the proposed syntax compared to the
already existing syntax. I'll admit that it has some appeal regarding
convenience, but it would very likely encourage anti-patterns (especially
for those with beginner and intermediate level experience with Python).

---

[1] - Although I'm not an 

[Python-ideas] Re: PYTHONLOGGING env variable

[Kyle Stanley]

> Let's plan a cmdline argument then.

> Should we go for -L much like -W?

Yeah, I think a command-line argument is much more reasonable, less likely
to cause issues with other implementations of logging configuration, and
more practically useful. I'd be +1 on something like a `-L`.

On Thu, Feb 20, 2020 at 2:05 PM Bar Harel  wrote:

> Let's plan a cmdline argument then.
>
> Should we go for -L much like -W?
> Or -x logging for setting a default value of debug or info?
>
> Per python specification, -x is implementation dependent.
>
> I'll advocate for -L. I think the same concerns as -W are relevant for all
> python implementations, and it gives more flexibility in terms of log level.
>
> Next question: Should -x dev include it?
> If it was python day-1, I would say yes. My issue is the double inclusion
> of stream handlers in case the logging was bootstrapped already.
> So -x dev probably shouldn't include it.
>
> On Thu, Feb 20, 2020, 8:28 PM Christopher Barker 
> wrote:
>
>> I think it’s a “Bad Idea” to use an environment variable — who knows what
>> Python script may be running on a given system?
>>
>> But a standard command line argument to the interpreter could be useful.
>>
>> -CHB
>>
>> On Thu, Feb 20, 2020 at 8:51 AM Mike Miller 
>> wrote:
>>
>>>
>>> On 2020-02-19 17:12, Bar Harel wrote:
>>> > Another idea I've had that may be of use:
>>> >
>>> > PYTHONLOGGING environment variable.
>>> >
>>> > Setting PYTHONLOGGING to any log level or level name will initialize
>>> > logging.basicConfig() with that appropriate level.
>>> >
>>> > Another option would be that -x dev or a different -x logging will
>>> initialize
>>> > basic config.
>>>
>>> As a heavy user of logging, I like this one a lot.  In a new project,
>>> the lack
>>> of such a feature allows one to stick with print() perhaps longer than
>>> is prudent.
>>>
>>> -Mike
>>> ___
>>> Python-ideas mailing list -- python-ideas@python.org
>>> To unsubscribe send an email to python-ideas-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-ideas@python.org/message/US4RHU5S7MAEUJG7MPQIFZWM324IXNB6/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> --
>> Christopher Barker, PhD
>>
>> Python Language Consulting
>>   - Teaching
>>   - Scientific Software Development
>>   - Desktop GUI and Web Development
>>   - wxPython, numpy, scipy, Cython
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/YTYC4XPF5SWGWYI4DIGCXIEA53HMTNQU/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/KGRSK4U2L4MNMY57IG3XD52WHLOWG6OG/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XBLAIDGK36WBXBMZQGXIJ2WCS3VQ2DSE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: PYTHONLOGGING env variable

[Kyle Stanley]

[re-sending this since I forgot to CC the list]

> Unfortunately, there is no logging.loglevel attribute.

> At the moment, the only way of doing this is modifying the program,
parsing input parameters and setting up the logging system.

Indeed, that's why I clarified in the follow-up message; sorry it's been a
long day :-).

"loglevel" was simply an arbitrary name chosen for that particular example
and getattr() was used as a way to prefix the module so it would access the
logging level constant instead of the literal name of the level as a string
(which would be invalid to pass to level). You'd likely use argparse or
some other command-line argument parsing utility to get it in the first
place. For example:


parser = argparse.ArgumentParser()
parser.add_argument("--log", dest='logging_level', default='WARNING')
args = parser.parse_args()

numeric_level = getattr(logging, args.logging_level.upper())
logging.basicConfig(level=numeric_level, ...)


(input validation omitted for simplification)

> If you want it only temporarily for debugging purposes, or permanently
for all scripts in your system, it's not really feasible.

Hmm, I'm not sure that I like the idea of overriding the default logging
level being set system-wide, or that there would be an especially strong
use case for it. Instead, it seems more practically common to want to set
it on an individual run of a program temporarily. In such cases, using some
form of command-line parsing utility is a perfectly a viable option.

> We have debugging flags for warnings, faulthandlers, asyncio etc...
Logging I would say is the most important and trivial of all, yet we don't
have a flag for it.

Arguably though, the easier it is to implement on your own, the less of a
need there is to be an environment variable to do it. Also, I think the
logging configuration should be on a per-program basis, rather than
globally (which is what env vars are typically used for).

FWIW I'm not opposed to the principle behind the idea, I'm just not
convinced yet that it would be that much better than the options that are
already available, and would be worthwhile to add an entirely new
environment variable for (or be good usage of an env var for that matter).
Perhaps there are other areas to explore for simplifying the process of
changing the logging level from the command-line though.

Also, although not the same proposal, there was a discussion that brought
up some similar points in a python-ideas thread back in 2014:
https://mail.python.org/archives/list/python-ideas@python.org/thread/7ZLMSQU2C65YZ2KVZKZ6MICPTWVFZVJ3/#GCJZGNJOO27WUWKN2DKUP3D6HLYNLFUI.
I think some of the points made in there are relevant to this discussion.

On Wed, Feb 19, 2020 at 10:26 PM Bar Harel  wrote:

> Thanks Kyle for the feedback :-)
>
> Unfortunately, there is no logging.loglevel attribute.
>
> At the moment, the only way of doing this is modifying the program,
> parsing input parameters and setting up the logging system.
>
> If you want it only temporarily for debugging purposes, or permanently for
> all scripts in your system, it's not really feasible.
>
> Think of -Wdefault. You can change settings in the warnings module and
> modify files each time you wish to enable all warnings. But then again, why
> would you?
>
> We have debugging flags for warnings, faulthandlers, asyncio etc...
> Logging I would say is the most important and trivial of all, yet we don't
> have a flag for it.
>
> On Thu, Feb 20, 2020, 4:24 AM Kyle Stanley  wrote:
>
>> > This can already be done from the command line through --log=LEVEL,
>> which sets the logging.loglevel attribute. In the how-to section of the
>> docs, this is demonstrated in the following example:
>>
>> Note: "loglevel" is an arbitrary name, but this can be implemented
>> trivially with something like argparse to check for a --log argument.
>>
>> On Wed, Feb 19, 2020 at 9:19 PM Kyle Stanley  wrote:
>>
>>> > Setting PYTHONLOGGING to any log level or level name will initialize
>>> logging.basicConfig() with that appropriate level.
>>>
>>> This can already be done from the command line through --log=LEVEL,
>>> which sets the logging.loglevel attribute. In the how-to section of the
>>> docs, this is demonstrated in the following example:
>>>
>>> # assuming loglevel is bound to the string value obtained from the# command 
>>> line argument. Convert to upper case to allow the user to# specify 
>>> --log=DEBUG or --log=debugnumeric_level = getattr(logging, 
>>> loglevel.upper(), None)if not isinstance(numeric_level, int):
>>> raise ValueError('Invalid log level: %s' % 
>>> loglevel)logging.basicConfig(level=numeric_level, ...)
>>>
>>> (https://docs.python.org/3/

[Python-ideas] Re: PYTHONLOGGING env variable

[Kyle Stanley]

> This can already be done from the command line through --log=LEVEL, which
sets the logging.loglevel attribute. In the how-to section of the docs,
this is demonstrated in the following example:

Note: "loglevel" is an arbitrary name, but this can be implemented
trivially with something like argparse to check for a --log argument.

On Wed, Feb 19, 2020 at 9:19 PM Kyle Stanley  wrote:

> > Setting PYTHONLOGGING to any log level or level name will initialize
> logging.basicConfig() with that appropriate level.
>
> This can already be done from the command line through --log=LEVEL, which
> sets the logging.loglevel attribute. In the how-to section of the docs,
> this is demonstrated in the following example:
>
> # assuming loglevel is bound to the string value obtained from the# command 
> line argument. Convert to upper case to allow the user to# specify 
> --log=DEBUG or --log=debugnumeric_level = getattr(logging, loglevel.upper(), 
> None)if not isinstance(numeric_level, int):
> raise ValueError('Invalid log level: %s' % 
> loglevel)logging.basicConfig(level=numeric_level, ...)
>
> (https://docs.python.org/3/howto/logging.html#logging-to-a-file)
>
> Is there a practical reason why the above doesn't work for you? Otherwise, I 
> see no reason to add a new environment variable that effectively does the 
> same thing.
>
>
> On Wed, Feb 19, 2020 at 8:16 PM Bar Harel  wrote:
>
>> Another idea I've had that may be of use:
>>
>> PYTHONLOGGING environment variable.
>>
>> Setting PYTHONLOGGING to any log level or level name will initialize
>> logging.basicConfig() with that appropriate level.
>>
>> Another option would be that -x dev or a different -x logging will
>> initialize basic config.
>>
>> Will be useful mostly for debugging purposes instead of temporarily
>> modifying the code.
>>
>> Kinda surprised it doesn't exist tbh.
>>
>> Bar Harel
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/I74LVJWJLE2LUCCZGOF5A5JDSDHJ6WX2/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CWFNU7ENCH7N25IIIT4NORS2V7K6WENJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: PYTHONLOGGING env variable

[Kyle Stanley]

> Setting PYTHONLOGGING to any log level or level name will initialize
logging.basicConfig() with that appropriate level.

This can already be done from the command line through --log=LEVEL, which
sets the logging.loglevel attribute. In the how-to section of the docs,
this is demonstrated in the following example:

# assuming loglevel is bound to the string value obtained from the#
command line argument. Convert to upper case to allow the user to#
specify --log=DEBUG or --log=debugnumeric_level = getattr(logging,
loglevel.upper(), None)if not isinstance(numeric_level, int):
raise ValueError('Invalid log level: %s' %
loglevel)logging.basicConfig(level=numeric_level, ...)

(https://docs.python.org/3/howto/logging.html#logging-to-a-file)

Is there a practical reason why the above doesn't work for you?
Otherwise, I see no reason to add a new environment variable that
effectively does the same thing.


On Wed, Feb 19, 2020 at 8:16 PM Bar Harel  wrote:

> Another idea I've had that may be of use:
>
> PYTHONLOGGING environment variable.
>
> Setting PYTHONLOGGING to any log level or level name will initialize
> logging.basicConfig() with that appropriate level.
>
> Another option would be that -x dev or a different -x logging will
> initialize basic config.
>
> Will be useful mostly for debugging purposes instead of temporarily
> modifying the code.
>
> Kinda surprised it doesn't exist tbh.
>
> Bar Harel
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/I74LVJWJLE2LUCCZGOF5A5JDSDHJ6WX2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VMT7NWEO7HWBLAJYM4YWJGLM4PK7KA7E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add __keys__ or __items__ protocol

[Kyle Stanley]

 Serhiy Storchaka wrote:
> 1. What special method should be added, `__keys__` or `__items__`? The
> former returns keys, it needs to call `__getitem__` repeatedly to get
> values. The latter returns key-value pairs, it does not need to call
> `__getitem__`, but you should always pay the cost of creating a tuple
> even if you do not need values.

Between __keys__ and __items__, I find that __keys__ is specifically more
unique to mapping types; whereas __items__ can be reasonably applied to any
type of container. Also, I personally find that I more frequently have the
need to access just all of the keys and some or none of the values, rather
than all of the key and value pairs. That being said, if the consensus ends
up being to use __items__ instead, I would be okay with that.

Serhiy Storchaka wrote:
> 2. What type should they return?

> * An iterator.

+0.

> * An iterable which can be iterated only once. Easy to implement in
> Python as generator methods.

-1.

> * An iterable which can be iterated multiple times.

+1.

> * More complex view object which may support other protocols (for
> example support `__or__` and `__and__`).

Strong -1.

I find that the requirement of an iterable that can be iterated multiple
times would be the most reasonable option, as this would be compatible with
the existing dict.keys(). A simple iterator seems a bit too generic and
similar to __iter__, and requiring it to be an iterable that can be
iterated over only once would be incompatible with dict.keys() (which seems
rather strange).

As for requiring a view object...

Steven D'Aprano wrote:
> That means that every object wanting to make use of this protocol needs
> to create a complex set-like view object, which is neither easy nor
> obvious, and may not even be appropriate for some classes.

Mainly for the above, I'm -1 on requirement of returning a view; that seems
needlessly restrictive and would significantly limit the dunder method's
usefulness for any user-created mappings which don't inherit from dict. If
it's intended to be used for any mapping object, it should be as
(reasonably) simple to implement as possible, IMO.



On Tue, Feb 18, 2020 at 7:26 PM Steven D'Aprano  wrote:

> On Tue, Feb 18, 2020 at 11:47:16AM -0800, Ethan Furman wrote:
>
> > Whatever `dict.items()` returns, both for consistency and because
> > `dict.items()` can become a simple alias for `dict.__items__`.
>
> That means that every object wanting to make use of this protocol needs
> to create a complex set-like view object, which is neither easy nor
> obvious, and may not even be appropriate for some classes.
>
>
>
> --
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/YZCK3XLRLV4G65EZGXQB5PU3IBA25TXP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UL35QYPRLMJHCBPJPCNIQEUOJR5BBVZ7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol

[Kyle Stanley]

> Isn’t the point of having a Code of Conduct that we shouldn’t have to
have these arguments? Or, if there is an argument about whether a single
figurative use as a word qualifies as “excessive swearing” per the CoC,
surely an -ideas thread isn’t the place to have it?

Yeah, I very much doubt that "shitty wrapper" constitutes as excessive
swearing. It'd be one thing if the post contained many or was directed at
someone else, but the OP was using it to refer to something they personally
wrote. Personally, I think it's a non-issue in that sort of context.

On Tue, Feb 18, 2020 at 10:40 PM Andrew Barnert via Python-ideas <
python-ideas@python.org> wrote:

> On Feb 18, 2020, at 17:11, Ethan Furman  wrote:
> >
> > On 02/18/2020 04:26 PM, Steven D'Aprano wrote:
> >> On Tue, Feb 18, 2020 at 08:44:07PM +, Rhodri James wrote:
> >
> >>> Language, sunshine.
> >> What about it?
> >> Just for the record, I had no issues with Soni's language, but I do
> >> object to attempts to shame him for it.
> >
> > Since when is simply drawing one's attention to something shaming?
> >
> >> This isn't the Disney Chanel,
> >> it's been over 100 years since Victoria was Queen of England, and we're
> >> all adults here.
> >
> > Really?  I didn't realize those under the legal age of majority were not
> allowed on this list.
>
> Isn’t the point of having a Code of Conduct that we shouldn’t have to have
> these arguments? Or, if there is an argument about whether a single
> figurative use as a word qualifies as “excessive swearing” per the CoC,
> surely an -ideas thread isn’t the place to have it?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/GOKPFN55FTAGLB7WJ6B6MOUJ3FLDWBZ7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QFBU42VVYYTC4IHEZ2HVRAB7AYJO2VXD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

[Kyle Stanley]

> Anyway, I think the spelled-out “Synchronous” may be a better name, to
avoid the (very likely) case of people mistakenly reading “Sync” as short
for “Synchronized”. It’s no longer than “ProcessPool”, and, although it is
easy to typo, tab-completion or copy-paste helps, and how many times do you
need to type it anyway? And there will always be more readers than writers,
and it’s more likely the writers will be familiar with the futures module
contents than the readers. And IIRC, this is the name Scala uses.

> Maybe “Serial” is ok too, but to me that implies serialized on a queue,
probably using a single background thread. That’s the naming used in the
third-party C++ and ObjC libs I’ve used most recently, and it may be more
common than that—but it may not, in which case my reading may be
idiosyncratic and not worth worrying about.

FWIW, I'm also in favor of SynchronousExecutor. I find that the term
"Serial" has a bit too many definitions depending on the context; whereas
"Synchronous" is very clear as to the behavior and purpose of the executor.
I'd rather the class name to be excessively verbose and more immediately
obvious as to what it does; rather than shorter to type and a bit ambiguous.

On Mon, Feb 17, 2020 at 9:05 PM Andrew Barnert via Python-ideas <
python-ideas@python.org> wrote:

> On Feb 17, 2020, at 15:41, Jonathan Crall  wrote:
> >
> > FWIW I found the term "SyncExecutor" really confusing when I was reading
> this thread. I thought it was short for Synchonized, but I just realized
> its actually short for Synchronous, which makes much more sense. While
> SynchronousExecutor makes more sense to me, it is also more verbose and
> difficult to spell.
>
> I think that’s my fault—I switched from “serial” to “sync” in the middle
> of a message without even realizing It, probably borrowed from an ObjC
> library I used recently.
>
> Anyway, I think the spelled-out “Synchronous” may be a better name, to
> avoid the (very likely) case of people mistakenly reading “Sync” as short
> for “Synchronized”. It’s no longer than “ProcessPool”, and, although it is
> easy to typo, tab-completion or copy-paste helps, and how many times do you
> need to type it anyway? And there will always be more readers than writers,
> and it’s more likely the writers will be familiar with the futures module
> contents than the readers. And IIRC, this is the name Scala uses.
>
> Maybe “Serial” is ok too, but to me that implies serialized on a queue,
> probably using a single background thread. That’s the naming used in the
> third-party C++ and ObjC libs I’ve used most recently, and it may be more
> common than that—but it may not, in which case my reading may be
> idiosyncratic and not worth worrying about.
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/5KVFCF7S2EBRL2RMZMDMQLA562CM5IG7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4HKKXTLKTHZBPMF4J57UVX7S3NAU77A3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

[Kyle Stanley]

> I'm much more lukewarm on set_state().  How hard is it to reimplement
> one's own Future if someone wants a different implementation?  By
> allowing people to change the future's internal state, we're also
> giving them a (small) gun to shoot themselves with.

Yeah, I don't feel quite as strongly about future.set_state(). My primary
motivation was to work on a complete means of extending the Future class
through a public API, starting with the future's state. But it might be too
(potentially) detrimental for the average user to be worth the more niche
case of being able to extend Future without needing to use the private
members.

Upon further consideration, I think it would be better to stick with
future.state() for now, since it has more of a general-purpose use case. If
it's specifically documented in similar manner to queue.qsize(); stating
something along the lines of "Return the approximate state of the future.
Note that this state is only advisory, and is not guaranteed."

> No strong opinion on this, but it sounds ok.  That means
> `future.state()` would return an enum value, not a bare string?

Yeah, presumably with each using auto() for the value. For simplified
review purposes though, I'll likely do these in separate PRs, but attached
to the same bpo issue: https://bugs.python.org/issue39645. I'll also update
the issue to reduce the scope a bit (mainly removing future.set_state()).

On Mon, Feb 17, 2020 at 5:08 AM Antoine Pitrou  wrote:

> On Sun, 16 Feb 2020 19:46:13 -0500
> Kyle Stanley  wrote:
> >
> > Based on the proposal in the OP, I had considered that it might also be
> > needed to be able to manually set the state of the future through
> something
> > like a `Future.set_state()`, which would have a parameter for accessing
> it
> > safely through the condition's RLock, and another without it (in case
> they
> > want to specify their own, such as in the OP's example code).
>
> I'm much more lukewarm on set_state().  How hard is it to reimplement
> one's own Future if someone wants a different implementation?  By
> allowing people to change the future's internal state, we're also
> giving them a (small) gun to shoot themselves with.
>
> > Lastly, it seemed also useful to be able to publicly use the future state
> > constants. This isn't necessary for extending them, but IMO it would look
> > better from an API design perspective to use
> `future.set_state(cf.RUNNING)`
> > instead of `future.set_state(cf._base.RUNNING)` or
> > `future.set_state("running") [1].
>
> No strong opinion on this, but it sounds ok.  That means
> `future.state()` would return an enum value, not a bare string?
>
> Regards
>
> Antoine.
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/APZRZPV5YPBQ7TOERXFODISMUII2MOUL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WI6NWWOT4BCXNZNNGX53KM64LSC2PPKJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

[Kyle Stanley]

> Hm, but doesn't the OP's example require *synchronously* reading and
writing the state?

Correct. But in the OP's example, they wanted to use their own
"FakeCondition" for reading and writing the state, rather than the
executor's internal condition (which is bypassed when you directly access
or modify the state through future._state instead of the public methods).
That's why I proposed to add something like future.state(). In the case of
the OP's example, they would presumably access future.state() through
"FakeCondition".

Or am I misunderstanding something?

On Mon, Feb 17, 2020 at 12:03 AM Guido van Rossum  wrote:

> Hm, but doesn't the OP's example require *synchronously* reading and
> writing the state?
>
> On Sun, Feb 16, 2020 at 4:47 PM Kyle Stanley  wrote:
>
>> > That sounds useful to me indeed.  I assume you mean something like a
>> > state() method?  We already have Queue.qsize() which works a bit like
>> > this (unlocked and advisory).
>>
>> Yep, a `Future.state()` method is exactly what I had in mind! I hadn't
>> considered that `Queue.qsize()` was analogous, but that's a perfect example.
>>
>> Based on the proposal in the OP, I had considered that it might also be
>> needed to be able to manually set the state of the future through something
>> like a `Future.set_state()`, which would have a parameter for accessing it
>> safely through the condition's RLock, and another without it (in case they
>> want to specify their own, such as in the OP's example code).
>>
>> Lastly, it seemed also useful to be able to publicly use the future state
>> constants. This isn't necessary for extending them, but IMO it would look
>> better from an API design perspective to use `future.set_state(cf.RUNNING)`
>> instead of `future.set_state(cf._base.RUNNING)` or
>> `future.set_state("running") [1].
>>
>> Combining the above, this would look something like
>> `future.set_state(cf.FINISHED)`, instead of the current private means of
>> modifying them with `future._state = cf._base.FINISHED` or `future._state =
>> "finished"`.
>>
>> Personally, I'm most strongly in favor of adding Future.state(), as it
>> would be personally useful for me (for reasons previously mentioned); but I
>> think that the other two would be useful for properly extending the Future
>> class without having to access private members. This was more formally
>> proposed in https://bugs.python.org/issue39645.
>>
>>
>> [1] - Setting running was just an example, although normally that would
>> be just done in the executor through `Future.set_running_or_notify_cancel
>> ()`.
>>
>> On Sun, Feb 16, 2020 at 6:00 PM Antoine Pitrou 
>> wrote:
>>
>>> On Sun, 16 Feb 2020 17:41:36 -0500
>>> Kyle Stanley  wrote:
>>> >
>>> > As a side note, are we still interested in expanding the public API
>>> for the
>>> > Future class? Particularly for a public means of accessing the state.
>>> The
>>> > primary motivation for it was this topic, but I could easily the same
>>> > issues coming up with custom Future and Executor classes; not to
>>> mention
>>> > the general debugging usefulness for being able to log the current
>>> state of
>>> > the future (without relying on private members).
>>>
>>> That sounds useful to me indeed.  I assume you mean something like a
>>> state() method?  We already have Queue.qsize() which works a bit like
>>> this (unlocked and advisory).
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>> ___
>>> Python-ideas mailing list -- python-ideas@python.org
>>> To unsubscribe send an email to python-ideas-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-ideas@python.org/message/7ZQE3IB4NR7ZPLLKWIY54PW3X5K6YWUF/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/BT5AKV4AOACU6TFYI6MIQXC6RK7BQFEK/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JQWCUIV5VLKAI54TVGNZQTR72R4LVK4E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

[Kyle Stanley]

> That sounds useful to me indeed.  I assume you mean something like a
> state() method?  We already have Queue.qsize() which works a bit like
> this (unlocked and advisory).

Yep, a `Future.state()` method is exactly what I had in mind! I hadn't
considered that `Queue.qsize()` was analogous, but that's a perfect example.

Based on the proposal in the OP, I had considered that it might also be
needed to be able to manually set the state of the future through something
like a `Future.set_state()`, which would have a parameter for accessing it
safely through the condition's RLock, and another without it (in case they
want to specify their own, such as in the OP's example code).

Lastly, it seemed also useful to be able to publicly use the future state
constants. This isn't necessary for extending them, but IMO it would look
better from an API design perspective to use `future.set_state(cf.RUNNING)`
instead of `future.set_state(cf._base.RUNNING)` or
`future.set_state("running") [1].

Combining the above, this would look something like
`future.set_state(cf.FINISHED)`, instead of the current private means of
modifying them with `future._state = cf._base.FINISHED` or `future._state =
"finished"`.

Personally, I'm most strongly in favor of adding Future.state(), as it
would be personally useful for me (for reasons previously mentioned); but I
think that the other two would be useful for properly extending the Future
class without having to access private members. This was more formally
proposed in https://bugs.python.org/issue39645.


[1] - Setting running was just an example, although normally that would be
just done in the executor through `Future.set_running_or_notify_cancel()`.

On Sun, Feb 16, 2020 at 6:00 PM Antoine Pitrou  wrote:

> On Sun, 16 Feb 2020 17:41:36 -0500
> Kyle Stanley  wrote:
> >
> > As a side note, are we still interested in expanding the public API for
> the
> > Future class? Particularly for a public means of accessing the state. The
> > primary motivation for it was this topic, but I could easily the same
> > issues coming up with custom Future and Executor classes; not to mention
> > the general debugging usefulness for being able to log the current state
> of
> > the future (without relying on private members).
>
> That sounds useful to me indeed.  I assume you mean something like a
> state() method?  We already have Queue.qsize() which works a bit like
> this (unlocked and advisory).
>
> Regards
>
> Antoine.
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/7ZQE3IB4NR7ZPLLKWIY54PW3X5K6YWUF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/BT5AKV4AOACU6TFYI6MIQXC6RK7BQFEK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

[Kyle Stanley]

> I don't think we need to be dogmatic here.  If someone wants to provide
> it on PyPI, then be it.  But if they'd rather contribute it to the
> stdlib, we should examine the relevant PR at face value.

> Asking it to be exercised first on PyPI is worthwhile if the domain
> space is complex or there are multiple possible APIs.  It's not really
> the case here: the API is basically constrained (it must be an
> Executor) and the main unknown seems to be whether execution is lazily
> or immediate (which may be as well governed by a constructor
> parameter).  And the implementation shouldn't be very hairy either :-)

Alright, fair enough. I suppose that I hadn't adequately considered how
constrained the API and straightforward the implementation would likely be.
If you think it would very likely receive widespread enough usage to
justify adding and maintaining it to the stdlib, I fully trust your
judgement on that. (:

As a side note, are we still interested in expanding the public API for the
Future class? Particularly for a public means of accessing the state. The
primary motivation for it was this topic, but I could easily the same
issues coming up with custom Future and Executor classes; not to mention
the general debugging usefulness for being able to log the current state of
the future (without relying on private members).

On Sun, Feb 16, 2020 at 9:49 AM Antoine Pitrou  wrote:

> On Sun, 16 Feb 2020 09:29:36 -0500
> Kyle Stanley  wrote:
> >
> > After Andrew explained his own use case for it with isolating bugs to
> > ensure that the issue wasn't occurring as a result of parallelism,
> threads,
> > processes, etc; I certainly can see how it would be useful. I could also
> > see a use case in a CLI tool for a conveniently similar parallel and
> > non-parallel version, although I'd likely prefer just having an entirely
> > separate implementation. Particularly if the parallel version includes
> > diving a large, computationally intensive task into many sub-tasks (more
> > common for PPE), that seems like it could result in significant
> additional
> > unneeded overhead for the non-parallel version.
> >
> > I think at this point, it's potential usefulness is clear though. But,
> IMO,
> > the main question is now the following: would it be better *initially*
> > placed in the standard library or on PyPI (which could eventually
> > transition into stdlib if it sees widespread usage)?
>
> I don't think we need to be dogmatic here.  If someone wants to provide
> it on PyPI, then be it.  But if they'd rather contribute it to the
> stdlib, we should examine the relevant PR at face value.
>
> Asking it to be exercised first on PyPI is worthwhile if the domain
> space is complex or there are multiple possible APIs.  It's not really
> the case here: the API is basically constrained (it must be an
> Executor) and the main unknown seems to be whether execution is lazily
> or immediate (which may be as well governed by a constructor
> parameter).  And the implementation shouldn't be very hairy either :-)
>
> Regards
>
> Antoine.
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/U5AOBMMGIANXFJEKMZYMWNQSY7D6RPL5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TPT67BK6XWUMYBXVAYVJYL46HXECNCPI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

[Kyle Stanley]

> FWIW, I agree with Andrew here.  Being able to swap a
> ThreadPoolExecutor or ProcessPoolExecutor with a serial version using
> the same API can have benefits in various situations. One is
> easier debugging (in case the problem you have to debug isn't a race
> condition, of course :-)).  Another is writing a library a command-line
> tool or library where the final decision of whether to parallelize
> execution (e.g. through a command-line option for a CLI tool) is up
> to the user, not the library developer.

After Andrew explained his own use case for it with isolating bugs to
ensure that the issue wasn't occurring as a result of parallelism, threads,
processes, etc; I certainly can see how it would be useful. I could also
see a use case in a CLI tool for a conveniently similar parallel and
non-parallel version, although I'd likely prefer just having an entirely
separate implementation. Particularly if the parallel version includes
diving a large, computationally intensive task into many sub-tasks (more
common for PPE), that seems like it could result in significant additional
unneeded overhead for the non-parallel version.

I think at this point, it's potential usefulness is clear though. But, IMO,
the main question is now the following: would it be better *initially*
placed in the standard library or on PyPI (which could eventually
transition into stdlib if it sees widespread usage)?

> It seems there are two possible design decisions for a serial executor:
> - one is to execute the task immediately on `submit()`
> - another is to execute the task lazily on `result()`

To me, it seems like the latter would be more useful for debugging
purposes, since that would be more similar to how the submitted
task/function would actually be executed. ``submit()`` could potentially
"fake" the process of scheduling the execution of the function, but without
directly executing it; perhaps with something like this:
``executor.submit()`` => create a pending item => add pending item to dict
=> add callable to call queue => fut.result() => check if in pending items
=> get from top of call queue  => run work item => pop from pending items
=> set result/exception => return result (skip last three if fut is not
in/associated with a pending item). IMO, that would be similar enough to
the general workflow followed in the executors without any of the
parallelization.

On Sun, Feb 16, 2020 at 6:29 AM Antoine Pitrou  wrote:

> On Sat, 15 Feb 2020 14:16:39 -0800
> Andrew Barnert via Python-ideas
>  wrote:
> > > On Feb 15, 2020, at 13:36, Jonathan Crall  wrote:
> > >
> > > Also, there is no duck-typed class that behaves like an executor, but
> does its processing in serial. Often times a develop will want to run a
> task in parallel, but depending on the environment they may want to disable
> threading or process execution. To address this I use a utility called a
> `SerialExecutor` which shares an API with
> ThreadPoolExecutor/ProcessPoolExecutor but executes processes sequentially
> in the same python thread:
> >
> > This makes sense. I think most futures-and-executors frameworks in other
> languages have a serial/synchronous/immediate/blocking executor just like
> this. (And the ones that don’t, it’s usually because they have a different
> way to specify the same functionality—e.g., in C++, you only use executors
> via the std::async function, and you can just pass a launch option instead
> of an executor to run synchronously.)
>
> FWIW, I agree with Andrew here.  Being able to swap a
> ThreadPoolExecutor or ProcessPoolExecutor with a serial version using
> the same API can have benefits in various situations.  One is
> easier debugging (in case the problem you have to debug isn't a race
> condition, of course :-)).  Another is writing a library a command-line
> tool or library where the final decision of whether to parallelize
> execution (e.g. through a command-line option for a CLI tool) is up
> to the user, not the library developer.
>
> It seems there are two possible design decisions for a serial executor:
> - one is to execute the task immediately on `submit()`
> - another is to execute the task lazily on `result()`
>
> This could for example be controlled by a constructor argument to
> SerialExecutor.
>
> Regards
>
> Antoine.
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/PCDN4JMKR7VCWXTEZSMWWIY55NTT3JOM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

[Kyle Stanley]

I opened a bpo issue to expand upon the public API for cf.Future:
https://bugs.python.org/issue39645. Any feedback would be greatly
appreciated. (:

On Sun, Feb 16, 2020 at 12:22 AM Guido van Rossum  wrote:

>
>
> On Sat, Feb 15, 2020 at 21:00 Kyle Stanley  wrote:
>
>> > I've never felt the need for either of these myself, nor have I
>> observed it in others I worked with. In general I feel the difference
>> between processes and threads is so large that I can't believe a realistic
>> application would work with either.
>>
>> Also, ThreadPoolExecutor and ProcessPoolExecutor both have their specific
>> purposes in concurrent.futures: TPE for IO-bound parallelism, and PPE for
>> CPU-bound parallelism, what niche would the proposed SerialExecutor fall
>> under? Fake/dummy parallelism? If so, I personally don't see that as being
>> worth the cost of adding it and then maintaining it in the standard
>> library. But, that's not to say that it wouldn't have a place on PyPI.
>>
>> > (Then again I've never had much use for ProcessExecutor period.)
>>
>> I've also made use of TPE far more times than PPE, but I've definitely
>> seen several interesting and useful real-world applications of PPE.
>> Particularly with image processing. I can also imagine it also being quite
>> useful for scientific computing, although I've not personally used it for
>> that purpose.
>>
>> > IOW I'm rather lukewarm about this -- even if you (Jonathan) have found
>> use for it, I'm not sure how many other people would use it, so I doubt
>> it's worth adding it to the stdlib. (The only thing the stdlib might grow
>> could be a public API that makes implementing this feasible without
>> overriding private methods.)
>>
>> Expanding a bit upon the public API for the cf.Future class would likely
>> allow something like this to be possible without accessing any private
>> members. In particular, I believe there would have to be an public means of
>> accessing the state of the future without having to go through the
>> condition (currently, this can only be done with ``future._state``), and
>> accessing a constant for each of the possible states: PENDING. RUNNING,
>> CANCELLED, CANCELLED_AND_NOTIFIED, and FINISHED.
>>
>> Since that would actually be quite useful for debugging purposes (I had
>> to access ``future._state`` several times while testing the new
>> *cancel_futures*), I'd be willing to work on implementing something like
>> this.
>>
>
> Excellent!
>
>
>>
>> On Sat, Feb 15, 2020 at 10:16 PM Guido van Rossum 
>> wrote:
>>
>>> Having tried my hand at a simpler version for about 15 minutes, I see
>>> the reason for the fiddly subclass of Future -- it seems over-engineered
>>> because concurrent.future is complicated.
>>>
>>> I've never felt the need for either of these myself, nor have I observed
>>> it in others I worked with. In general I feel the difference between
>>> processes and threads is so large that I can't believe a realistic
>>> application would work with either. (Then again I've never had much use for
>>> ProcessExecutor period.)
>>>
>>> The "Serial" variants somehow remind me of the "dummy_thread.py" module
>>> we had in Python 2. It was removed in Python 3, mostly because we ran out
>>> of cases where real threads weren't an option.
>>>
>>> IOW I'm rather lukewarm about this -- even if you (Jonathan) have found
>>> use for it, I'm not sure how many other people would use it, so I doubt
>>> it's worth adding it to the stdlib. (The only thing the stdlib might grow
>>> could be a public API that makes implementing this feasible without
>>> overriding private methods.)
>>>
>>> On Sat, Feb 15, 2020 at 3:16 PM Jonathan Crall 
>>> wrote:
>>>
>>>> This implementation is a proof-of-concept that I've been using for
>>>> awhile
>>>> <https://gitlab.kitware.com/computer-vision/ndsampler/blob/master/ndsampler/util_futures.py>.
>>>> Its certain that any version that made it into the stdlib would have to be
>>>> more carefully designed than the implementation I threw together. However,
>>>> my implementation demonstrates the concept and there are reasons for the
>>>> choices I made.
>>>>
>>>> First, the choice to create a SerialFuture object that inherits from
>>>> the base Future was because I only wanted a process to run if the
>>>> SerialFuture.result method was called. The most obvi

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

[Kyle Stanley]

> No; the point of launch policies is that you can (without needing an
executor object[1]) tell the task to run “async” (on its own thread[2]),
“deferred” (serially[3] on first demand), or “immediate” (serially right
now)[4]. You can even or together multiple policies to let the
implementation choose, and IIRC the default is async|deferred.

> [2] Actually “as if on its own thread”. But AFAIK, every implementation
handles this by spawning a thread. I think the distinction is for future
expansions, either so they can do something like Java’s ForkJoinPool, or so
they can use fibers or coroutines that don’t care what thread they’re on.

Ah, so the fact that std::async spawns a separate thread in many
implementations is more of an internal detail that could be changed, rather
than a guaranteed behavior. Thanks for the clarification and detailed
explanation, libstdc++ is definitely not an area of expertise for me. (:

> The concrete use case I’ve run into is this: I’ve got some parallel code
that has a bug. I’m pretty sure the bug isn’t actually related to the
shared data or the parallelism itself, but I want to be sure. I replace the
ThreadPoolExecutor with a SyncExecutor and change nothing else about the
code, and the bug still happens. Now I’ve proven that the bug isn’t related
to parallelism. And, as a bonus, I’ve got nice logs that aren’t interleaved
into a big mess, so it’s easier to track down the problem.

That sounds like it would be quite a useful utility class for general
executor debugging purposes. But, I'm just not convinced that it would see
wide enough usage to justify adding it to concurrent.futures. IMO, this
makes it a perfect candidate for a decent PyPI package. If that package
ends up being significantly popular, it might be worth re-examining it's
membership in the stdlib once it becomes mature. This would reduce the risk
of burdening CPython development time with an underused feature, and gives
it far more room for growth/improvement [1].

Also, as a sidenote, I much more prefer the term "SyncExecutor" rather than
"SerialExecutor". I think the former is a bit more clear at defining it's
actual purpose.


[1] - Once something gets added to the standard library, it has to adhere
as much as reasonably possible to backwards compatibility, making any
changes in behavior and API drastically more difficult. Also, its
development time becomes limited by CPython's release cycle rather than
having its own.

On Sun, Feb 16, 2020 at 12:43 AM Andrew Barnert  wrote:

> On Feb 15, 2020, at 20:29, Kyle Stanley  wrote:
>
>
> *Add a SerialExecutor, which does not use threads or processes*
>
> Andrew Barnert wrote:
> > e.g., in C++, you only use executors via the std::async function, and
> you can just pass a launch option instead of an executor to run
> synchronously
>
> In the case of C++'s std::async though, it still launches a thread to run
> the function within, no?
>
>
> No; the point of launch policies is that you can (without needing an
> executor object[1]) tell the task to run “async” (on its own thread[2]),
> “deferred” (serially[3] on first demand), or “immediate” (serially right
> now)[4]. You can even or together multiple policies to let the
> implementation choose, and IIRC the default is async|deferred.
>
> At any rate, I’m not suggesting that C++ is a design worth looking at,
> just parenthetically noting it as an example of how when libraries don’t
> have a serial executor, it’s often because they already have a different
> way to specify the same thing.
>
> This doesn't require the user to explicitly create or interact with the
> thread in any way, but that seems to go against what OP was looking for:
>
> Jonathan Crall wrote:
> > Often times a develop will want to run a task in parallel, but depending
> on the environment they may want to disable threading or process execution.
>
> The *concrete* purpose of what that accomplishes (in the context of
> CPython) isn't clear to me. How exactly are you running the task in
> parallel without using a thread, process, or coroutine [1]?
>
>
> I’m pretty sure what he meant is that the developer _usually_ wants the
> task to run in parallel, but in some specific situation he wants it to
> _not_ run in parallel.
>
> The concrete use case I’ve run into is this: I’ve got some parallel code
> that has a bug. I’m pretty sure the bug isn’t actually related to the
> shared data or the parallelism itself, but I want to be sure. I replace the
> ThreadPoolExecutor with a SyncExecutor and change nothing else about the
> code, and the bug still happens. Now I’ve proven that the bug isn’t related
> to parallelism. And, as a bonus, I’ve got nice logs that aren’t interleaved
> into a big mess, so it’s easier to track down the problem.
>
> I have no idea if this is Jonathan’s use, 

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

[Kyle Stanley]

> I've never felt the need for either of these myself, nor have I observed
it in others I worked with. In general I feel the difference between
processes and threads is so large that I can't believe a realistic
application would work with either.

Also, ThreadPoolExecutor and ProcessPoolExecutor both have their specific
purposes in concurrent.futures: TPE for IO-bound parallelism, and PPE for
CPU-bound parallelism, what niche would the proposed SerialExecutor fall
under? Fake/dummy parallelism? If so, I personally don't see that as being
worth the cost of adding it and then maintaining it in the standard
library. But, that's not to say that it wouldn't have a place on PyPI.

> (Then again I've never had much use for ProcessExecutor period.)

I've also made use of TPE far more times than PPE, but I've definitely seen
several interesting and useful real-world applications of PPE. Particularly
with image processing. I can also imagine it also being quite useful for
scientific computing, although I've not personally used it for that purpose.

> IOW I'm rather lukewarm about this -- even if you (Jonathan) have found
use for it, I'm not sure how many other people would use it, so I doubt
it's worth adding it to the stdlib. (The only thing the stdlib might grow
could be a public API that makes implementing this feasible without
overriding private methods.)

Expanding a bit upon the public API for the cf.Future class would likely
allow something like this to be possible without accessing any private
members. In particular, I believe there would have to be an public means of
accessing the state of the future without having to go through the
condition (currently, this can only be done with ``future._state``), and
accessing a constant for each of the possible states: PENDING. RUNNING,
CANCELLED, CANCELLED_AND_NOTIFIED, and FINISHED.

Since that would actually be quite useful for debugging purposes (I had to
access ``future._state`` several times while testing the new
*cancel_futures*), I'd be willing to work on implementing something like
this.


On Sat, Feb 15, 2020 at 10:16 PM Guido van Rossum  wrote:

> Having tried my hand at a simpler version for about 15 minutes, I see the
> reason for the fiddly subclass of Future -- it seems over-engineered
> because concurrent.future is complicated.
>
> I've never felt the need for either of these myself, nor have I observed
> it in others I worked with. In general I feel the difference between
> processes and threads is so large that I can't believe a realistic
> application would work with either. (Then again I've never had much use for
> ProcessExecutor period.)
>
> The "Serial" variants somehow remind me of the "dummy_thread.py" module we
> had in Python 2. It was removed in Python 3, mostly because we ran out of
> cases where real threads weren't an option.
>
> IOW I'm rather lukewarm about this -- even if you (Jonathan) have found
> use for it, I'm not sure how many other people would use it, so I doubt
> it's worth adding it to the stdlib. (The only thing the stdlib might grow
> could be a public API that makes implementing this feasible without
> overriding private methods.)
>
> On Sat, Feb 15, 2020 at 3:16 PM Jonathan Crall  wrote:
>
>> This implementation is a proof-of-concept that I've been using for awhile
>> .
>> Its certain that any version that made it into the stdlib would have to be
>> more carefully designed than the implementation I threw together. However,
>> my implementation demonstrates the concept and there are reasons for the
>> choices I made.
>>
>> First, the choice to create a SerialFuture object that inherits from the
>> base Future was because I only wanted a process to run if the
>> SerialFuture.result method was called. The most obvious way to do that was
>> to overload the `result` method to execute the function when called.
>> Perhaps there is a better way, but in an effort to KISS I just went with
>> the <100 line version that seemed to work well enough.
>>
>> The `set_result` is overloaded because in Python 3.8, the base
>> Future.set_result function asserts that the _state is not FINISHED when it
>> is called. In my proof-of-concept implementation I had to set state of the
>> SerialFuture._state to FINISHED in order for `as_completed` to yield it.
>> Again, there may be a better way to do this, but I don't claim to know what
>> that is yet.
>>
>> I was thinking that a factory function might be a good idea, but if I was
>> designing the system I would have put that in the abstract Executor class.
>> Maybe something like
>>
>>
>> ```
>> @classmethod
>> def create(cls, mode, max_workers=0):
>> """ Create an instance of a serial, thread, or process-based executor
>> """
>> from concurrent import futures
>> if mode == 'serial' or max_workers == 0:
>> return futures.SerialExecutor()
>> elif mode == 'thread':
>> return 

[Python-ideas] Re: SerialExecutor for concurrent.futures + Convenience constructor

[Kyle Stanley]

This seems to be two separate proposals:

1) Add a new way to create and specify executor
2) Add a SerialExecutor, which does not use threads or processes

So, I'll respond to each one separately.


*Add a new way to create and specify executor*

Jonathan Crall wrote:
> The library's ThreadPoolExecutor and ProcessPoolExecutor are excellent
tools, but there is currently no mechanism for configuring which type of
executor you want.

The mechanism of configuring the executor type is by instantiating the type
of executor you want to use. For IO-bound parallelism you use
``cf.ThreadPoolExecutor()`` or for CPU-bound parallelism you use
``cf.ProcessPoolExecutor()``. So I'm not sure that it would be practically
beneficial to provide multiple ways to configure the type of executor to
use. That seems to go against the philosophy of preferring "one obvious way
to do it" [1].

I think there's a very reasonable argument for using a
``cf.Executor.create()`` or ``cf.create_executor()`` that works as a
factory to initialize and return an executor class based on parameters that
are passed to it, but to me, that seems better suited for a different
library/alternative interface. I guess that I just don't see a practical
benefit in having both means of specifying the type of executor for
concurrent.futures in the standard library, both from a development
maintenance perspective and feature bloat. If a user wants to be able to
specify the executor used in this manner, it's rather trivial to implement
it in a few lines of code without having to access any private members;
which to me seems to indicate that there's not a whole lot of value in
adding it to the standard library.

That being said, if there are others that would like to use an alternative
interface for concurrent.futures, it could very well be uploaded as a small
package on PyPI. I just personally don't think it has a place in the
existing concurrent.futures module.


[1] - One could say that context managers provide an alternative means of
creating and using the executors, but context managers provide a
significant added value in the form of resource cleanup. To me, there
doesn't seem to be much real added value in being able to both use the
existing``executor = cf.ThreadPoolExecutor()`` and a new ``executor =
cf.create_executor(mode="thread")`` / ``executor =
cf.Executor.create(mode="thread")``.


*Add a SerialExecutor, which does not use threads or processes*

Andrew Barnert wrote:
> e.g., in C++, you only use executors via the std::async function, and you
can just pass a launch option instead of an executor to run synchronously

In the case of C++'s std::async though, it still launches a thread to run
the function within, no? This doesn't require the user to explicitly create
or interact with the thread in any way, but that seems to go against what
OP was looking for:

Jonathan Crall wrote:
> Often times a develop will want to run a task in parallel, but depending
on the environment they may want to disable threading or process execution.

The *concrete* purpose of what that accomplishes (in the context of
CPython) isn't clear to me. How exactly are you running the task in
parallel without using a thread, process, or coroutine [1]? Without using
one of those constructs (directly or indirectly), you're really just
executing the tasks one-by-one, not with any form of parallelism, no? That
seems to go against the primary practical purpose of using
concurrent.futures in the first place. Am I misunderstanding something
here? Perhaps it would help to have some form of real-world example where
this might be useful, and how it would benefit from using something like
SerialExecutor over other alternatives.

Jonathan Crall wrote:
> The `set_result` is overloaded because in Python 3.8, the base
Future.set_result function asserts that the _state is not FINISHED when it
is called. In my proof-of-concept implementation I had to set state of the
SerialFuture._state to FINISHED in order for `as_completed` to yield it.
Again, there may be a better way to do this, but I don't claim to know what
that is yet.

The main purpose of `cf.as_completed()` is to yield the results
asynchronously as they're completed (FINISHED or CANCELLED), which is
inherently *not* going to be serial. If you want to instead yield each
result in the same order they're submitted, but as each one is completed
[2], you could do something like this:

```
executor = cf.ThreadPoolExecutor()
futs = []
for item in to_do:
 fut = executor.submit(do_something, item)
 futs.append(fut)
for fut in futs:
yield fut.result()
```

(The above would be presumably part of some generator function/method where
you could pass a function *do_something* and an iterable of IO-bound tasks
*to_do*)

This would allow you to execute tasks the parallel, while ensuring the
results yielded are serial/synchronous.


[1] - You could also create subinterpreters to run tasks in parallel
through the C-API, or through the upcoming 

[Python-ideas] Re: Asyncio Future.set_result_threadsafe

[Kyle Stanley]

Oops, I forgot to start the thread in the above example for
compute_something(), ``th.start()`` should go right after initializing the
thread but before the try-finally block

On Thu, Feb 13, 2020 at 6:15 PM Kyle Stanley  wrote:

> > I agree that is counter to the basic philosophy, though there are
> potential use cases where it may be unavoidable to use threads.  It seems
> the best solutions when it is then are to use run_in_executor, which
> automatically handles the waking up of the loop.
>
> While this doesn't provide an immediate solution, we're currently working
> on implementing an asyncio.ThreadPool for Python 3.9, which is made to be
> more self-contained (initialized and finalized within an async context
> manager using ``async with``) and with a higher level API compared to
> loop.run_in_executor(). In general, I'd recommend making use of either of
> those two compared to directly using threads in asyncio.
>
> That being said, if you need more fine-grained control, there's also one
> other alternative low-level construct/design pattern you can use for
> working with threads in asyncio that we use in the implementation of some
> coroutine methods, such as one that I recently added to 3.9 called
> loop.shutdown_default_executor(). Using a similar format to your above
> example:
>
> ```
> def _do_complete_something(fut, loop, ...):
> try:
> # perform some action, optionally get a result to set to the future
> loop.call_soon_threadsafe(future.set_result, None) # replace None
> with actual result if needed
> except Exception as ex:
> loop.call_soon_threadsafe(future.exception, ex)
>
> async def compute_something(...):
> # In general, I'd recommend using get_running_loop() to get the event
> loop within a coroutine function.
> # It has more predictable behavior compared to get_event_loop().
> (Python 3.7+)
> loop = asyncio.get_running_loop()
> fut = loop.create_future()
> th = threading.Thread(target=_do_compute_something, args=(fut,loop,
> ...))
> try:
> # use "return await future" if the value is desired
> await future
> finally:
> th.join()
>
> async def compute_stuff():
> result = await compute_something(...)
>
> asyncio.run(compute_stuff())
> ```
>
> Hopefully that helps a bit, or at least gives you some some ideas.
>
> On Thu, Feb 13, 2020 at 9:31 AM Brianvanderburg2 via Python-ideas <
> python-ideas@python.org> wrote:
>
>> I agree that is counter to the basic philosophy, though there are
>> potential use cases where it may be unavoidable to use threads.  It seems
>> the best solutions when it is then are to use run_in_executor, which
>> automatically handles the waking up of the loop.  While the Queue object
>> doesn't work with threads, though there does seem to be a library janus
>> which provides a thread-capbable queue object which could be another
>> solution.
>>
>>
>> -Original Message-
>> From: Guido van Rossum 
>> To: Brian Allen Vanderburg II 
>> Cc: Python-Ideas 
>> Sent: Thu, Feb 13, 2020 1:18 am
>> Subject: Re: [Python-ideas] Asyncio Future.set_result_threadsafe
>>
>> Sharing futures between threads like that goes counter to asyncio's basic
>> philosophy (which is not to use threads :-).
>>
>> You already showed the solution:
>> future._loop.call_soon_threadsafe(future.set_result, ...). If that's
>> unacceptable for you, maybe you can wrap the future in a wrapper class that
>> calls call_soon_threadsafe.
>>
>> On Wed, Feb 12, 2020 at 8:51 PM Brian Allen Vanderburg II via
>> Python-ideas  wrote:
>>
>> Currently asyncio.futures.Future.set_result will result in any callbacks
>> being scheduled using loop.call_soon instead of
>> loop.call_soon_threadsafe. However in situations where the future's
>> result is set from a different thread, the loop might not wake up as a
>> result of this if it is currently sleeping.
>>
>>
>> def compute_something(...):
>> loop = asyncio.get_event_loop()
>> future = loop.create_future()
>> manager.run_computation(future, ...) # add the computation to a
>> thread that is running it
>> return future
>>
>> async def compute_stuff():
>> result = await compute_something(...)
>>
>> loop = asyncio.get_event_loop()
>> loop.run_until_complete(compute stuff())
>> loop.close()
>>
>>
>> The reasoning behind it is after yielding the Future object, the ready
>> list of the event loop is empty so when waiting for selectors the
>> timeout value is None and it just wait

[Python-ideas] Re: Asyncio Future.set_result_threadsafe

[Kyle Stanley]

> I agree that is counter to the basic philosophy, though there are
potential use cases where it may be unavoidable to use threads.  It seems
the best solutions when it is then are to use run_in_executor, which
automatically handles the waking up of the loop.

While this doesn't provide an immediate solution, we're currently working
on implementing an asyncio.ThreadPool for Python 3.9, which is made to be
more self-contained (initialized and finalized within an async context
manager using ``async with``) and with a higher level API compared to
loop.run_in_executor(). In general, I'd recommend making use of either of
those two compared to directly using threads in asyncio.

That being said, if you need more fine-grained control, there's also one
other alternative low-level construct/design pattern you can use for
working with threads in asyncio that we use in the implementation of some
coroutine methods, such as one that I recently added to 3.9 called
loop.shutdown_default_executor(). Using a similar format to your above
example:

```
def _do_complete_something(fut, loop, ...):
try:
# perform some action, optionally get a result to set to the future
loop.call_soon_threadsafe(future.set_result, None) # replace None
with actual result if needed
except Exception as ex:
loop.call_soon_threadsafe(future.exception, ex)

async def compute_something(...):
# In general, I'd recommend using get_running_loop() to get the event
loop within a coroutine function.
# It has more predictable behavior compared to get_event_loop().
(Python 3.7+)
loop = asyncio.get_running_loop()
fut = loop.create_future()
th = threading.Thread(target=_do_compute_something, args=(fut,loop,
...))
try:
# use "return await future" if the value is desired
await future
finally:
th.join()

async def compute_stuff():
result = await compute_something(...)

asyncio.run(compute_stuff())
```

Hopefully that helps a bit, or at least gives you some some ideas.

On Thu, Feb 13, 2020 at 9:31 AM Brianvanderburg2 via Python-ideas <
python-ideas@python.org> wrote:

> I agree that is counter to the basic philosophy, though there are
> potential use cases where it may be unavoidable to use threads.  It seems
> the best solutions when it is then are to use run_in_executor, which
> automatically handles the waking up of the loop.  While the Queue object
> doesn't work with threads, though there does seem to be a library janus
> which provides a thread-capbable queue object which could be another
> solution.
>
>
> -Original Message-
> From: Guido van Rossum 
> To: Brian Allen Vanderburg II 
> Cc: Python-Ideas 
> Sent: Thu, Feb 13, 2020 1:18 am
> Subject: Re: [Python-ideas] Asyncio Future.set_result_threadsafe
>
> Sharing futures between threads like that goes counter to asyncio's basic
> philosophy (which is not to use threads :-).
>
> You already showed the solution:
> future._loop.call_soon_threadsafe(future.set_result, ...). If that's
> unacceptable for you, maybe you can wrap the future in a wrapper class that
> calls call_soon_threadsafe.
>
> On Wed, Feb 12, 2020 at 8:51 PM Brian Allen Vanderburg II via Python-ideas
>  wrote:
>
> Currently asyncio.futures.Future.set_result will result in any callbacks
> being scheduled using loop.call_soon instead of
> loop.call_soon_threadsafe. However in situations where the future's
> result is set from a different thread, the loop might not wake up as a
> result of this if it is currently sleeping.
>
>
> def compute_something(...):
> loop = asyncio.get_event_loop()
> future = loop.create_future()
> manager.run_computation(future, ...) # add the computation to a
> thread that is running it
> return future
>
> async def compute_stuff():
> result = await compute_something(...)
>
> loop = asyncio.get_event_loop()
> loop.run_until_complete(compute stuff())
> loop.close()
>
>
> The reasoning behind it is after yielding the Future object, the ready
> list of the event loop is empty so when waiting for selectors the
> timeout value is None and it just waits. When the other thread calls
> set_result, the loop ready list will get updated, but it's still waiting
> on selectors.  The other thread could call
> future._loop.call_soon_threadsafe(future.set_result, ...), which writes
> some bytes to a socket in order to wake the even loop.
>
> I'm aware there are other ways to do things like this (queues,
> executors, etc). It just seems that, regardless of any method used, the
> code holding the future with the purpose of setting a result on it
> should be able to be blind to other implementation details.
> future.set_result should be all it needs to know to call even if from
> another thread, or if not, a method like future.set_result_threadsafe,
> which would hide the details of the loop.call_soon/call_soon_threadsafe
>
>
> ___
> Python-ideas mailing list -- 

[Python-ideas] Re: Multi-threading interface idea

[Kyle Stanley]

> 2. I couldn't wrap my head around the async/await and future constructs
particularly quickly, and I was concerned that my team would also struggle
with this change.

> 3. I believe the procedural style glue code we have is quite easy to
comprehend, which I think has a positive impact on scale.

While I can certainly understand the appeal of the simplicity of the ``@
parallel.task`` decorator used in the example, I strongly suspect that it
will end up becoming increasingly tangled as the needs grow in complexity.
I'd bet on something like this being highly convenient in the short term,
but very costly in the long term if it eventually becomes unmaintainable
and has to be reconstructed from the ground up (which seems rather likely).
It would also lose out on much of the useful functionality of futures, such
as cancellations, timeouts, and asynchronously iterating over results in
the order of completion, as they finish, rather than order sent (with
``cf.as_completed()``); just to name a few.

I can also understand that it takes some time to get used to how futures
work, but it's well worth the effort and time to develop a solid
fundamental understanding for building scalable back-end systems. Many
asynchronous and concurrent frameworks (including in other languages, such
as C++, Java, and C#) utilize futures in a similar manner, so the general
concepts apply universally for the most part. It's a similar story with
async/await syntax (which is present in C# and JS; and upcoming in C++20).

That being said, I think the above syntax could be useful for simple
scripts, prototyping, or perhaps for educational purposes. I could see it
potentially having some popularity on PyPI for those use cases, but I don't
think it has a place in the standard library.

On Sat, Feb 8, 2020 at 6:08 PM Sean McIntyre  wrote:

> Hi folks,
>
> I'd like to get some feedback on a multi-threading interface I've been
> thinking about and using for the past year or so. I won't bury the lede, see
> my approach here
> 
> .
>
> *Background / problem:*
>
> A couple of years ago, I inherited my company's codebase to get data into
> our data warehouse using an ELT approach (extract-and-loads done in python,
> transforms done in dbt/SQL). The codebase has dozens of python scripts to
> integrate first-party and third-party data from databases, FTPs, and APIs,
> which are run on a scheduler (typically daily or hourly). The scripts I
> inherited were single-threaded procedural scripts, looking like glue code,
> and spending most of their time in network I/O. (See example.
> )
> This got my company pretty far!
>
> As my team and I added more and more integrations with more and more data,
> we wanted to have faster and faster scripts to reduce our dev cycles and
> reduce our multi-hour nightly jobs to minutes. Because our scripts were
> network-bound, multi-threading was a good way to accomplish this, and so I
> looked into concurrent.futures (example
> )
> and asyncio (example
> ),
> but I decided against these options because:
>
> 1. It wasn't immediately apparently how to adapt my codebase to use these
> libraries without either some fundamental changes to our execution platform
> and/or reworking of our scripts from the ground up and/or adding
> significant lines of multi-threading code to each script.
>
> 2. I couldn't wrap my head around the async/await and future constructs
> particularly quickly, and I was concerned that my team would also struggle
> with this change.
>
> 3. I believe the procedural style glue code we have is quite easy to
> comprehend, which I think has a positive impact on scale.
>
> *Solution:*
>
> And so, as mentioned at the top, I designed a different interface to
> concurrent.futures.ThreadPoolExecutor that we are successfully using for
> our extract-and-load pattern, see a basic example here
> .
> The design considerations of this interface include:
>
> - The usage is minimally-invasive to the original unthreaded approach of
> the codebase. (And so, teaching the library to team members has been fairly
> straightforward despite the multi-threaded paradigm shift.)
>
> - The @parallel.task decorator should be used to encapsulate a homogeneous
> method accepting different parameters. The contents of the method should be
> primarily I/O to achieve the concurrency gains of python multi-threading.
>
> - If no parallel.threads context manager has been entered, the
> @parallel.task decorator acts as a no-op (and the code runs serially).
>
> - If an environment variable is set 

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Kyle Stanley]

> Then if Executor.__exit__ detects an exception it would call shutdown
with cancel_futures set to True.

Oh, I see. That should be rather simple:

```
def __exit__(self, exc_type, exc_val, exc_tb):
if exc_val is not None and self._cancel_on_error:
self.shutdown(wait=True, cancel_futures=True)
else:
self.shutdown(wait=True)
return False
```

(I believe the above would have to be specified as an override for each of
ThreadPoolExecutor and ProcessPoolExecutor if *cancel_on_error* were
supplied as a parameter to the executor constructor. They both currently
use the ``__exit__`` specified in the abstract Executor class, which
wouldn't have access to *cancel_on_error*.)

My previous consideration about waiting for user input of
*cancel_futures *after
the release of 3.9*, *before adding *cancel_on_error* to the constructor
still applies to some degree though. If it directly uses
executor.shutdown() with cancel_futures set to True, the maintenance would
be rather minimal. But, it would potentially risk adding an underutilized
parameter to the executor constructor (which contributes to feature bloat).

On Mon, Feb 3, 2020 at 4:03 AM Miguel Ángel Prosper <
miguelangel.pros...@gmail.com> wrote:

> > Hmm, it should be possible. Do you specifically mean cancelling the
> pending
> > futures once a single one of the submitted functions raises an exception,
> > or cancelling the pending futures when the Executor itself raises an
> > exception (I.E. BrokenProcessPool)? I would assume the prior, since that
> > seems more useful to me.
>
> I was referring to any exception within the context manager of the
> executor, not the futures or just the executor's exceptions, something such
> as the following:
>
> with ThreadPoolExecutor(cancel_on_error=True) as executor:
> # Code here
> raise Exception
>
>  Then if Executor.__exit__ detects an exception it would call shutdown
> with cancel_futures set to True.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/OAMG4JUFK54FTAASHQG5WNAA7BU7KRS7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GCKG347LGTROINXHYUVTN3IN4QTLGYRK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Kyle Stanley]

Miguel Ángel Prosper wrote:
> Thank you so much for the work, I was very confused on how to even start
implementing it in the ProcessPoolExecutor, but you finished everything
super quick!

No problem! The ProcessPoolExecutor implementation wasn't immediately clear
to me either, but after some experimentation (and some great suggestions
from Brian Quinlain to improve it) I was able to get it functioning as
intended. Thank you for the proposal, I think it will be a good addition
for the Executor API.

Miguel Ángel Prosper wrote:
> I'm suppose that this might be better in another thread but... Could it
be possible to changing the context manager of the executor to cancel
futures on uncaught exceptions?

Hmm, it should be possible. Do you specifically mean cancelling the pending
futures once a single one of the submitted functions raises an exception,
or cancelling the pending futures when the Executor itself raises an
exception (I.E. BrokenProcessPool)? I would assume the prior, since that
seems more useful to me.

Miguel Ángel Prosper wrote:
> If not I could also be added as an optional parameter to the executor
constructor, "cancel_on_error" set to False to not changing anything.

Assuming we did implement something like this, I'd say it should be done as
an optional parameter for the constructor (as stated above) rather than as
the default option when an exception occurs, because it would significantly
different from the current behavior. At the moment, the executor doesn't
mind when exceptions are raised from submitted functions; it simply sets
the exception to the future (via ``future.set_exception()``) and moves on
to the next.

So, cancelling the pending futures after one raises an exception would be a
significant behavioral change; one that could easily result in breaking
existing code. As a result, I think it should be an optional parameter if
anything.

As far as whether or not it should be implemented, I'm not 100% certain.
While I'm not against the idea of it, I think it would be reasonable to
wait until the new parameter is released in Python 3.9 and see what users
think of it prior to adding a similar parameter to the executor constructor
as well. The cost of adding a new feature demanded by users is relatively
minimal, but once it's added we're stuck with maintaining it.

If the idea is approved, I'd be happy to help with implementing it. I just
want to make sure that we actually want it in the first place.

What do you think, Guido? I've also CC'd Brian Quinlain and Antoine Pitrou,
in case they have any feedback but didn't see the original thread.

On Mon, Feb 3, 2020 at 2:27 AM Miguel Ángel Prosper <
miguelangel.pros...@gmail.com> wrote:

> Thank you so much for the work, I was very confused on how to even start
> implementing it in the ProcessPoolExecutor, but you finished everything
> super quick!
>
> I'm suppose that this might be better in another thread but... Could it be
> possible to changing the context manager of the executor to cancel futures
> on uncaught exceptions? If not I could also be added as an optional
> parameter to the executor constructor, "cancel_on_error" set to False to
> not changing anything. Personally I was thinking of using the executor like
> that; currently I have to either place everything in a try except block,
> handle it and then reraise it or subclass each executor.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/R5VMWP75K35KF3DJN2F5MBZXD5UOQIV3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/PMIV2XIYGHULEBZVDG2Q5PBQJJXYB2LD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Kyle Stanley]

> I would certainly be willing to look into it.

As an update to this thread for anyone interested in this feature, it's
been implemented in Python 3.9 for both ProcessPoolExecutor and
ThreadPoolExecutor as a new parameter to Executor.shutdown(),
*cancel_futures*.

For a description of the feature, see the updated documentation:
https://docs.python.org/3.9/library/concurrent.futures.html#concurrent.futures.Executor.shutdown

For implementation details, see the PR:
https://github.com/python/cpython/pull/18057

Also, thank you Guido for bringing attention to the issue. The
implementation was a bit more involved than I initially anticipated
(particularly for ProcessPoolExecutor), but I found it to be well worth the
effort! Both for the benefit of the new feature, and for the opportunity to
work with a part of the internals of the executors.

On Wed, Jan 15, 2020 at 5:59 PM Kyle Stanley  wrote:

> > Is anyone else interested in implementing this small feature for
> concurrent.futures?
>
> I would certainly be willing to look into it. We've been discussing the
> possibility of a native threadpool for asyncio in the future (
> https://bugs.python.org/issue32309), so it would certainly be beneficial
> for me to build some experience in working with the internals of the
> executors. I think implementing this small feature would be a good
> introduction.
>
> On Wed, Jan 15, 2020 at 5:37 PM Guido van Rossum  wrote:
>
>> (Belatedly)
>>
>> Is anyone else interested in implementing this small feature for
>> concurrent.futures?
>>
>> On Fri, Jan 3, 2020 at 18:28 Miguel Ángel Prosper <
>> miguelangel.pros...@gmail.com> wrote:
>>
>>> > It looks like you have a good handle on the code -- do you want to
>>> submit a PR to GitHub to add such a parameter?
>>>
>>> Thanks, but I'm not really sure how to implement it in the
>>> ProcessPoolExecutor, I just think the solution is probably related to the
>>> code responsible of handling a failed initializer (since they do very
>>> similar things). On the ThreadPoolExecutor maybe I could, but I haven't
>>> really checked for side effects or weird things.
>>> ___
>>> Python-ideas mailing list -- python-ideas@python.org
>>> To unsubscribe send an email to python-ideas-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-ideas@python.org/message/FGTSBGTXW63SSXFQBASLCTXAT4M6RGGN/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> --
>> --Guido (mobile)
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/WSE2KMKFMPG2RYSJSGMRZPDJLM7DZKSG/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IEEZK3T4QAU2RSSYXHZQVYYGVIF6JWEJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Compound statement colon (Re: Re: Improve SyntaxError for obvious issue:)

[Kyle Stanley]

> What if the colon were made optional, with an eye to perhaps eventually
no longer using it as the preferred style for new code?

> But if a line beginning as a compound statement and ending without a
colon is *never* going to have a valid meaning as something else... what's
the point of the colon, otherwise? Seems like just grit on the screen.

That's a very interesting idea, and could perhaps be a viable design
decision for making a new language. But, I strongly suspect that the
current usage of the colon is far too fundamentally nested in the
foundations of Python, and is likely not going to change.

Even if it was simply made optional, it would no doubt introduce quite a
lot of confusion and add an extra cost to Python's learning curve, for both
newer and older developers. Introductory Python guides all over the place
would also have to updated, to mention that in version x, developers may
see this code example with or without the colon. In order to justify this,
it has to be worth the cost.

You main argument for it seems to be that it's unnecessary, and ends up
being an extra character without a real purpose. I don't find this to be
worth the cost associated with making it optional.

Also, that assumes that it *is* entirely without purpose. Personally, I
find that it helps as a visual queue, to indicate that the next line is
within the block of the statement that proceeds the colon. This isn't a
huge deal in one line statements, but when it carries over to the next line
it greatly helps to distinguish one level from the next.

For example, PEP8 mentions the following code formats as "acceptable
options" of handling multi-line statements:


(1)
# No extra indentation.
if (this_is_one_thing and
that_is_another_thing):
do_something()

(2)
# Add a comment, which will provide some distinction in editors
# supporting syntax highlighting.
if (this_is_one_thing and
that_is_another_thing):
# Since both conditions are true, we can frobnicate.
do_something()

(3)
# Add some extra indentation on the conditional continuation line.
if (this_is_one_thing
and that_is_another_thing):
do_something()


Now, let's take the colon away and see what it looks like:


(1)
# No extra indentation.
if (this_is_one_thing and
that_is_another_thing)
do_something()

(2)
# Add a comment, which will provide some distinction in editors
# supporting syntax highlighting.
if (this_is_one_thing and
that_is_another_thing)
# Since both conditions are true, we can frobnicate.
do_something()

(3)
# Add some extra indentation on the conditional continuation line.
if (this_is_one_thing
and that_is_another_thing)
do_something()


This might be a bit subjective, but to my eyes this makes the block that's
within the statement rather difficult to distinguish; particularly for (1)
and (2). The effect would only be amplified in more complex code.

On Thu, Jan 16, 2020 at 1:15 PM Random832  wrote:

> On Tue, Jan 14, 2020, at 18:15, David Mertz wrote:
> > For what it's worth, after 20+ years of using Python, forgetting the
> > colon for blocks remains the most common error I make by a fairly wide
> > margin. Of course, once I see the error message—even being not all that
> > descriptive of the real issue—I immediately know what to fix too.
>
> What if the colon were made optional, with an eye to perhaps eventually no
> longer using it as the preferred style for new code?
>
> We had a post a while ago about the possibility of using the lack of a
> colon as an implicit line continuation (like with parentheses, e.g. "if
> a\nand b:", and this was (reasonably) rejected. But if a line beginning as
> a compound statement and ending without a colon is *never* going to have a
> valid meaning as something else... what's the point of the colon,
> otherwise? Seems like just grit on the screen.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/J47TEY2KFGATFMQ7RSZJO7B4RV7KEYWJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FMXPQWF2Z43Z725Y3EFNHGZZC5IGSV2C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Change List Metadata

[Kyle Stanley]

> I meant by composing them in another class, which could then have
> whatever interface makes sense for this data structure.

Ah, I see. I had misunderstood you and thought you were advising the OP to
combine list and Counter into their own new data structure, which seemed a
bit overkill.

That makes a lot more sense, thanks for the clarification.

On Wed, Jan 15, 2020 at 8:20 PM Eric V. Smith  wrote:

> On 1/15/2020 7:43 PM, Kyle Stanley wrote:
> > > I suggest that when you need this functionality you create your own
> > data
> > > structure combining a list and a collections.Counter and keep track of
> > > this yourself.
> >
> > I concur with the usage of collections.Counter here. Storing the count
> > for every single item in a list could end up being rather redundant
> > with duplicate elements, so Counter ends up being much more space
> > efficient. With it being implemented as a dictionary, the lookup
> > speeds are incredibly quick for getting an on-demand count for any
> > item within it.
> >
> > But, is it practically beneficial to combine the functionality of a
> > list and collections.Counter instead of just using both data
> > structures together?
> >
> I meant by composing them in another class, which could then have
> whatever interface makes sense for this data structure.
>
> Eric
>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3THZZAFKUFB4FQGKDHYYXAFEKE5GLABW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Change List Metadata

[Kyle Stanley]

> I suggest that when you need this functionality you create your own data
> structure combining a list and a collections.Counter and keep track of
> this yourself.

I concur with the usage of collections.Counter here. Storing the count for
every single item in a list could end up being rather redundant with
duplicate elements, so Counter ends up being much more space efficient.
With it being implemented as a dictionary, the lookup speeds are incredibly
quick for getting an on-demand count for any item within it.

But, is it practically beneficial to combine the functionality of a list
and collections.Counter instead of just using both data structures together?

Here's a rough example of what I'm talking about. I recall using something
vaguely similar to this before:

```
items = [...]
counter = collections.Counter(items)
for item in items:
# use counter[item] instead of items.count(item)
if count := counter[item] > x:
# do something with item and its count
```

(Note: omit "count :=" if the count isn't needed for anything beyond the
conditional check)

On Wed, Jan 15, 2020 at 10:34 AM Eric V. Smith  wrote:

> On 1/14/2020 11:03 PM, Hunter Jones wrote:
> > Hey everyone,
> >
> > I recently used list.count() in a coding interview and the question
> > arose about how scale-able this solution was for sufficiently large
> > input. Currently, list.count iterates through the list, incrementing
> > the count as it goes and returning the value at the end. This does not
> > lend itself well to large inputs.
> >
> > I propose either modifying the list struct to include a map of item ->
> > count, or implementing a new structure with this information that is
> > allocated separately when a list is instantiated.
> >
> > Maybe I'm just seeing this since it's an edge case that I recently
> > dealt with, but I believe it would be a useful enhancement overall.
> >
> > Thoughts?
>
> That seems like a pretty niche requirement. In all of my years using
> Python I've never needed such a thing.
>
> I suggest that when you need this functionality you create your own data
> structure combining a list and a collections.Counter and keep track of
> this yourself.
>
> Eric
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/QX7V3BFBVJBARTP6E3NSSAG3XQDDZ7FX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XXHVXJWPRLAPOUCMXC2U33NP6NUJ5RK2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown

[Kyle Stanley]

> Is anyone else interested in implementing this small feature for
concurrent.futures?

I would certainly be willing to look into it. We've been discussing the
possibility of a native threadpool for asyncio in the future (
https://bugs.python.org/issue32309), so it would certainly be beneficial
for me to build some experience in working with the internals of the
executors. I think implementing this small feature would be a good
introduction.

On Wed, Jan 15, 2020 at 5:37 PM Guido van Rossum  wrote:

> (Belatedly)
>
> Is anyone else interested in implementing this small feature for
> concurrent.futures?
>
> On Fri, Jan 3, 2020 at 18:28 Miguel Ángel Prosper <
> miguelangel.pros...@gmail.com> wrote:
>
>> > It looks like you have a good handle on the code -- do you want to
>> submit a PR to GitHub to add such a parameter?
>>
>> Thanks, but I'm not really sure how to implement it in the
>> ProcessPoolExecutor, I just think the solution is probably related to the
>> code responsible of handling a failed initializer (since they do very
>> similar things). On the ThreadPoolExecutor maybe I could, but I haven't
>> really checked for side effects or weird things.
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/FGTSBGTXW63SSXFQBASLCTXAT4M6RGGN/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> --
> --Guido (mobile)
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/WSE2KMKFMPG2RYSJSGMRZPDJLM7DZKSG/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KALEV4452O2OABDDUELNQVJOWFCG5FJN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Recommend UTF-8 mode on Windows

[Kyle Stanley]

Inada Naoki wrote:
> Current header is:

> Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC
> v.1916 64 bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.

> I'm proposing adding one more line:

> UTF-8 mode is disabled. (See https://url.to/utf8mode)

Ah, that should be fine. I don't have any issues with (2) and (3) then.

Inada Naoki wrote:
> If the Python installer set PYTHONUTF8=1 environment variable,
> it may affects applications using embeddable Python or py2exe.

> So it may break applications which assume the default text
> encoding is "mbcs".

In that case, I think we should hold off on (4) until it's tested for a
decent variety of different applications, or at the least consider leaving
it off by default (unticked checkbox) with a note of some form that warns
users of the possible legacy incompatibility with MBCS.

I'll admit though that I have no experience working with any application
that implicitly assumes MBCS is the default encoding format, is this a
common occurrence in some older applications? I could imagine this being
the case in legacy applications that need international encoding, but I
have zero idea of how common it is in reality.

As a side note, Microsoft recommends not using MBCS for encoding in newer
applications in their Visual C++ docs:

"MBCS is a legacy technology and is not recommended for new development." (
https://docs.microsoft.com/en-us/cpp/text/support-for-multibyte-character-sets-mbcss?view=vs-2019
).

This might also be useful for getting a general idea of MBCS expectations
when it comes to Windows development, although some of it is specific to
C++:
https://docs.microsoft.com/en-us/cpp/text/general-mbcs-programming-advice?view=vs-2019
.

On Sun, Jan 12, 2020 at 1:28 AM Inada Naoki  wrote:

> On Sat, Jan 11, 2020 at 11:03 AM Kyle Stanley  wrote:
> >
> > > 1. Recommend it in the official document "Using Python on Windows" [2].
> > > 2. Show the UTF-8 mode status in the command line mode header [3] on
> Windows.
> > > 3. Show the link to the UTF-8 mode document in the command line mode
> header too.
> > > 4. Add checkbox to set "PYTHONUTF8=1" environment variable in the
> installer.
> >
> > > How do you think?
> >
> > At the least, I'm in favor of recommending UTF-8 mode in the
> documentation for Windows users. It seems like it would fit well under the
> "Configuring Python" section of the page (
> https://docs.python.org/3/using/windows.html#configuring-python).
> >
> > I'm undecided on the others, as I don't know what (2) and (3) would
> specifically entail.
>
> Current header is:
>
>   Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC
> v.1916 64 bit (AMD64)] on win32
>   Type "help", "copyright", "credits" or "license" for more information.
>
> I'm proposing adding one more line:
>
>   UTF-8 mode is disabled.  (See https://url.to/utf8mode)
>
>
> > As far as (4) goes:
> >
> > > If setting "PYTHONUTF8=1" environment variable is too danger
> > > to recommend widely, we may be able to add per-installation
> > > (and per-venv if needed) option file (site.cfg in the directory same to
> > > python.exe) to enable UTF-8 mode.
> >
> > Would you mind elaborating on this point? In particular, what specific
> dangers/risks might be associated with setting that specific env var during
> installation? IIRC, the installer already configures a few others by
> default (I don't recall their names).
> >
>
> If the Python installer set PYTHONUTF8=1 environment variable,
> it may affects applications using embeddable Python or py2exe.
>
> So it may break applications which assume the default text
> encoding is "mbcs".
>
>
> > > But it may make Python startup process more complex...
> >
> > I would definitely prefer to have a checkbox to configure "PYTHONUTF8"
> during installation rather than requiring it to be done manually, assuming
> it can be done safely and effectively across different systems. Not only
> for the sake of lowering complexity, but also because it takes less time
> and effort. That can add up significantly when you consider the volume of
> users.
> >
>
> Even when we add per-installation config file, we can add it from
> the Python installer.
>
>
> --
> Inada Naoki  
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/D337PG5Z3KW4DC3RQTSGHXHL3JWK5LJY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Recommend UTF-8 mode on Windows

[Kyle Stanley]

 > 1. Recommend it in the official document "Using Python on Windows" [2].
> 2. Show the UTF-8 mode status in the command line mode header [3] on
Windows.
> 3. Show the link to the UTF-8 mode document in the command line mode
header too.
> 4. Add checkbox to set "PYTHONUTF8=1" environment variable in the
installer.

> How do you think?

At the least, I'm in favor of recommending UTF-8 mode in the documentation
for Windows users. It seems like it would fit well under the "Configuring
Python" section of the page (
https://docs.python.org/3/using/windows.html#configuring-python).

I'm undecided on the others, as I don't know what (2) and (3) would
specifically entail. As far as (4) goes:

> If setting "PYTHONUTF8=1" environment variable is too danger
> to recommend widely, we may be able to add per-installation
> (and per-venv if needed) option file (site.cfg in the directory same to
> python.exe) to enable UTF-8 mode.

Would you mind elaborating on this point? In particular, what specific
dangers/risks might be associated with setting that specific env var during
installation? IIRC, the installer already configures a few others by
default (I don't recall their names).

> But it may make Python startup process more complex...

I would definitely prefer to have a checkbox to configure "PYTHONUTF8"
during installation rather than requiring it to be done manually, assuming
it can be done safely and effectively across different systems. Not only
for the sake of lowering complexity, but also because it takes less time
and effort. That can add up significantly when you consider the volume of
users.

On Fri, Jan 10, 2020 at 6:47 AM Inada Naoki  wrote:

> Hi, all.
>
> I believe UTF-8 should be chosen by default for text encoding.
>
> * The default encoding for Python source file is UTF-8.
> * VS Code and even notepad uses UTF-8 by default.
> * Text files downloaded from the Internet is probably UTF-8.
> * UTF-8 is used when you use WSL regardless your system code page.
> * Windows 10 (1903) adds per-process option to change active code page
> to UTF-8 and call the system code page "legacy". [1]
>
> But it is difficult to change the default text encoding of Python
> for backward compatibility.  So I want to recommend the UTF-8 mode:
>
> * The default text encoding become UTF-8
> * When you need to use legacy ANSI code page, you can
>   use "mbcs" codec.
> * You can disable it when you need to run Python application
>   relying on the legacy system encoding.
>
> But it is not well known yet.  And setting the environment variable
> is a bit difficult for people who are learning programming with Python.
>
> So I want to propose this:
>
> 1. Recommend it in the official document "Using Python on Windows" [2].
> 2. Show the UTF-8 mode status in the command line mode header [3] on
> Windows.
> 3. Show the link to the UTF-8 mode document in the command line mode
> header too.
> 4. Add checkbox to set "PYTHONUTF8=1" environment variable in the
> installer.
>
> How do you think?
>
> If setting "PYTHONUTF8=1" environment variable is too danger
> to recommend widely, we may be able to add per-installation
> (and per-venv if needed) option file (site.cfg in the directory same to
> python.exe) to enable UTF-8 mode.
> But it may make Python startup process more complex...
>
> Regards,
>
> [1]:
> https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page
> [2]: https://docs.python.org/3/using/windows.html
> [3]: Currently, Python version and "Type "help",..." are printed.
>
> --
> Inada Naoki  
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/524JQFZ4RMRLU7DBPSQDR735Z2UMSWQG/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/2PFSB6AZZDAY32HFCMCVTO5NHATIXOP7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: `set` API and various sorting problems

[Kyle Stanley]

Marco Sulla wrote:
> First, C and assembly has no boolean type.

Not to focus on this because it's not particularly relevant to the original
topic, but this isn't true: C99 added the boolean type "_Bool", which can
be aliased as "bool" with the  header.

On Thu, Dec 26, 2019 at 8:53 PM Marco Sulla via Python-ideas <
python-ideas@python.org> wrote:

> Andrew Barnert wrote:
> > if you’re going to be a pedant, the floats in
> > whatever Python you’re using right now are probably 854/754-1985
> doubles, not 754-2019
> > binary64s.
>
> Mr. Andrew Barnet,
> if pedant means adhere to the standard, yes, I'm a pedant.
>
> > > This is because NaN, IMHO, it's not the right name
> > > for the entity. A number can't be Not a Number. It's an Undefined
> Number.
> > OK, so you really are arguing against IEEE float here.
>
> So now I'm no more a pedant. No, I'm an heretic, because I don't like the
> Not A Number name but I prefer Undefined Number?
> A rose with any another name has the same smell.
>
> > > But IEEE 754-2019 is written mainly for C and
> > > Assembly. And you can't return nothing in a comparison between two
> floats. So they take
> > > the most practical solution: return false.
> > > They defined a whole complicated system of signaling vs. quiet nan
> values and
> > signaling (on sNaN) vs. non-signaling operations and multiple forms of
> exception handling
> > from traps to checkable flags to substitution of default values. You can
> use much of that
> > in C (and all of it in assembly) if you want to, even if people usually
> don’t.
>
> Yes, I perfectly know, I read the specs. You forgot the payload. Thank you
> for confirming what I said.
>
> > Also, who says you can’t return nothing in a comparison between two
> floats? You can
> > write a named function that has any API you want (say, it returns -1, 0,
> 1, or NaN, or
> > even an enum of less, greater, equal, left nan, right nan, both nan,
> other incomparable)
> > instead of a Boolean function that has to return true or false.
>
> First, C and assembly has no boolean type. And I can write whatever I
> want, but IEEE 754-2019 clearly says that every comparison with NaN must
> return false, that is zero in C.
>
> > But the fact that C (and countless other languages, including Python)
> also provides the
> > operators and people use them extensively implies that many people don’t
> think the
> > operators are useless, even if you do.
>
> I never said the operators are useless. Can you quote please?
>
> > > This obviously does not apply to total ordering. Even
> > > if I think mathematically has no sense, IEEE 754 clearly defined a
> total ordering, also
> > > for NaNs. So a total_ordering() key function, passed to sort
> functions, should order as
> > > IEEE 754 declares, without raising exceptions. There's a bug opened
> somewhere about this
> > > key function.
> > > This is what I've done. Unluckily, the API now works this way. It's
> not a big problem
> > > anyway. You have only to pass as key to the sort function:
> > > def sortNans(x):
> > >   try:
> > >   if math.isnan(x):
> > >   return float("+inf")
> > >   except TypeError
> > >   pass
> > >
> > >   return x
> > >
> > But this isn’t the same thing that IEEE totalOrder defines.
>
> I clearly stated that the key function total_order() is being developed,
> that there's a bug opened by Tim Peters and that what I wrote has nothing
> to do with total ordering. Can you please read my post with more attention?
> It's a bit frustrating to explain the things two times, and it's not
> respectful for who writes.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/NJOTBTLR3JIJBATTF56J3ZCY2DSIFL5J/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/NIWHN7LBYUITZYFNYUVB3S5S4S5BFDSW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Giving Loops names

[Kyle Stanley]

A very similar proposal was rejected:
https://www.python.org/dev/peps/pep-3136/.

Also, see the discussion in
https://mail.python.org/archives/list/python-ideas@python.org/thread/26PAGUTJ66S3BIZTL57IHUIYAHA7YOTV/#23H2S2AJFPRK46UID5MBHJJ7A3ERUBJA
.

If you'd still like to propose this, you'll have to present an argument
against the reasons why the other proposals were rejected.

On Mon, Dec 30, 2019 at 10:19 AM Oskar Promeuschel <
promeuschel_os...@lsh-marquartstein.de> wrote:

> Hello,
>
> I was thinking about giving loops names. And It would be interesting if
> through this name in a continue or break statement could be called.
>
> A little example what I mean:
>
>
>
> loop *"outloop" **while True*:
> actmonth = datetime.datetime.now().month
> loop *"innerloop" **for *i *in *range(0, int(input())):
> *if *datetime.datetime.now().month != actmonth:
> *break *
> *"outloop"**else*:
> print(*"Waiting for a new Month..."*)
>
>
>
> *# Do some stuff if loop ended before new Month*
>
>
>
> Without giving names it’s always needed to define a new variable and a if
> statement outside of the innerloop, like this:
>
>
>
> *while True*:
> actmonth = datetime.datetime.now().month
> shouldend =
> *Falsefor *i *in *range(0, int(input())):
> *if *datetime.datetime.now().month != actmonth:
> shouldend =
>
> *Truebreakelse*:
> print(*"Waiting for a new Month..."*)
>
> *if *shouldend:
>
>
> *break**# Do some stuff if loop ended before new Month*
>
>
>
> It’s possible without it, but it is more readable.
>
> Maybe another option is, but that’s too deep, to make loops to objects.
>
>
>
> Best Regards,
>
> Oskar Promeuschel
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/MWOQ67ARKHAUE6GN6WOVITPU3WAU5BEL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DKU3AGEBLZCAGUAZ37PPUATYP7A5BLW2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: "and if" and "or if" for trailing if statements - including explicit fallthrough

[Kyle Stanley]

> Python is not an OOP language. If you want to breathe and preach OOP, you
should quite frankly just use Java.

Python isn't *strictly* or *exclusively* an object-oriented programming
language, but since it does have strong support for OOP features, saying
that "Python is not an OOP language" simply isn't true. As a highly general
purpose language, Python allows its users to choose between paradigms based
on unique needs and preferences.

I understand where you're coming from here, but I don't think it's
particularly productive to push those who have a strong preference for OOP
over to Java. Instead, try to explain why you prefer a different paradigm
in this particular situation. If you can't see eye-to-eye with the other
person, that's perfectly fine, but at least you have the chance of engaging
in a productive discussion and perhaps learning a new perspective.


On Sat, Dec 21, 2019 at 3:10 PM Soni L.  wrote:

>
>
> On 2019-12-21 4:58 p.m., Gregory Salvan wrote:
>
> Years ago there was an interesting movement called anti-if campaign, now
> it's more promotional, but the concept of "anti if" may help you find ways
> to remove the cases where your suggest "and if" and "or if" can apply.
> This article is particularly well written:
> https://code.joejag.com/2016/anti-if-the-missing-patterns.html
>
> If you have cases where you think "and if" and "or if" can be helpful, you
> probably underuse oop.
>
>
> I would very much rather not be told that OOP is the be-all end-all of
> programming. Python is not an OOP language. If you want to breathe and
> preach OOP, you should quite frankly just use Java.
>
>
> Le sam. 21 déc. 2019 à 20:32, Soni L.  a écrit :
>
>>
>>
>> On 2019-12-21 4:15 p.m., Andrew Barnert wrote:
>> > > On Dec 21, 2019, at 08:41, Soni L.  wrote:
>> > >
>> > > I'd like to see the ability to do:
>> > >
>> > > if x:
>> > >  1
>> > > and if y:
>> > >  2
>> > > or if z:
>> > >  3
>> > >
>> > > The truth table for these would be:
>> > >
>> > > x | y | z | result
>> > > 0 | _ | 0 | (none)
>> > > 0 | _ | 1 | 3
>> > > 1 | 0 | _ | 1,3
>> > > 1 | 1 | _ | 1,2,3
>> > >
>> > > and each statement is evaluated once, when encountered. (as such, y
>> and z may not be evaluated at all, if their evaluation is not necessary to
>> determine the outcome.)
>> >
>> > So this is equivalent to:
>> >
>> > if x:
>> > if y:
>> > 1, 2, 3
>> > else:
>> > 1, 3
>> > elif z:
>> > 3
>> >
>> > I can see how the former saves me having to repeat the 3 three times.
>> But the cost is being less obvious about when exactly I get a 3 so I’m
>> forced to work it through step by step—including the confusion about the
>> 1,0,0 case, which, as you mentioned, is only clear if you imagine putting
>> an else at the end (although maybe you’d get used to that once you’d read
>> through enough of these?). It’s even less obvious if you do throw in an
>> elif, or just add an and if to the end (so now the condition to get there
>> is not “x and y or z and w” but, I think, “((x and y) or z) and w”?
>> >
>> > Does that advantage outweigh the disadvantage? Certainly not for this
>> example. But that’s probably because even the rewritten example is
>> meaningless and useless. Maybe it would be different with a realistic use
>> case, but I can’t imagine what that would be. Surely you must have some
>> case where you really wanted this, that motivated you to propose it?
>> >
>> >
>> >
>>
>> "1" and "2" and "3" are pieces of code, ofc.
>>
>> it's actually more like:
>>
>> if flag := x:
>>1
>>if y:
>>  2
>> if flag or z: # note that "or" is short-circuiting.
>>3
>>
>> but more efficiently implemented in bytecode without that "flag" local
>> showing up.
>>
>> it's literally meant for easing the translation of switch statements
>> from other languages and nothing else. (well, at least the "or if" part.
>> the "and if" part isn't very useful. maybe to reduce deep indentation I
>> guess?)
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/NNVZQ5HITRW2357ED5SEWL5UDL4X5LWR/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to 
> python-ideas-leave@python.orghttps://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/WNJUFZRW2ZJGE6CEVDTIESDBP47VD4N6/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to 

[Python-ideas] Re: Argumenting in favor of first()

[Kyle Stanley]

Stephen J. Turnbull wrote:
> Worse, running the same program again with the *same* set can change
> which element is first, I believe.

Yes, this is correct. For example, if you had a minimal Python file named
"some_set.py" with only the following:

```
s = set('abcdefg')
```

and then run something like this "import some_set; print(list(some_set.s))"
the order of the elements will be randomized each time. Within each
individual interpreter, the order will not change though (including if you
del and then import some_set again):

```
[aeros:~]$ python -c "import some_set; print(list(some_set.s));
print(list(some_set.s))"
['f', 'a', 'c', 'g', 'b', 'd', 'e']
['f', 'a', 'c', 'g', 'b', 'd', 'e']
[aeros:~]$ python -c "import some_set; print(list(some_set.s));
print(list(some_set.s))"
['a', 'd', 'f', 'b', 'e', 'g', 'c']
['a', 'd', 'f', 'b', 'e', 'g', 'c']
[aeros:~]$ python -c "import some_set; print(list(some_set.s)); del
some_set; import some_set; print(list(some_set.s))"
['f', 'b', 'g', 'e', 'c', 'a', 'd']
['f', 'b', 'g', 'e', 'c', 'a', 'd']
[aeros:~]$ python -c "import some_set; print(list(some_set.s)); del
some_set; import some_set; print(list(some_set.s))"
['c', 'e', 'a', 'f', 'g', 'b', 'd']
['c', 'e', 'a', 'f', 'g', 'b', 'd']
```

IIUC, the "ordering seed" for all sets is randomly determined when the
interpreter is first created, and remains the same throughout its lifespan.
I'm not 100% certain that it's interpreter dependent, but it seems like it
would be.

Personally, I don't think this makes `first()` any worse off though. I
think it should be expected that attempting to extract the "first" element
from an unordered container would not consistently return same one.

On Wed, Dec 11, 2019 at 8:25 PM Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Chris Angelico writes:
>
>  > And ordered vs unordered is also a big difference. Should first()
>  > raise an error with sets because there's no real concept of "the first
>  > element"?
>
> Probably not.  I would prefer that it not be implemented at all, but
> if it is implemented, its behavior should respect the intuition of the
> majority of those who want it, which seems to me to be "a variant of
> next() that doesn't raise and returns None by default on an empty
> iterable."
>
>  > With a list, first(x) will remain the same value even if you add
>  > more to the end of the list, but unrelated mutations to a set might
>  > change which element is "first".
>
> Worse, running the same program again with the *same* set can change
> which element is first, I believe.  Also, the elements of the set
> might have a natural (pre)order, which first() won't respect.  I'm not
> sure if this last is a real problem, given that sequences have the
> same issue (the sequence's order differs from the natural order).
> However, to me the fact that a set's iteration order is implicit while
> a sequence's is explicit suggests it might in some contexts.
>
>  > Does that mean that first() and next() are undefined for sets?
>
> first() is undefined.  next() is defined by reference to iterating
> over the set (that's why I don't have a problem with iterating over a
> set).
>
>  > No. We just accept that there are these differences.
>
> Well, if first() is implemented, we'll have to accept it.  It's not
> clear to me that we should.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/7722U6NFSGKEFVFZM6SCXB246AAS6ETN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XHCEOUSKMYHAFJC7UF2NMPXFGZ3JT7FW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Argumenting in favor of first()

[Kyle Stanley]

Tim Peters wrote:
> It's fair enough, although rather than "wrong" I'd say more that it's
> inappropriately applying design principles that work well in most of
> Python's libraries to an area where they don't.  The very fact that
> half the itertools docs are devoted to "recipes" kinda suggests it
> knows it's leaving its intended audience hanging ;-)

Yeah I'm personally not a huge fan of how large the recipes section of
itertools is, compared to the actual module contents. My personal opinion
is that the recipes section should be primarily for niche or complex
algorithms that wouldn't be general purpose enough to be used for the
module, but still have legitimate use cases. If it's general purpose enough
and has a legitimate use case, it should be added as a function, not as a
recipe.

Instead, it seems to have gradually turned into: "Put everything in the
recipes section that might be useful and can be implemented using existing
tools, even if adding a dedicated function would be more readable. Only add
a new function if it doesn't already exist in some other form, no matter
how obscure." (oversimplified, but that's been my interpretation). The last
time a new function was added to itertools was back in 3.2 (2011), for
itertools.accumulate().

I think itertools.first() is a good example of something that would be
useful and general-purpose enough to be included as a function in
itertools. `first_item = first(iterable)` is much easier to read than
`first_item = next(iter(iterable))`, not to mention first(iterable) raising
a ValueError instead of StopIteration on an empty iterable, which seems
much more clear to me.

I also find the default version (where an exception isn't raised) to be a
lot easier to read with first, especially if *default* is able to be
passed as a keyword argument. Compare `first_item = first(iterable,
default=None)` to `first_item = `next(iter(iterable), None)` (next doesn't
take any kwargs).

If even those who have been involved with Python's development since its
infancy aren't overly familiar with 2-arg next(), how could we reasonably
expect the average user to be? I'll admit that I had not even heard about
the 2-arg next() until reading the posts in this thread. My method of
extracting the first value from an iterable (when I wanted it to default to
None instead of raising StopIteration) was:

```py
try:
first_item = next(iter(iterable))
except StopIteration:
first_item = None
```

On Tue, Dec 10, 2019 at 8:15 PM Tim Peters  wrote:

> [Andrew Barnert ]
> > ...
> > I think you’re arguing that the philosophy of itertools is just wrong,
> at least for
> > the kind of code you usually write with it and the kind of people who
> usually
> > write that code. Is that fair, or am I misrepresenting you here?
>
> It's fair enough, although rather than "wrong" I'd say more that it's
> inappropriately applying design principles that work well in most of
> Python's libraries to an area where they don't.  The very fact that
> half the itertools docs are devoted to "recipes" kinda suggests it
> knows it's leaving its intended audience hanging ;-)
>
> It's hardly coincidence that the more-itertools and itertoolz packages
> are richer, in very similar ways, than Python's version.  The primary
> itertoolz author does a nice job of explaining that project's heritage
> starting here:
>
> https://toolz.readthedocs.io/en/latest/heritage.html
>
>
> > Meanwhile, I think Guido and some others accept the itertools philosophy
>
> Don't know about Guido, but certainly applies to Raymond.  Guido is
> more a practicality-beats-purity guy. but has no natural attraction to
> functional languages (Raymond partly does, with his APL background).
> Guido just sees a function here that would be nice to have.
>
> I do like functional languages, and always have, so it seems to fall
> on me here to advocate for what those weirdos value.  In part, no, the
> itertools namespace is not a precious resource that must be vigilantly
> minimized ;-)
>
>
> > but argue that first needs to be there because many of the kinds of
> people who
> > need it actually can’t just write it themselves (they don’t know about
> 2-arg next,
> > or they don’t understand the subtleties of leaking StopIteration, or
> whatever).
> > That’s a pretty different argument. (Not that there can’t be something
> to both
> > arguments, of course.)
>
> And I expect Raymond would say `first()` isn't needed at all - if it
> has to be addressed, make it recipe #30.
>
> There's something to that argument too.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/JPOU2L3RWCRCRZ37EPSNJLI4DF4LUNBX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

[Python-ideas] Re: Sets for easy interning(?)

[Kyle Stanley]

> It also interns many ints, and they can't be used as names at all.

To clarify on "many ints", integers in the range of -5 to 256 (inclusive)
are interned. This can be demonstrated with the following:

```py
>>> a = 256
>>> b = 256
>>> a is b
True
>>> a = 257
>>> b = 257
>>> a is b
False
>>> a = -5
>>> b = -5
>>> a is b
True
>>> a = -6
>>> b = -6
>>> a is b
False
```

On Wed, Dec 4, 2019 at 3:30 AM Steven D'Aprano  wrote:

> On Wed, Dec 04, 2019 at 07:11:39PM +1300, Greg Ewing wrote:
> > On 4/12/19 12:53 pm, Soni L. wrote:
> > >Okay, sometimes it's also used for that. But the main use-case is for
> > >lowering RAM usage for immutable objects.
> >
> > Citation needed. If that's true, why does Python intern
> > names used in code, but not strings in general?
>
> py> s = "ab1234z"
> py> t = "ab1234z"
> py> s is t
> True
>
>
> CPython doesn't *just* intern names. Nor does it intern every string.
> But it interns a lot of strings which aren't used as names, including
> some which cannot be used as names:
>
> py> a = "+"
> py> b = "+"
> py> a is b
> True
>
> It also interns many ints, and they can't be used as names at all.
>
> Here's a good explanation of interning in Python 2.7, including a great
> example of how interning strings can reduce memory usage by 68%.
>
>
> http://guilload.com/python-string-interning/
>
>
>
> --
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/H6WOEL5CJ67P36AE3H425EDY4MGWE7K2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UPN7C7I4I3QBJXZ4K3NX5B5IJECHBMC7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Argumenting in favor of first()

[Kyle Stanley]

Serhiy Storchaka wrote:
> Would not adding a recipe in the itertools documentation or the tutorial
> help?

I think adding a recipe in the itertools documentation might help, but I
don't know that it would be a great fit for the tutorial. It seems a bit
too specific and could be a distraction from the main purpose of the
tutorial, which is to get a basic understanding of the fundamentals of
Python.

On Mon, Dec 9, 2019 at 6:29 AM Serhiy Storchaka  wrote:

> 09.12.19 07:41, Guido van Rossum пише:
> > Nobody is going to write a blog post about 2-arg next() (there just
> > isn't enough for more than a sentence or two) but people write tutorials
> > about itertools all the time, since it's such a rich module. So I think
> > it's likely that first() will get some exposure that way.
>
> Would not adding a recipe in the itertools documentation or the tutorial
> help?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/IZWPWG5QRSA3GKNDQCMU6RLWOXFTVTPZ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JDULZHPX3GUG5O4XTW7FJTA5GQPPPKUB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Fwd: Re: Fwd: re.findfirst()

[Kyle Stanley]

> Hold on, what is finditer().group(1) supposed to mean here? You’d need
next(finditer()).group(1) or next(m.group(1) for m in finditer()) or
something.

That was a mistake, I intended to write re.search().group(1), not
re.finditer().group(1). I clarified this in another reply to the thread
about an hour ago. Sorry for the confusion, I wrote the reply after being
up for a while and got re.findter() and re.search() mixed up in my head.
You're correct.

On Sat, Dec 7, 2019 at 7:57 PM Andrew Barnert  wrote:

> On Dec 7, 2019, at 04:51, Kyle Stanley  wrote:
>
>
> Alternatively: creating a new section under
> https://docs.python.org/3.8/library/re.html#regular-expression-examples,
> titled "Finding the first match", where it briefly explains the difference
> in behavior between using re.findall()[0] and re.finditer().group(1) (or
> re.finditer.group() when there's not a subgroup).
>
>
> Hold on, what is finditer().group(1) supposed to mean here? You’d need
> next(finditer()).group(1) or next(m.group(1) for m in finditer()) or
> something. But if you just want the first match, why are you using either
> findall or finditer instead of just search? Isn’t that exactly the
> confusion this thread was hoping to resolve, rather than forcing even more
> novices to deal with it by pushing them into it in a section named “Finding
> the first match”?
>
> Also (when there are subgroups), surely the relevant difference is either
> between findall()[0][0] and next(finditer()).group(1), which both return
> the first group of the first match, or between findall()[0] and
> next(finditer()).groups(), which both return a tuple of groups of the first
> match, not between findall()[0] and next(finditer()).group(1), which return
> a tuple vs. just the first one?
>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/O4AUPOBOJJPPUMBPXJ2PRZPFY33CTJ6J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Fwd: Re: Fwd: re.findfirst()

[Kyle Stanley]

> Alternatively: creating a new section under
https://docs.python.org/3.8/library/re.html#regular-expression-examples,
titled
"Finding the first match", where it briefly explains the difference in
behavior between using re.findall()[0] and re.finditer().group(1) (or
re.finditer.group() when there's not a subgroup). Based on the discussions
in this thread and code examples, this seems to be rather commonly
misunderstood.

Another clarification: The above should be about re.search(), not
re.finditer(). I sent this reply late at night and got the two mixed up.

On Sat, Dec 7, 2019 at 7:47 AM Kyle Stanley  wrote:

> > Code examples should of course be used sparingly, but I think
> re.finditer() could benefit from at least one
>
> Clarification: I see that there's an example of it being used in
> https://docs.python.org/3.8/library/re.html#finding-all-adverbs-and-their-positions
> and one more complex example with
> https://docs.python.org/3.8/library/re.html#writing-a-tokenizer. I was
> specifically referring to including a basic example directly within
> https://docs.python.org/3.8/library/re.html#re.finditer, similar to the
> section for https://docs.python.org/3.8/library/re.html#re.split or
> https://docs.python.org/3.8/library/re.html#re.sub.
>
> Alternatively: creating a new section under
> https://docs.python.org/3.8/library/re.html#regular-expression-examples,
> titled "Finding the first match", where it briefly explains the difference
> in behavior between using re.findall()[0] and re.finditer().group(1) (or
> re.finditer.group() when there's not a subgroup). Based on the discussions
> in this thread and code examples, this seems to be rather commonly
> misunderstood.
>
> On Sat, Dec 7, 2019 at 7:29 AM Kyle Stanley  wrote:
>
>> Serhiy Storchaka wrote:
>> > My concern is that this will add complexity to the module documentation
>> > which is already too complex. re.findfirst() has more complex semantic
>> > (if no capture groups returns this, if one capture group return that,
>> > and in other cases return even something of different type) than
>> > re.search() which just returns a match object or None. This will
>> > increase chance that the user miss the appropriate function and use
>> > suboptimal functions like findall()[0].
>>
>> > re.finditer() is more modern and powerful function than re.findall().
>> > The latter may be even deprecated in future.
>>
>> Hmm, perhaps another consideration then would be to think of improvements
>> to make to the existing documentation, particularly with including some
>> code examples or expanding upon the docs for re.finditer() to make its
>> usage more clear. Personally, it took me quite a while to understand its
>> role in the module (as someone who does not use it on a frequent basis).
>> Code examples should of course be used sparingly, but I think re.finditer()
>> could benefit from at least one. Especially considering that far less
>> complex functions in the module have several examples. See
>> https://docs.python.org/3.8/library/re.html#re.finditer.
>>
>> Serhiy Storchaka wrote:
>> > > Another option to consider might be adding a boolean parameter to
>> > > re.search() that changes the behavior to directly return a string
>> > > instead of a match object, similar to re.findall() when there are not
>> > > multiple subgroups.
>>
>> > Oh, no, this is the worst idea!
>>
>> Yeah, after having some time to reflect on that idea a bit more I don't
>> think it would work. That would just end up adding confusion to
>> re.search(), ultimately defeating the purpose of the parameter in the first
>> place. It would be too drastic of a change in behavior for a single
>> parameter to make.
>>
>> Thanks for the honesty though, not all of my ideas are good ones. But, if
>> I can come up with something half-decent every once in a while I think it's
>> worth throwing them out there. (:
>>
>>
>>
>>
>> On Sat, Dec 7, 2019 at 2:56 AM Serhiy Storchaka 
>> wrote:
>>
>>> 06.12.19 23:20, Kyle Stanley пише:
>>> > Serhiy Storchaka wrote:
>>> >  > It seems that in most cases the author just do not know about
>>> >  > re.search(). Adding re.findfirst() will not fix this.
>>> >
>>> > That's definitely possible, but it might be just as likely that they
>>> saw
>>> > re.findall() as being more simple to use compared to re.search().
>>> > Although it has worse performance by a substantial amount when parsing
>>> > decent amounts of text (assuming th

[Python-ideas] Re: Fwd: Re: Fwd: re.findfirst()

[Kyle Stanley]

> Code examples should of course be used sparingly, but I think
re.finditer() could benefit from at least one

Clarification: I see that there's an example of it being used in
https://docs.python.org/3.8/library/re.html#finding-all-adverbs-and-their-positions
and one more complex example with
https://docs.python.org/3.8/library/re.html#writing-a-tokenizer. I was
specifically referring to including a basic example directly within
https://docs.python.org/3.8/library/re.html#re.finditer, similar to the
section for https://docs.python.org/3.8/library/re.html#re.split or
https://docs.python.org/3.8/library/re.html#re.sub.

Alternatively: creating a new section under
https://docs.python.org/3.8/library/re.html#regular-expression-examples,
titled "Finding the first match", where it briefly explains the difference
in behavior between using re.findall()[0] and re.finditer().group(1) (or
re.finditer.group() when there's not a subgroup). Based on the discussions
in this thread and code examples, this seems to be rather commonly
misunderstood.

On Sat, Dec 7, 2019 at 7:29 AM Kyle Stanley  wrote:

> Serhiy Storchaka wrote:
> > My concern is that this will add complexity to the module documentation
> > which is already too complex. re.findfirst() has more complex semantic
> > (if no capture groups returns this, if one capture group return that,
> > and in other cases return even something of different type) than
> > re.search() which just returns a match object or None. This will
> > increase chance that the user miss the appropriate function and use
> > suboptimal functions like findall()[0].
>
> > re.finditer() is more modern and powerful function than re.findall().
> > The latter may be even deprecated in future.
>
> Hmm, perhaps another consideration then would be to think of improvements
> to make to the existing documentation, particularly with including some
> code examples or expanding upon the docs for re.finditer() to make its
> usage more clear. Personally, it took me quite a while to understand its
> role in the module (as someone who does not use it on a frequent basis).
> Code examples should of course be used sparingly, but I think re.finditer()
> could benefit from at least one. Especially considering that far less
> complex functions in the module have several examples. See
> https://docs.python.org/3.8/library/re.html#re.finditer.
>
> Serhiy Storchaka wrote:
> > > Another option to consider might be adding a boolean parameter to
> > > re.search() that changes the behavior to directly return a string
> > > instead of a match object, similar to re.findall() when there are not
> > > multiple subgroups.
>
> > Oh, no, this is the worst idea!
>
> Yeah, after having some time to reflect on that idea a bit more I don't
> think it would work. That would just end up adding confusion to
> re.search(), ultimately defeating the purpose of the parameter in the first
> place. It would be too drastic of a change in behavior for a single
> parameter to make.
>
> Thanks for the honesty though, not all of my ideas are good ones. But, if
> I can come up with something half-decent every once in a while I think it's
> worth throwing them out there. (:
>
>
>
>
> On Sat, Dec 7, 2019 at 2:56 AM Serhiy Storchaka 
> wrote:
>
>> 06.12.19 23:20, Kyle Stanley пише:
>> > Serhiy Storchaka wrote:
>> >  > It seems that in most cases the author just do not know about
>> >  > re.search(). Adding re.findfirst() will not fix this.
>> >
>> > That's definitely possible, but it might be just as likely that they
>> saw
>> > re.findall() as being more simple to use compared to re.search().
>> > Although it has worse performance by a substantial amount when parsing
>> > decent amounts of text (assuming the first match isn't at the end),
>> > ``re.findall()[0]`` /consistently/ returns the first string that was
>> > matched, as long as no subgroups were used. This allows them to
>> > circumvent the usage of match objects entirely, which makes it a bit
>> > easier to learn. Especially for those who are less familiar with OOP,
>> or
>> > are already familiar with other popular flavors of regex (such as JS).
>> >
>> > I'll admit this is mostly speculation, but I think there's an
>> especially
>> > large number of re users (compared to other modules) that aren't
>> > necessarily developers, and might just be someone who wants to write a
>> > script to quickly parse some documents. These types of users are the
>> > ones who would likely benefit the most from the proposed
>> re.findfirst(),
>> > particularly if it directly returns a strin

[Python-ideas] Re: Fwd: Re: Fwd: re.findfirst()

[Kyle Stanley]

Serhiy Storchaka wrote:
> My concern is that this will add complexity to the module documentation
> which is already too complex. re.findfirst() has more complex semantic
> (if no capture groups returns this, if one capture group return that,
> and in other cases return even something of different type) than
> re.search() which just returns a match object or None. This will
> increase chance that the user miss the appropriate function and use
> suboptimal functions like findall()[0].

> re.finditer() is more modern and powerful function than re.findall().
> The latter may be even deprecated in future.

Hmm, perhaps another consideration then would be to think of improvements
to make to the existing documentation, particularly with including some
code examples or expanding upon the docs for re.finditer() to make its
usage more clear. Personally, it took me quite a while to understand its
role in the module (as someone who does not use it on a frequent basis).
Code examples should of course be used sparingly, but I think re.finditer()
could benefit from at least one. Especially considering that far less
complex functions in the module have several examples. See
https://docs.python.org/3.8/library/re.html#re.finditer.

Serhiy Storchaka wrote:
> > Another option to consider might be adding a boolean parameter to
> > re.search() that changes the behavior to directly return a string
> > instead of a match object, similar to re.findall() when there are not
> > multiple subgroups.

> Oh, no, this is the worst idea!

Yeah, after having some time to reflect on that idea a bit more I don't
think it would work. That would just end up adding confusion to
re.search(), ultimately defeating the purpose of the parameter in the first
place. It would be too drastic of a change in behavior for a single
parameter to make.

Thanks for the honesty though, not all of my ideas are good ones. But, if I
can come up with something half-decent every once in a while I think it's
worth throwing them out there. (:




On Sat, Dec 7, 2019 at 2:56 AM Serhiy Storchaka  wrote:

> 06.12.19 23:20, Kyle Stanley пише:
> > Serhiy Storchaka wrote:
> >  > It seems that in most cases the author just do not know about
> >  > re.search(). Adding re.findfirst() will not fix this.
> >
> > That's definitely possible, but it might be just as likely that they saw
> > re.findall() as being more simple to use compared to re.search().
> > Although it has worse performance by a substantial amount when parsing
> > decent amounts of text (assuming the first match isn't at the end),
> > ``re.findall()[0]`` /consistently/ returns the first string that was
> > matched, as long as no subgroups were used. This allows them to
> > circumvent the usage of match objects entirely, which makes it a bit
> > easier to learn. Especially for those who are less familiar with OOP, or
> > are already familiar with other popular flavors of regex (such as JS).
> >
> > I'll admit this is mostly speculation, but I think there's an especially
> > large number of re users (compared to other modules) that aren't
> > necessarily developers, and might just be someone who wants to write a
> > script to quickly parse some documents. These types of users are the
> > ones who would likely benefit the most from the proposed re.findfirst(),
> > particularly if it directly returns a string as Guido is suggesting.
> >
> > I think at the end of the day, the critical question to answer is this:
> >
> > *Do we want to add a new helper function that's easy to use, consistent,
> > and provides good performance for finding the first match, even if the
> > functionality already exists within the module?*
>
> My concern is that this will add complexity to the module documentation
> which is already too complex. re.findfirst() has more complex semantic
> (if no capture groups returns this, if one capture group return that,
> and in other cases return even something of different type) than
> re.search() which just returns a match object or None. This will
> increase chance that the user miss the appropriate function and use
> suboptimal functions like findall()[0].
>
> re.finditer() is more modern and powerful function than re.findall().
> The latter may be even deprecated in future.
>
> In future we may add yet few functions/methods: re.rmatch() (like
> re.match(), but matches at the end of the string instead of the start),
> re.rsearch() (searches from the end), re.rfinditer() (iterates in the
> reversed order). Unlike to findfirst() they will implement features that
> cannot be easily expressed using existing functions.
>
> > Another option to consider might be adding a boolean parameter to
> > re.search() that ch

[Python-ideas] Re: Fwd: Re: Fwd: re.findfirst()

[Kyle Stanley]

Serhiy Storchaka wrote:
> Thank you Kyle for your investigation!

No problem, this seemed like an interesting feature proposal and I was
personally curious about the potential use cases. Thanks for the detailed
analysis, I learned a few new things from it. (:

Serhiy Storchaka wrote:
> -   clerk_name = name_re.findall(clerk)[0]
> +   clerk_name = name_re.search(clerk).group(1)

This pattern seems to be common across most of the above examples (minus
the last two), specifically replacing ``re.findall()[0]`` with
``re.findall().group(1)`` when there are subgroups within the regex or
``re.findall().group()`` without subgroups.

Serhiy Storchaka wrote:
> It seems that in most cases the author just do not know about
> re.search(). Adding re.findfirst() will not fix this.

That's definitely possible, but it might be just as likely that they saw
re.findall() as being more simple to use compared to re.search(). Although
it has worse performance by a substantial amount when parsing decent
amounts of text (assuming the first match isn't at the end),
``re.findall()[0]`` *consistently* returns the first string that was
matched, as long as no subgroups were used. This allows them to circumvent
the usage of match objects entirely, which makes it a bit easier to learn.
Especially for those who are less familiar with OOP, or are already
familiar with other popular flavors of regex (such as JS).

I'll admit this is mostly speculation, but I think there's an especially
large number of re users (compared to other modules) that aren't
necessarily developers, and might just be someone who wants to write a
script to quickly parse some documents. These types of users are the ones
who would likely benefit the most from the proposed re.findfirst(),
particularly if it directly returns a string as Guido is suggesting.

I think at the end of the day, the critical question to answer is this:

*Do we want to add a new helper function that's easy to use, consistent,
and provides good performance for finding the first match, even if the
functionality already exists within the module?*

Personally, I lean a bit more towards "yes", but I think that "no" would
also be a reasonable response.

>From my perspective, a significant reason why Python is appealing to so
many users that aren't professional developers is that it's much easier to
pick up the basics. Python allows users write a quick script with *decent*
performance without having to learn too much, compared to most other
mainstream programming languages. IMO, the addition of an re.findfirst()
helps to reinforce that reason.

Another option to consider might be adding a boolean parameter to
re.search() that changes the behavior to directly return a string instead
of a match object, similar to re.findall() when there are not multiple
subgroups. For example:

>>> re.search(" (\w) ", "there is a one letter word in the middle",
match_obj=False)
'a'

The above would have the same exact return value as
``pattern.findall()[0]``, but it's more efficient since it would only parse
the text until the first match is found, and it doesn't need to create a
list. For backwards compatibility, this parameter would default to True.
Feel free to change the name if you like the idea, "match_obj" was simply
the first one that came to my head.

The cons of this solution is that it might be excessively overloading
re.search(), and that it not be as noticeable or easy to find as the
addition of a new function. But, it could provide the same functionality as
the proposed re.findfirst(), without adding an entirely new function for
behavior that already exists.

On Fri, Dec 6, 2019 at 2:47 AM Serhiy Storchaka  wrote:

> 05.12.19 23:47, Kyle Stanley пише:
> >
> > Serhiy Storchaka wrote:
> >  > We still do not know a use case for findfirst. If the OP would show
> his
> >  > code and several examples in others code this could be an argument for
> >  > usefulness of this feature.
> >
> > I'm not sure about the OP's exact use case, but using GitHub's code
> > search for .py files that match with "first re.findall" shows a decent
> > amount of code that uses the format ``re.findall()[0]``. It would be
> > nice if GitHub's search properly supported symbols and regular
> > expressions, but this presents a decent number of examples. See
> > https://github.com/search?l=Python=first+re.findall=Code.
> >
> > I also spent some time looking for a few specific examples, since there
> > were a number of false positives in the above results. Note that I
> > didn't look much into the actual purpose of the code or judge it based
> > on quality, I was just looking for anything that seemed remotely
> > practical and contained something along the lines of
> > ``re.findall()[0]``. Several of the links below conta

[Python-ideas] Re: Fwd: Re: Fwd: re.findfirst()

[Kyle Stanley]

Serhiy Storchaka wrote:
> We still do not know a use case for findfirst. If the OP would show his
> code and several examples in others code this could be an argument for
> usefulness of this feature.

I'm not sure about the OP's exact use case, but using GitHub's code search
for .py files that match with "first re.findall" shows a decent amount of
code that uses the format ``re.findall()[0]``. It would be nice if GitHub's
search properly supported symbols and regular expressions, but this
presents a decent number of examples. See
https://github.com/search?l=Python=first+re.findall=Code.

I also spent some time looking for a few specific examples, since there
were a number of false positives in the above results. Note that I didn't
look much into the actual purpose of the code or judge it based on quality,
I was just looking for anything that seemed remotely practical and
contained something along the lines of ``re.findall()[0]``. Several of the
links below contain multiple lines where findfirst would likely be a better
alternative, but I only included one permalink per code file.

https://github.com/MohamedAl-Hussein/my_projects/blob/15feca5254fe1b2936d39369365867496ce5b2aa/fifa_workspace/fifa_market_analysis/fifa_market_analysis/items.py#L325
https://github.com/MohamedAl-Hussein/FIFA/blob/2b1390fe46f94648e5b0bcfd28bc67a3bc43f09d/fifa_data/fifa_data/items.py#L370
https://github.com/democracyworks/dog-catcher/blob/9f6200084d4505091399d36ab0d5e3379b04588c/new_jersey.py#L82
https://github.com/democracyworks/dog-catcher/blob/9f6200084d4505091399d36ab0d5e3379b04588c/connecticut.py#L182
https://github.com/jessyL6/CQUPTHUB-spiders_task1/blob/db73c47c0703ed01eb2a6034c37edd9e18abb2e0/ZhongBiao2/spiders/zhongbiao2.py#L176
https://github.com/kerinin/giscrape/blob/d398206ed4a7e48e1ef6afbf37b4f98784cf2442/giscrape/spiders/people_search.py#L26
https://github.com/songweifun/parsebook/blob/529a86739208e9dc07abbb31363462e2921f00a0/dao/parseMarc.py#L211

I'm sure there are far more examples and perhaps some more "realistic"
ones, I only went through the first few pages of results.

On Thu, Dec 5, 2019 at 3:08 PM Serhiy Storchaka  wrote:

> 05.12.19 21:07, Guido van Rossum пише:
> > The case for findfirst() becomes stronger! There seem plenty of ways to
> > get this wrong.
>
> I write several functions every day. There are many ways to get this
> wrong. But I do not propose to include all these functions in the
> stdlib. If I want to include even a single function, I try to find
> several examples that would benefit from adding this function in the
> stdlib. If I found less examples than I expected I withdraw my idea.
>
> We still do not know a use case for findfirst. If the OP would show his
> code and several examples in others code this could be an argument for
> usefulness of this feature.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/YKVXRQAST6B7CRNN7LFBZXWVHH6G42YC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/7P4ZZMIL2ZFJOONUSZPNUBOZTAAEMASY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Moving PEP 584 forward (dict + and += operators)

[Kyle Stanley]

> I think the small time difference you noticed is only due to method
lookup.

I believe it's a bit more than just the Python method lookup, but that
makes some difference. Within the C-API, list += other uses list_inplace_concat
[1]; whereas list.extend uses _PyList_Extend [2]. They both call list_extend
[3], but they're not exactly equivalent. With the in-place operation, the
intermediate results can be discarded right away. Also, at the bytecode
level, += can use the STORE_FAST instruction to directly the push the
result from the TOS to local var; whereas list.extend uses POP_TOP.

But I'll leave it at that, I don't want to focus much on performance;
especially not with comparing list += other vs ls.extend(other). As
mentioned elsewhere in the topic, performance isn't the goal of this PEP. I
was mostly just curious if dict |= other would likely be very slightly
faster than dict.update(other), and wanted some elaboration on where
performance might be a mild concern.

[1]:
https://github.com/python/cpython/blob/894331838b256412c95d54051ec46a1cb96f52e7/Objects/listobject.c#L996
[2]:
https://github.com/python/cpython/blob/894331838b256412c95d54051ec46a1cb96f52e7/Objects/listobject.c#L990
[3]:
https://github.com/python/cpython/blob/894331838b256412c95d54051ec46a1cb96f52e7/Objects/listobject.c#L996

On Tue, Dec 3, 2019 at 12:21 PM Antoine Rozo  wrote:

> I think the small time difference you noticed is only due to method lookup.
>
> Le mar. 3 déc. 2019 à 13:57, Kyle Stanley  a écrit :
> >
> > > Actually there's no need to optimize the |= operator -- for strings we
> have to optimize += *because* strings are immutable, but for dicts we would
> define |= as essentially an alias for .update(), just like the relationship
> between += and .extend() for lists, and then no unnecessary objects would
> be created.
> >
> > Yeah that's why I noted that any form of optimization for the |=
> operator on dicts would not be the same as += is for strings. I wasn't
> actually sure of what form any potential optimization would take for the |=
> operator though. What exactly was the performance question/point in
> reference to? The question seemed to imply that there would be some minor
> performance detriment from using |=, but it's not clear to me as to when
> that would be a factor.
> >
> > > ## What about performance?
> >
> > > Performance is not the only objective when using Python. Switching to
> inplace operators (here |=) is a generally useful and well-known technique
> (it also applies to string and list concatenation, for example).
> >
> > Also with lists, I recall that using the += operator is very slightly
> faster than list.extend() in most situations:
> >
> > >>> ls_plus_eq = """\
> > for i in range(1_000):
> >  ls += [x for x in range(10)]
> > """
> > >>> ls_extend = """\
> > for i in range(1_000):
> >  ls.extend([x for x in range(10)])
> > """
> > >>> timeit.timeit(ls_plus_eq, setup="ls = []", number=10_000)
> > 6.563132778996078
> > >>> timeit.timeit(ls_extend, setup="ls = []", number=10_000)
> > 6.695127692000824
> > >>> timeit.timeit("ls+=other", setup="ls = []; other=[i for i in
> range(100_000)]", number=10_000)
> > 4.400735091003298
> > >>> timeit.timeit("ls.extend(other)", setup="ls = []; other=[i for i in
> range(100_000)]", number=10_000)
> > 4.574331789997814
> > >>> timeit.timeit("ls+=other", setup="ls = []; other=[i for i in
> range(100)]", number=10_000_000)
> > 3.5332175369985634
> > >>> timeit.timeit("ls.extend(other)", setup="ls = []; other=[i for i in
> range(100)]", number=10_000_000)
> > 3.7756526679950184
> >
> > (Python 3.8)
> >
> > It seems to be a difference of only ~2-4% in most cases (~6-7% with the
> last set), but I find it interesting that += is barely faster. Of course,
> some of the above examples are fairly unrealistic, for most practical use
> cases they're essentially the same. I tend to prefer ls.extend() most of
> the time myself (the behavior is a bit more obvious). I'm mostly just
> curious if the difference between |= and dict.update() would end up being
> similar as far as performance goes, with |= having a negligible advantage
> over dict.update() in most situations.
> > ___
> > Python-ideas mailing list -- python-ideas@python.org
> > To unsubscribe send an email to python-ideas-le...@python.org
> > https://mail.python.org/mailman3/lists/python-ideas.python.org/
> &g

[Python-ideas] Re: Moving PEP 584 forward (dict + and += operators)

[Kyle Stanley]

> Actually there's no need to optimize the |= operator -- for strings we
have to optimize += *because* strings are immutable, but for dicts we would
define |= as essentially an alias for .update(), just like the relationship
between += and .extend() for lists, and then no unnecessary objects would
be created.

Yeah that's why I noted that any form of optimization for the |= operator
on dicts would not be the same as += is for strings. I wasn't actually sure
of what form any potential optimization would take for the |= operator
though. What exactly was the performance question/point in reference to?
The question seemed to imply that there would be some minor performance
detriment from using |=, but it's not clear to me as to when that would be
a factor.

> ## What about performance?

> Performance is not the only objective when using Python. Switching to
inplace operators (here |=) is a generally useful and well-known technique
(it also applies to string and list concatenation, for example).

Also with lists, I recall that using the += operator is *very slightly*
faster than list.extend() in most situations:

>>> ls_plus_eq = """\
for i in range(1_000):
 ls += [x for x in range(10)]
"""
>>> ls_extend = """\
for i in range(1_000):
 ls.extend([x for x in range(10)])
"""
>>> timeit.timeit(ls_plus_eq, setup="ls = []", number=10_000)
6.563132778996078
>>> timeit.timeit(ls_extend, setup="ls = []", number=10_000)
6.695127692000824
>>> timeit.timeit("ls+=other", setup="ls = []; other=[i for i in
range(100_000)]", number=10_000)
4.400735091003298
>>> timeit.timeit("ls.extend(other)", setup="ls = []; other=[i for i in
range(100_000)]", number=10_000)
4.574331789997814
>>> timeit.timeit("ls+=other", setup="ls = []; other=[i for i in
range(100)]", number=10_000_000)
3.5332175369985634
>>> timeit.timeit("ls.extend(other)", setup="ls = []; other=[i for i in
range(100)]", number=10_000_000)
3.7756526679950184

(Python 3.8)

It seems to be a difference of only ~2-4% in most cases (~6-7% with the
last set), but I find it interesting that += is barely faster. Of course,
some of the above examples are fairly unrealistic, for most practical use
cases they're essentially the same. I tend to prefer ls.extend() most of
the time myself (the behavior is a bit more obvious). I'm mostly just
curious if the difference between |= and dict.update() would end up being
similar as far as performance goes, with |= having a negligible advantage
over dict.update() in most situations.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/243KH67NWORPWGR75D3ZWRNGKUPRWG5F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Sets for easy interning(?)

[Kyle Stanley]

> FWIW, you can do it with dict already.

> o = memo.setdefault(o, o)

I don't think this has quite the same behavior as the OP is looking for,
since dict.setdefault() will insert the key and return the default when
it's not present, instead the OP wanted to raise "KeyError if the object is
not present". In order to raise a KeyError from a missing key and have the
values be the same as the keys, one could build a dictionary like this:

```
d = {}
for item in sequence:
d[item] = item
```

or using comprehension:

```
d = {item: item for item in sequence}
```

and then

```
try:
val = d['foo']
except KeyError:
...
```

But yeah, this behavior already exists for dictionaries. Personally, I
think some_set['foo'] would likely:

1) Not make much sense for usage in sets, from a design perspective.
2) Lack practical value, as opposed to simply using a dictionary for the
same purpose. To me, this feature doesn't seem worthwhile to implement or
maintain.

There are probably other reasons to consider as well.

However, if the OP wants to personally implement this behavior for their
own subclass of sets (instead of adding it to the language), that could be
done rather easily:

```
>>> class MySet(set):
... def __getitem__(self, key):
... if key not in self:
... raise KeyError(f'{key} not present in set')
... else:
... return key
...
>>> s = MyClass({'a', 'b', 'c'})
>>> s['a']
'a'
>>> s['d']
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 4, in __getitem__
KeyError: 'd' not present in set
```

This may not have the same performance as d[key] since it has the
conditional check for membership, but it provides the same functionality.
In 90% of use cases the performance difference should be very negligible. I
don't think that I'd advocate for using the above instead of a dict, but
it's rather straightforward to implement if desired.

On Mon, Dec 2, 2019 at 7:39 PM Inada Naoki  wrote:

> FWIW, you can do it with dict already.
>
> o = memo.setdefault(o, o)
>
> On Tue, Dec 3, 2019 at 9:29 AM Soni L.  wrote:
> >
> > This is an odd request but it'd be nice if, given a set s = {"foo"},
> > s["foo"] returned the "foo" object that is actually in the set, or
> > KeyError if the object is not present.
> >
> > Even use-cases where you have different objects whose differences are
> > ignored for __eq__ and __hash__ and you want to grab the one from the
> > set ignoring their differences would benefit from this.
> > ___
> > Python-ideas mailing list -- python-ideas@python.org
> > To unsubscribe send an email to python-ideas-le...@python.org
> > https://mail.python.org/mailman3/lists/python-ideas.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/T3Z32DEMWK46EBPULYB4CVI2QF4FS3WJ/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Inada Naoki  
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/Q2EPKWPSVG55A3CKVCLJGRX6SPKKSIEE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GYCWO2YUSSVSBIJXNXWDJYH5SLZKZQ6L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Moving PEP 584 forward (dict + and += operators)

[Kyle Stanley]

> I don't particularly care about learnability or discoverability in this
case -- I don't think beginners should be taught these operators as a major
tool in their toolbox (let them use .update()), and for anyone wondering
how to do something with a dict without access to the official docs,
there's always help(dict).

I initially had some concern about the |= operator not being overly clear
to beginners, but that's a good point: it's not really made for beginners
or intended to be an "essential-to-know" feature of Python.

Although, I think the dict documentation [1] could benefit from a brief
example that demonstrates equivalent code if PEP 584 is implemented, with
one using dict.update() and the other using the operator.

> Performance is not the only objective when using Python. Switching to
inplace operators (here |=) is a generally useful and well-known technique
(it also applies to string and list concatenation, for example).

Also, I think it's worth noting that we can optimize the actual behavior of
the in-place operator under the hood, similar to what is done with
consecutive string concatenations [2]. Of course, the optimization wouldn't
be the same as it is with strings (since strings are immutable and dicts
are mutable), but it still works as an example. IMO, that's a significant
benefit of high-level languages such as Python: we can simplify the syntax
while optimizing the exact behavior in the internals.


[1]: https://docs.python.org/3/library/stdtypes.html#mapping-types-dict
[2]: A recent blog post by Paul Ganssle explains the string concat
optimization quite well in:
https://blog.ganssle.io/articles/2019/11/string-concat.html (in the
"Optimizing string concatenation" section)

On Mon, Dec 2, 2019 at 2:57 PM Guido van Rossum  wrote:

> A month ago I promised the Steering Council I'd review PEP 584 and
> promptly got distracted by other things. But here it is, at last.
>
> Some procedural things first: you (the PEP authors) should apply to the
> Steering Council for a formal review. They most likely will assign one of
> the SC members as a BDFL-Delegate who will guide the PEP to a resolution
> (either acceptance or rejection, after any number of revisions). Don't wait
> until you think the PEP is perfect! It doesn't have to be for a
> BDFL-Delegate to be assigned.
>
> Note that the SC elections have just started, and in a few weeks we'll
> have a new SC. I won't be in it (I withdrew from the ballot). I don't know
> how this will affect the chances of this PEP. (Also note that after an
> election, a SC can overturn decisions by the previous SC, so I don't mean
> this as a reason to hurry -- just something to consider.)
>
> Now to the content of PEP 584.
>
> ## Should the operators be + and +=, or | and |= ?
>
> I haven't read _all_ the posts on this topic, but I've seen quite a few,
> and I still strongly favor | and |= . My reason for this is mainly that the
> operation really does do a set union **on the keys**.
>
> Also, + is already overloaded enough (numbers, strings, sequences) and we
> don't need yet another, different meaning. The | operator should be known
> to most users from sets (which are introduced before dicts in the official
> Python tutorial).
>
> I don't particularly care about learnability or discoverability in this
> case -- I don't think beginners should be taught these operators as a major
> tool in their toolbox (let them use .update()), and for anyone wondering
> how to do something with a dict without access to the official docs,
> there's always help(dict). The natural way of discovering these would be
> through tutorials, reading of the library docs, or by reading other
> people's code, not by trying random key combinations in the REPL.
>
> ## Does it matter that the operation is lossy, and not commutative in the
> values?
>
> Centithreads have been wasted on this topic, and I really don't care. I
> **certainly** don't think that introducing a new operator or using a
> different operator (<<, really?) is needed because of the operator's
> asymmetry. A tutorial can show this in two or three lines, and anybody who
> remembers these operators exists but has forgotten how they handle
> conflicting values (and doesn't remember they correspond to.update()) can
> type one line in the REPL to figure it out:
>
> >>> {"a": 1} | {"a": 2}
> {"a": 2}
> >>>
>
> ## What about the Counter class?
>
> The Counter class is an over-engineered contraption that has way too many
> APIs to remember, and is quite inconsistent in what it tries to be. (When
> does it delete zeros? Why does it sometimes discard negative values?) We
> should not use it for guidance nor should we feel constrained by its design.
>
> ## What about performance?
>
> Performance is not the only objective when using Python. Switching to
> inplace operators (here |=) is a generally useful and well-known technique
> (it also applies to string and list concatenation, for example).
>
> ## Other objections
>
> The PEP 

[Python-ideas] Re: namedtuple for dict.items()/collections.abc.Mappings.items()

[Kyle Stanley]

 >>> for item in data.items(): item[0], item[1]
874 µs ± 21.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
>>> for key, value in data.items(): key, value
524 µs ± 4.26 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
>>> for item in items_tuple(data): item.key, item.value
5.82 ms ± 117 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Thanks for sharing the results, in particular the amount of difference
between "for item in data.items(): item[0], item[1]" and "for key in
data.items(): key, value" is a bit surprising to me. I'd have assumed
they'd be a bit closer in performance. I expected the named tuple to be
significantly slower than the other two, but not quite by that much. Good
to know.

I'm -1 on the proposal overall. It's not a bad idea, but in practice it
would likely be too much of a detriment to performance and backwards
compatibility to "dict.items()". I wouldn't be opposed to considering a
different method though, such as "dict.named_items()" or something similar
that allowed usage of "item.key" and "item.value".

On Sun, Dec 1, 2019 at 7:56 AM Wes Turner  wrote:

> My mistake, I skimmed and assumed that the question was asking for this:
>
> from collections import namedtuple
> data = {'1': 1, 'two-2': 2}
> x = namedtuple('x', data.keys())
> # ValueError: Type names and field names must be valid identifiers: '1'
>
> # But now understand that the request was for this:
>
> from collections import namedtuple
> Item = namedtuple('Item', ['key', 'value'])
>
> def items_tuple(self):
> for key, value in self.items():
> yield Item(key, value)
>
> # So that this test would pass:
>
> def test_items():
> data = {'1': 1, 'two-2': 2}
> for item in data.items():
> assert hasattr('key', item)
> assert hasattr('value', item)
> key, value = item
> assert item.key == key
> assert item.value == value
>
> # FWIW, here are rough timings:
>
> data = dict.fromkeys(range(1))
>
> def timeit(code):
> print(f">>> {code}")
> get_ipython().run_line_magic('timeit', code)
>
> timeit('for item in data.items(): item[0], item[1]')
> timeit('for key, value in data.items(): key, value')
> timeit('for item in items_tuple(data): item.key, item.value')
>
> >>> for item in data.items(): item[0], item[1]
> 874 µs ± 21.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
> >>> for key, value in data.items(): key, value
> 524 µs ± 4.26 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
> >>> for item in items_tuple(data): item.key, item.value
> 5.82 ms ± 117 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>
> On Sun, Dec 1, 2019 at 1:24 AM Andrew Barnert  wrote:
> >
> > On Nov 30, 2019, at 20:21, Wes Turner  wrote:
> > >
> > > What about keys that contain invalid characters for attribute names?
> >
> > What about them?
> >
> > > items = {'1': 1, 'two-3': 4,}
> > > x = object()
> > > x.__dict__.update(items) # dangerous
> > > x = AttrDict(**items)
> > > x.1 # error
> > > x.two-3 # error
> >
> > The message you quoted was about how in Python 2 (but not 3) you could
> destructure parameters:
> >
> > sorted({1:300, 2:4}.items(), key=lambda (key, value): value)
> >
> > The wider discussion is about how if items() were a view of namedtuples
> instead of just sequences you could do something even better:
> >
> > sorted({1:300, 2:4}.items(), key=lambda it: it.value)
> >
> > What does either of those have to do with using a dict whose keys are
> not identifiers as an attribute dictionary for an object? Neither restoring
> Python 2’s parameter destructuring nor making items namedtuples would in
> any way affect any of the code you wrote.
> >
> >
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/OA7NQRF4VTFG2DJVNJAAJ7YIQNB3IVVR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/76Y6PK74TLQFWNLE4CO3WAGBUPMQ62WL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Renaming json.load()

[Kyle Stanley]

> I happen to be teaching essential std lib modules this week, i had a
headache with how naming goes on (i was looking at how a beginner would
learn).   Poor me, what could be an intuitive learning journey has some
clogs down the road.

To some degree, that's an inevitable part of the learning process. The most
we can realistically do is try to be aware of it during design, but I
suspect there's always going to be some "more intuitive" way we could have
done it when looking back with hindsight.

On Thu, Nov 28, 2019 at 6:04 AM Abdur-Rahmaan Janhangeer <
arj.pyt...@gmail.com> wrote:

> For the future, at least
>
> I happen to be teaching essential std lib modules this week, i had a
> headache with how naming goes on (i was looking at how a beginner would
> learn).   Poor me, what could be an intuitive learning journey has some
> clogs down the road.
>
> Abdur-Rahmaan Janhangeer
> http://www.pythonmembers.club | https://github.com/Abdur-rahmaanJ
> Mauritius
>
> On Thu, 28 Nov 2019, 14:57 Steven D'Aprano,  wrote:
>
>> On Thu, Nov 28, 2019 at 02:06:08PM +0400, Abdur-Rahmaan Janhangeer wrote:
>> > It's about ambiguity. Maybe loads is short for load string which in
>> english
>> > is also a verb.
>>
>> True. It took me a long time to stop getting confused between
>> pickle.load and .loads until I stopped reading it as "loads" and started
>> reading it as "load-s(tring)".
>>
>> I agree with you that this is a sub-optimal naming convention, and we
>> would have been better if a different choice was made at the beginning.
>> But unfortunately it is a widely used naming convention:
>>
>> * pickle
>> * marshall
>> * json
>> * yaml (I think)
>> * and probably more
>>
>> and not just in Python. But I agree with the others that the pain and
>> disruption from changing it is greater than the benefit. We all just
>> have to memorise that "loads" means "load-string" and not the present
>> tense of load.
>>
>>
>> --
>> Steven
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/FVP522GAT6EMIZP36BKRULCXCJCUEHCC/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/7ATKEJ55WPT24UJKXBIKPEQUAOLJI2Y7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FJ4F77EUWA5AOAS43S2XYVFTF6R3HE54/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Calendar.year()

[Kyle Stanley]

> Replacing it would break backward compatibility. But adding a year
function as an alias for calendar wouldn’t. And you could note in > the
docs that year is preferred for new code, or even deprecate the old name
over a few versions if it seems worth it.

> What about prcal (as opposed to pryear) vs. prmonth?

+1 on the idea of adding a `calendar.year()` alias, that would make sense.
I don't think there would be a need to immediately deprecate
`calendar.calendar()` though. If we did consider deprecating it at some
point, the removal would have to be incredibly far out considering that
it's been around for quite a while and is likely one of the most widely
used functions in the calendar module.

Regarding `calendar.pryear()`, I think we should wait to see how widely
used `calendar.year()` ends up being in comparison to `calendar.calendar()`
before considering the addition of a second similar alias.

On Wed, Nov 27, 2019 at 10:02 PM Andrew Barnert via Python-ideas <
python-ideas@python.org> wrote:

> On Nov 27, 2019, at 12:39, Abdur-Rahmaan Janhangeer 
> wrote:
> >
> >
> > In the calendar module, i find it terribly unintuitive to have a
> calendar.month but not a calendar.year, the default being calendar.calendar
> >
> > Suggestion: replace calendar.calendar by calendar.year
>
> Replacing it would break backward compatibility. But adding a year
> function as an alias for calendar wouldn’t. And you could note in the docs
> that year is preferred for new code, or even deprecate the old name over a
> few versions if it seems worth it.
>
> What about prcal (as opposed to pryear) vs. prmonth?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/RFHTAKFOFOFDEWIMR2ZSVG74EII642MM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WABEGVAWCBTVM2J5FEX6DUPLUO7IQ4HT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Percent notation for array and string literals, similar to Perl, Ruby

[Kyle Stanley]

> If you can get by with a single literal list of words, write it once.
> ...if it's long enough to be annoying or becomes a maintenance burden,
use the `split()` idiom.
> ...if that's considered a "hack" or "bad form", then run it in the shell
once and copy/paste the result.
> ...if it might get mutated in a loop, copy it (words[:]). You'd be
constructing a new one anyways.
> ...if it's just too long to maintain in code, just load it from a txt
file once at runtime.

Well said, Brandt.

My personal preference is to run "...".split() in the shell and copy/paste
the output, as it takes an incredibly minimal amount of time to start up
the REPL for simple one liners. In my experience, folks often seem to
forget that the REPL (or IDLE shell) exists outside of demo examples; it's
highly useful for quick micro-scripts.

I'm not 100% opposed to the proposed functionality, but I'm against the
syntax and don't consider the lack of a shortcut to be particularly
detrimental in this case. IMO, anything that falls under the category of
being a syntactical shortcut should be highly readable and fairly obvious
as to what it's doing at a first glance. Otherwise, it adds an unnecessary
cost to the learning curve of Python (which can very rapidly accumulate if
it's not kept in check).


On Thu, Oct 24, 2019 at 12:39 PM Brandt Bucher 
wrote:

> This is getting a little ridiculous.
>
> If you can get by with a single literal list of words, write it once.
> ...if it's long enough to be annoying or becomes a maintenance burden, use
> the `split()` idiom.
> ...if that's considered a "hack" or "bad form", then run it in the shell
> once and copy/paste the result.
> ...if it might get mutated in a loop, copy it (words[:]). You'd be
> constructing a new one anyways.
> ...if it's just too long to maintain in code, just load it from a txt file
> once at runtime.
>
> These are simple engineering solutions to simple engineering problems. As
> a bonus, they don't require implementation maintainers to redesign their
> tokenizers/parsers or build a brand-new preprocesser.
>
> Python-ideas truly is a unique, wonderful, exhausting place. Back to the
> "plus/pipe" thread (formerly known as the "PEP 584" thread)...
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/THBBTIQ34GYZSBW6PQXKKO3L3UADYBXB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZRVXFZTTKBN4OUL4NLNNCLYM6GFUTI2J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Consider blocking heterogeneous chained comparison

[Kyle Stanley]

> Please recall that chained comparisons such are
> >>> A < B < C
> is to produce simple code when B is not an identifier, but an expression.
For example
> >>> low < len(x) < high
> or
> >>> inner ** 2 < x**2 + y**2 + z**2 < outer**2
> which determines if (x, y, z) is in a spherical shell.

Even in those cases, each individual expression could be stored in a
variable before doing the comparison:

>>> a, b, c = inner ** 2, x**2 + y**2 + z**2, outer**2
>>> a < b and b < c

But then again I don't find the rich boolean comparison operators (<, <=,
>, =>, ==, !=) to be an issue when they're chained together on standard
numeric types such as the above example, particularly when it's the same
operator. IMO, it only becomes difficult to read when the operators start
to become "mixed up"; more so when the objects are not the builtin types
and have overridden operators:

>>> a is b < c in d

Not only do you have to sort out what is happening logically with the
operators, you also have to figure out what those operators actually do for
that particular object. Whether or not that is "readable" might be somewhat
subjective though.

But I think we're probably veering a bit off topic here, the main focus of
this thread is on whether or not "A in B < C" should be permissible syntax.
Although I'm not a fan of the syntax, I don't think it would be worthwhile
for it to raise a SyntaxError for the reasons I mentioned previously. This
seems to be something to either enforce in a linter or manually revise in a
code review process, rather than through the language itself.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TGS4TRV54E2EB3OQJ5W7FMQ6GHPXJNI7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Consider blocking heterogeneous chained comparison

[Kyle Stanley]

> Some people find this legal syntax confusing:

> A in B < C

I agree, that syntax definitely seems counter-intuitive. If it came up in a
code review, I would request a revision to "A in B and B < C".

> I feel like most code reviews would not allow this kind of code to be
checked in, at least without a comment explaining what the code does.  What
are the motivating reasons for allowing this syntax?  Is there any
idiomatic code that uses it?

I can't personally imagine any practical use case for the syntax that
couldn't be expressed differently in a way that's far more clear to
readers. But, I don't think this is so much of a case of us choosing to
*allow* the syntax, this seems to be more of a case where we don't
explicitly *prevent* the syntax (through raising a SyntaxError).

>From my experience, SyntaxError is raised primarily from clear mistakes
that are easy to make (particularly for those who are new to the language
or accidental typos) rather than simply syntax that is obscure,
impractical, and/or hard to understand. It's realistically impossible to
isolate every case of highly counter-intuitive usage of Python syntax, not
to mention the resources it takes to implement a SyntaxError for each of
them.

To me, it seems that the usage of "A in B  C" is
something that's not easy to do by accident, I can't imagine what someone
would be intending to practically accomplish and mistakenly end up with
that syntax, not knowing it could be expressed differently. A few important
questions to consider:

1) Is this syntax misused enough to justify implementing a SyntaxError?

2) Would raising a SyntaxError help to point those who misuse it in the
right direction?

3) Is it worth the resources it would take to implement the SyntaxError?

4) Does implementing the SyntaxError provide enough positive benefit to
justify breaking backwards compatibility? Even if it's very rarely used and
could be expressed differently, this still applies.

> Also, in my opinion, the existence is a deterrent to Python and takes
away from Python's beauty as a language.

I would have to disagree, every language has its obscure corner cases and
confusing ways to write expressions. I don't think the number of ways that
users can misuse the language in an obscure manner to write confusing code
should at all detract from it's beauty. If it were easy to misuse the
language and *accidentally* write confusing code that would be one thing,
but I don't think "A in B < C" is an example of that scenario.

As the others mentioned, I think the best solution to dealing with this
type of syntax would be with a linter rather than directly preventing its
usage.

(resent because the CC address was incorrect)

On Sun, Nov 17, 2019 at 6:01 AM Kyle Stanley  wrote:

> > Some people find this legal syntax confusing:
>
> > A in B < C
>
> I agree, that syntax definitely seems counter-intuitive. If it came up in
> a code review, I would request a revision to "A in B and B < C".
>
> > I feel like most code reviews would not allow this kind of code to be
> checked in, at least without a comment explaining what the code does.  What
> are the motivating reasons for allowing this syntax?  Is there any
> idiomatic code that uses it?
>
> I can't personally imagine any practical use case for the syntax that
> couldn't be expressed differently in a way that's far more clear to
> readers. But, I don't think this is so much of a case of us choosing to
> *allow* the syntax, this seems to be more of a case where we don't
> explicitly *prevent* the syntax (through raising a SyntaxError).
>
> From my experience, SyntaxError is raised primarily from clear mistakes
> that are easy to make (particularly for those who are new to the language
> or accidental typos) rather than simply syntax that is obscure,
> impractical, and/or hard to understand. It's realistically impossible to
> isolate every case of highly counter-intuitive usage of Python syntax, not
> to mention the resources it takes to implement a SyntaxError for each of
> them.
>
> To me, it seems that the usage of "A in B  C" is
> something that's not easy to do by accident, I can't imagine what someone
> would be intending to practically accomplish and mistakenly end up with
> that syntax, not knowing it could be expressed differently. A few important
> questions to consider:
>
> 1) Is this syntax misused enough to justify implementing a SyntaxError?
>
> 2) Would raising a SyntaxError help to point those who misuse it in the
> right direction?
>
> 3) Is it worth the resources it would take to implement the SyntaxError?
>
> 4) Does implementing the SyntaxError provide enough positive benefit to
> justify breaking backwards compatibility? Even if it's very rarely used and
> could be expressed differentl

[Python-ideas] Re: Ability to set precedence of classdict (returned from __prepare__) for a metaclass type

[Kyle Stanley]

> That said, though, I'm not sure that this was even what you were asking
about :)

It would be significantly more helpful if the OP had an example for the
specific case where they wanted contextually defined variables to take
priority over the classdict, and explained _why_ this behavior might be
desirable:

> > There is at least one case in which I would like to do the reverse, and
have the classdict be secondary to (masked by) any variables defined in the
context surrounding the execution of the class body.

> > I'm not sure what the best way to deal with this would be, but my first
thought is to maybe have a property that can be set on the metaclass type
such as `metacls.relictant_classdict = True`.

Describing the specific use case in detail and providing an example would
also help with determining the practical value of adding the described
property. I'm not certain that a property would be the best way to do it (a
new optional parameter for the __prepare__ attribute might be an
alternative). But before that, we have to determine whether or not the
feature would have enough practical value to justify implementing in any
capacity.

This is of course assuming that what the OP wants doesn't already exist in
some other form (if Chris's example isn't what the OP is looking for) or
that it's even reasonably possible to implement in the first place.

On Sat, Oct 19, 2019 at 11:39 PM Chris Angelico  wrote:

> On Sun, Oct 20, 2019 at 1:58 PM Steve Jorgensen 
> wrote:
> >
> > When using a custom classdict to implement a DSL or use in the body of a
> class definition, from what I can tell by experiment, the classdict takes
> priority, and the surrounding context is only referenced if trying to get
> an item from the classdict raises `KeyError`.
> >
> > There is at least one case in which I would like to do the reverse, and
> have the classdict be secondary to (masked by) any variables defined in the
> context surrounding the execution of the class body.
> >
> > I have been able to at least unmask builtins by having the classdict
> object first try to get a result from `__builtins__` and then fall back to
> itself. Also, in the actual class body, declaring a variable as `global`
> seems to be a workaround for global variables, but that has to be done
> explicitly in the body of any subclass that needs it. Also, that does not
> handle non-globals in its surrounding context.
> >
> > I'm not sure what the best way to deal with this would be, but my first
> thought is to maybe have a property that can be set on the metaclass type
> such as `metacls.relictant_classdict = True`.
> >
>
> Not entirely sure I follow here; are you talking about unadorned names
> during the execution of the class body itself, or the way attribute
> lookup is done on the resulting object? If you want to be able to mess
> with attribute lookup even when the attribute does exist,
> __getattribute__ is your friend:
>
> https://docs.python.org/3/reference/datamodel.html#object.__getattribute__
>
> For changing the behaviour of class body execution, it's a bit harder.
> Do you actually need the class members to be available as unadorned
> names, but such that the surrounding context takes precedence? That
> could be awkward, since there's no way (as far as I know) to get a
> "hook" whenever NameError would have been raised. But you can return
> any dict-like object from the metaclass's __prepare__ method, and if
> its __getitem__ and __setitem__ are disconnected, you could get the
> effect you want. Here's one possibility that gets kinda close:
>
> class BlackBox(dict):
> def __getitem__(self, item):
> if item == "v": return self
> raise KeyError
> __getattr__ = dict.__getitem__
>
> class Meta(type):
> @classmethod
> def __prepare__(meta, cls, bases):
> return BlackBox()
>
> var = "global"
> class Demo(metaclass=Meta):
> var = "local"
> print("var =", var)
> print("v.var =", v.var)
>
> Accessing local names can be done with the "v." prefix, global names
> are directly accessible, but missing globals won't be looked up
> locally.
>
> That said, though, I'm not sure that this was even what you were asking
> about :)
>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/64VR434UDBIEJD3JYSLS3V7BPDTT36U3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/BJTETDZVSPN3HDKNYO47GENO6SB56HSS/
Code of 

[Python-ideas] Re: Add collections.abc.Struct as virtual base class of namedtuple classes and dataclass-decorated classes.

[Kyle Stanley]

> Maybe borrowing a word from a different language—record or datatype or
something—would avoid that problem?

Hmm, I don't think "record" would work. The terms "record", "struct", and
"tuple" are far too synonymous with one another (
https://en.wikipedia.org/wiki/Record_(computer_science)). This term would
have to be something that would apply to namedtuples (or similarly
structured datatype) without applying to tuples as well.

Perhaps something like "associative container" might be more viable? This
would apply to any type that could be directly converted to or represented
as a dictionary.

On Wed, Oct 16, 2019 at 8:18 PM Andrew Barnert via Python-ideas <
python-ideas@python.org> wrote:

> On Oct 16, 2019, at 13:48, Steven D'Aprano  wrote:
> >
> > I don't think that "structural" is the word you are looking for. Lists
> > and tuples and sets are all structural. That's why they're called "data
> > structures".
>
> I think what he’s looking for is the adjective that goes with the
> quasi-word “struct” as used in C, and in Python’s ctypes and C API. The
> type name “Struct” isn’t bad here, but the adjective “structural” has to
> many other connotations.
>
> Maybe borrowing a word from a different language—record or datatype or
> something—would avoid that problem?
>
> But I think the big problem here is the one you identified. We can make a
> list of things that are “like C structs” (which should probably include
> structseq builtin/extension types, ctypes.Struct, third-party stuff like
> attrs and maybe numpy structured dtypes, in addition to namedtuple and
> dataclass). Maybe we could even come up with a way to either
> programmatically detect them or get them all registered. But what would you
> do with it?
>
> It might be useful for reflection if there were a way to enumerate all of
> the “struct fields” that worked for all “struct types”, but there isn’t.
>
> Also, while all of these things _allow_ you to define a plain-old-data
> struct, none of them _require_ it. You can have a dataclass, namedtuple
> subclass, structseq, ctypes.Struct, etc. that has all kinds of fancy
> behavior, including not just methods, but @property and weird __new__
> overloads and even custom __getattribute__. And of course you can just as
> easily define a regular old class and then use it as a plain-old-data
> struct. So, would this even be useful as a hint, much less a guarantee, of
> anything?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/SCOXP4BYJ4HO4I74E7WPMRCJ5TLDUD4B/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3C7IJ3T3LPCGKZKYJTGO5SKH6FRKMY62/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add Subscriptable ABC

[Kyle Stanley]

> This supports my point that this ought to be handled once, correctly, in
the standard library, like the other ABCs, instead of leaving it up to
people like me to get it wrong.

It would be pretty reasonable to assume that most users would get
this incorrectly. My first assumption to this problem would've been
something like ``hasattr(T, "__getitem__")``, but based on the current
comments, it seems like that would be inadequate for this
scenario. I'm not aware of any solution that would be remotely
intuitive to users.

> (2) If not, is there any reason why we shouldn't add Subscriptable
to the collection.abc module? I think I have the implementation:

Another important question might be "would users utilize the
addition of Subscriptable in collections.abc enough for it to justify
its addition to the standard library?". I think it looks interesting,
but I think this should be implemented as a PyPI package (if
something similar hasn't been already) so that we can
evaluate it's usage rate before implementing it.

Raymond also brought up a strong point to consider in
https://bugs.python.org/issue25988:

> The OP has a sense that Mapping and Sequence are "too heavy"
but I think the reality that useful classes almost never use
__getitem__ in isolation; rather, it is part of a small constellation
of methods that are typically used together.


On Fri, Sep 27, 2019 at 7:21 PM Steven D'Aprano  wrote:

> On Fri, Sep 27, 2019 at 12:03:52PM -0700, Andrew Barnert via Python-ideas
> wrote:
> > On Sep 27, 2019, at 10:15, MRAB  wrote:
> > >
> > >> On 2019-09-27 17:05, Steven D'Aprano wrote:
> > >>
> > >> But doing it correctly is too painful:
> > >> if any(getattr(T, '__getitem__', None) is not None for T in
> type(obj).mro())
> > > [snip]
> > >
> > > Is there are reason why you're using 'getattr' instead of 'hasattr'?
> >
> > There is a difference here: if some class defines __getitem__ = None,
> > blocks it from looking like a subscriptable type.
>
> I wish I had been clever enough to have thought of that, but I didn't. I
> used getattr because I completely forgot about the existence of hasattr.
>
> And I walked the MRO because oops.
>
> On the other hand, the collections.abc walks the MRO explicitly checking
> for None, so perhaps I might have accidentally been less wrong than had
> I just tried hasattr(type(obj)). See the _check_methods function here:
>
> https://github.com/python/cpython/blob/3.7/Lib/_collections_abc.py
>
> This supports my point that this ought to be handled once, correctly, in
> the standard library, like the other ABCs, instead of leaving it up to
> people like me to get it wrong.
>
> > That idiom is used for other things like __hash__, even if it’s not
> > used 100% consistently for all protocols.
>
> If you look at the 3.7 collections.abc, _check_methods is now
> used consistently by all of the __subclasshook__ methods.
>
>
> --
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/7TSCXL4JQGMVWJVT22O264W3GU7UPTMY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RZMYE6OBHNL4YEYJULJ6XJAASOF6MIFW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Error handling toggle / levels

[Kyle Stanley]

It looks like Serhiy answered your main question, but I wanted to mention
that rather than simply printing the exception, you could also print the
full traceback using the "traceback" module:

>>> import traceback
>>> try:
... 1/0
... except Exception as e:
... traceback.print_exc()
...
Traceback (most recent call last):
  File "", line 2, in 
ZeroDivisionError: division by zero

Whereas with simply using `print(e)` you only get:

division by zero

The former is far more useful and gives you all of the information from
the last exception without raising it. I'd recommend looking further into
the traceback module if you'd like more customization, but using
traceback.print_exc() is a substantial improvement by itself.

On Thu, Sep 26, 2019 at 8:40 AM  wrote:

> Hello,
>
> So when coding, at deferent stages we need different levels of error
> handling. For example at stage of finding the concept, initial
> implementation, alpha, releasing testing, etc.
>
> By some bits of code we can simulate enable/disable error handling.
>
> But it would be very helpful to have such thing built in the language with
> some more pythonic salt .
>
>
> For example,
> Assume this behavior..
>
> 
> SetExceptionLevel(50)
> try:
> x = 1 / 0
> except(0) Exception as e:
> print(e)
> except (40) ZeroDivisionError as e1:
> x = math.inf
> except (40) ValueError as e2:
> x = None
> except(60) Exception as e3:
> raise e3
>
> 
>
> i.e. one exception can have more than one handler and each handler has a
> level to enable it,
> so now for this example the first 2 exception will run only,
> If we want the last one to run also we need to change the first line to
> "SetExceptionLevel(70)" for example.
>
> The third one " ValueError " will never be triggered.
>
> And if no exception met the requirements "level and exception type" then
> the error will be raised out.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/KHFQ4D2KQFZKHNUGV6SM27U3CI7EOXWJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DPRNDQGI45VSCU5M2LLJPPXTFT4INKVZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Simpler syntax for async generators

[Kyle Stanley]

Steven D'Aprano wrote:
> The interpreter should infer that a plain `def` function is actually
an `async def` from the presence of `async yield`, similar to the way
the interpreter infers that a plain `def` function is actually a
generator from the presence of a `yield`.

I would have to strongly disagree with this part of the author's proposal
especially. The whole purpose of the `async def` declaration was to help
ensure that a normal subroutine function was not mistakenly used as a
coroutine. Allowing the interpreter to infer that a regularly declared
function is actually a subroutine would ultimately go against this
intention by making the distinction between subroutines and coroutines far
less explicit.

Steven D'Aprano wrote:
> What if you're not using asyncio, but some other framework?

This point is also especially important. The "async/await" syntax was meant
to essentially be its own separate API of sorts, to allow for the usage of
entirely distinct async frameworks such as curio. Asyncio is dependent on
the "async/await" syntax, but not the other way around. I'm not certain
that I understand how these changes could be applied without causing
compatibility issues.

On Sun, Sep 15, 2019 at 8:52 PM Steven D'Aprano  wrote:

> On Mon, Sep 16, 2019 at 11:36:45AM +1200, Greg Ewing wrote:
> > George Fischhof wrote:
> > >With this syntax the use / programmer do not have to write async def,
> > >nor await, because after the yield the await should come automatically.
> >
> > I don't see how you can eliminate the await in your example generator.
> > Can you show us how it would look in its entirety under your proposal?
>
> I think the proposal is for two things:
>
> 1. `async yield i` to be syntactic sugar for `yield i; await
> asyncio.sleep(delay)`.
>
> 2. The interpreter should infer that a plain `def` function is actually
> an `async def` from the presence of `async yield`, similar to the way
> the interpreter infers that a plain `def` function is actually a
> generator from the presence of a `yield`.
>
>
> # Original
> async def ticker(delay, to):
> """Yield numbers from 0 to to every delay seconds."""
> for i in range(to):
> yield i
> await asyncio.sleep(delay)
>
>
> # Re-written
> def ticker(delay, to):
> """Yield numbers from 0 to to every delay seconds."""
> for i in range(to):
> async yield i
>
>
> So you save one line and perhaps 25-30 key presses when writing the
> function, at the cost of making it much less clear what's going on when
> reading the function.
>
> How does it known how long to sleep?
>
> What if you're not using asyncio, but some other framework?
>
>
>
> --
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/KKAAH3S4LM52NL5572LAY5QMHYUUUN7T/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TZKXEGXZUQJL57IWKMKVQUEADTQNURXJ/
Code of Conduct: http://python.org/psf/codeofconduct/