[Python-Dev] Re: Pass the Python thread state to internal C functions

[Antoine Pitrou]

On Fri, 15 Nov 2019 14:21:53 -0800
Nathaniel Smith  wrote:
> As you know, I'm skeptical that PEP 554 will produce benefits that are
> worth the effort, but let's assume for the moment that it is, and
> we're all 100% committed to moving all globals into the threadstate.
> Even given that, the motivation for this change seems a bit unclear to
> me.
> 
> I guess the possible goals are:
> 
> - Get rid of the "ambient" threadstate entirely
> - Make accessing the threadstate faster
> 
> For the first goal, I don't think this is possible, or desirable.
> Obviously if we remove the GIL somehow then at a minimum we'll need to
> make the global threadstate a thread-local. But I think we'll always
> have to keep it around as a thread-local, at least, because there are
> situations where you simply cannot pass in the threadstate as an
> argument. One example comes up when doing FFI: there are C libraries
> that take callbacks, and will run them later in some arbitrary thread.
> When wrapping these in Python, we need a way to bundle up a Python
> function into a C function that can be called from any thread. So,
> ctypes and cffi and cython all have ways to do this bundling, and they
> all start with some delicate dance to figure out whether or not the
> current thread holds the GIL, acquiring the GIL if not, then checking
> whether or not this thread has a Python threadstate assigned, creating
> it if not, etc. This is completely dependent on having the threadstate
> available in ambient context. If threadstates were always passed as
> arguments, then it would become impossible to wrap these C libraries.

Most well-designed C libraries let you pass an additional "void*"
parameter for user callbacks to be called with.  A couple of them
don't, unfortunately (OpenSSL perhaps?  I don't remember).

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PUUP6NN6U6L7XTVYJQGPUW6LT5P6Y253/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Pass the Python thread state to internal C functions

[Victor Stinner]

Le sam. 16 nov. 2019 à 20:55, Neil Schemenauer
 a écrit :
>  If you use threadstate often,
> passing it explicitly (which likely uses a CPU register) could be a
> win.  If you use it rarely, that CPU register would be better
> utilized for passing function arguments you actually use.

Currently, I would say that it's used "rarely". But. If we want to
implement subinterpreters, we have to use way more often. Since each
interpreter must have its isolated namespace, I expect that even 1+1
should use tstate to get the 2 "singleton" from its private namespace,
rather than using a "global" singleton. Basically, all builtin types
and all builtin modules should be modified to have one namespace per
interpreter.

For C extensions, it's an old project to have a "state" passed to
module functions, and so be able to have 2 separated instances of the
same C extension, rather than having a single global namespace.
Examples:

https://www.python.org/dev/peps/pep-0489/
https://www.python.org/dev/peps/pep-0573/

I would like to implement subinterpreters. IMHO the project is
feasible and if it works, it would make Python more competitive with
other programming languages!

IMHO fixing the C API (or write a new one) and subinterpreters are the
only two most feasible and most realistic projects to optimize CPython
right now.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SZ7TP3AO3L4MT7RMZ53BIVXX6IZIVGDR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: thes mapping data type and thescfg cfg file module - review and suggestion request

[Tal Einat]

Hi Dave,

Thesaurus looks interesting and it is obvious that you've put a lot of
effort into it!

This list is for the discussion of the development *of* Python itself,
however, rather than development *with* Python, so it's not an appropriate
place for such posts.

I suggest you post this on python-list and/or python-announce, to get this
in front of a wider audience.

- Tal Einat

On Mon, Nov 18, 2019 at 7:17 AM Dave Cinege  wrote:

> If you are not aware:
>
>   - Thesaurus is a mapping data type with recursive keypath map
> and attribute aliasing. It is a subclass of dict() and is mostly
> compatible as a general use dictionary replacement.
>
>   - ThesaurusExtended is a subclass of Thesaurus providing additional
> usability methods such as recursive key and value searching.
>
>   - ThesaurusCfg is a subclass of ThesaurusExtended providing a nested
> key configuration file parser and per key data coercion methods.
>
> The README.rsl will give a better idea:
>  https://git.cinege.com/thesaurus/
>
>
> After 7 years I might have reached the point of 'interesting' with
> my Thesaurus and ThesaurusCfg modules.
>
> To anyone that is overly bored, I'd appreciate your terse review and
> comments on how mundane and worthless they still actually are. :-)
>
> I'm primarily interested in suggestions to what I've done conceptually
> here. While I'm not completely ashamed of the state of the Thesaurus
> code, ThesaurusCfg is not far beyond the original few hours I slapped it
> together one day in frustration.
>
> After considering suggestions I intend to make changes towards a formal
> release.
>
> Thanks in advance,
>
> Dave
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CTXKJUTTJKOS47T6XI2O6WU7EYVEQQ3N/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HZSR7QHEHW45H4K4KA3V3U6V4AN3NEV2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: thes mapping data type and thescfg cfg file module - review and suggestion request

[Tal Einat]

On Mon, Nov 18, 2019 at 5:30 PM Dave Cinege  wrote:

> Hello Tal,
>
> Yes I understand that. I posted this here because it's been suggested by
> those less in the know that Thesaurus and ThesaurusCfg (or some of the
> the concepts contained in them) might have a place in the future mainline.
>

Ah, that's something else then. I didn't understand that from your original
post. Thanks for clarifying!


> Remember that Thesaurus is a data type and ThesaurusCfg is a alternative
> to configparser. Additionally I could bring this full circle to
> something that integrates with or provides an alternative to argparse.
> (providing an integrated cfg file and command line argument solution
> which currently does not exist.)
>

These days, thanks to pip and PyPI, anyone can publish libraries and it is
easy for developers to find and use them if they like. There is no longer a
need to add things to the stdlib just to make them widely available.

On the other hand, bringing something into the stdlib means undertaking an
obligation by the Python core devs to maintain it for a very long period of
time. It also ties their releases to the Python release cycle, which is not
appropriate for many libraries.

Therefore, our current policy for additions to the standard library is to
add as few things as possible. We've recently been adding to the standard
library only in special cases, such as wanting to use something in the of
the standard library.

With Thesaurus I can't currently think of a reason for such special
consideration.


> I hope with this in mind this thread is considered on-topic and I'd
> highly value any feedback from the 'serious people' on this list if they
> should take a look.
>

We are, after all, a group of people developing a language named after
Monty Python. You will find very few 'serious people' here! ;)

- Tal Einat
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CAMLWWX3A6KGCK7KEIXQAENIBO27VOOE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: thes mapping data type and thescfg cfg file module - review and suggestion request

[Dave Cinege]

Hello Tal,

Yes I understand that. I posted this here because it's been suggested by 
those less in the know that Thesaurus and ThesaurusCfg (or some of the 
the concepts contained in them) might have a place in the future mainline.


Remember that Thesaurus is a data type and ThesaurusCfg is a alternative 
to configparser. Additionally I could bring this full circle to 
something that integrates with or provides an alternative to argparse. 
(providing an integrated cfg file and command line argument solution 
which currently does not exist.)


I hope with this in mind this thread is considered on-topic and I'd 
highly value any feedback from the 'serious people' on this list if they 
should take a look.


Dave



On 2019/11/18 08:51, Tal Einat wrote:

Hi Dave,

Thesaurus looks interesting and it is obvious that you've put a lot of 
effort into it!


This list is for the discussion of the development *of* Python itself, 
however, rather than development *with* Python, so it's not an 
appropriate place for such posts.


I suggest you post this on python-list and/or python-announce, to get 
this in front of a wider audience.


- Tal Einat

On Mon, Nov 18, 2019 at 7:17 AM Dave Cinege > wrote:


If you are not aware:

   - Thesaurus is a mapping data type with recursive keypath map
and attribute aliasing. It is a subclass of dict() and is mostly
compatible as a general use dictionary replacement.

   - ThesaurusExtended is a subclass of Thesaurus providing additional
usability methods such as recursive key and value searching.

   - ThesaurusCfg is a subclass of ThesaurusExtended providing a nested
key configuration file parser and per key data coercion methods.

The README.rsl will give a better idea:
https://git.cinege.com/thesaurus/


After 7 years I might have reached the point of 'interesting' with
my Thesaurus and ThesaurusCfg modules.

To anyone that is overly bored, I'd appreciate your terse review and
comments on how mundane and worthless they still actually are. :-)

I'm primarily interested in suggestions to what I've done conceptually
here. While I'm not completely ashamed of the state of the Thesaurus
code, ThesaurusCfg is not far beyond the original few hours I
slapped it
together one day in frustration.

After considering suggestions I intend to make changes towards a formal
release.

Thanks in advance,

Dave
___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/CTXKJUTTJKOS47T6XI2O6WU7EYVEQQ3N/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VXA7EZYVADF3I3OV7WADMMD5PO4YB2OU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Pass the Python thread state to internal C functions

[Random832]

On Mon, Nov 18, 2019, at 05:26, Antoine Pitrou wrote:
> > For the first goal, I don't think this is possible, or desirable.
> > Obviously if we remove the GIL somehow then at a minimum we'll need to
> > make the global threadstate a thread-local. But I think we'll always
> > have to keep it around as a thread-local, at least, because there are
> > situations where you simply cannot pass in the threadstate as an
> > argument. One example comes up when doing FFI: there are C libraries
> > that take callbacks, and will run them later in some arbitrary thread.
> > When wrapping these in Python, we need a way to bundle up a Python
> > function into a C function that can be called from any thread. So,
> > ctypes and cffi and cython all have ways to do this bundling, and they
> > all start with some delicate dance to figure out whether or not the
> > current thread holds the GIL, acquiring the GIL if not, then checking
> > whether or not this thread has a Python threadstate assigned, creating
> > it if not, etc. This is completely dependent on having the threadstate
> > available in ambient context. If threadstates were always passed as
> > arguments, then it would become impossible to wrap these C libraries.
> 
> Most well-designed C libraries let you pass an additional "void*"
> parameter for user callbacks to be called with.  A couple of them
> don't, unfortunately (OpenSSL perhaps?  I don't remember).

I think you've missed the fact that the C library runs the callback on an 
arbitrary thread. The threadstate associated with the thread that made the 
original call is therefore *not the one you want*; you want a threadstate 
associated with the thread the callback is run on.

Alternately, if a thread state is not in any sense associated with a thread 
(would these situations then mean you simply always create a brand-new 
interpreter state?), maybe it shouldn't be called a thread state at all.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NSRJVU7PZOVBWWYS3R5QLRRYP2N6NKTY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Pass the Python thread state to internal C functions

[Antoine Pitrou]

On Mon, 18 Nov 2019 12:39:00 -0500
Random832  wrote:
> On Mon, Nov 18, 2019, at 05:26, Antoine Pitrou wrote:
> > > For the first goal, I don't think this is possible, or desirable.
> > > Obviously if we remove the GIL somehow then at a minimum we'll need to
> > > make the global threadstate a thread-local. But I think we'll always
> > > have to keep it around as a thread-local, at least, because there are
> > > situations where you simply cannot pass in the threadstate as an
> > > argument. One example comes up when doing FFI: there are C libraries
> > > that take callbacks, and will run them later in some arbitrary thread.
> > > When wrapping these in Python, we need a way to bundle up a Python
> > > function into a C function that can be called from any thread. So,
> > > ctypes and cffi and cython all have ways to do this bundling, and they
> > > all start with some delicate dance to figure out whether or not the
> > > current thread holds the GIL, acquiring the GIL if not, then checking
> > > whether or not this thread has a Python threadstate assigned, creating
> > > it if not, etc. This is completely dependent on having the threadstate
> > > available in ambient context. If threadstates were always passed as
> > > arguments, then it would become impossible to wrap these C libraries.  
> > 
> > Most well-designed C libraries let you pass an additional "void*"
> > parameter for user callbacks to be called with.  A couple of them
> > don't, unfortunately (OpenSSL perhaps?  I don't remember).  
> 
> I think you've missed the fact that the C library runs the callback on an 
> arbitrary thread. The threadstate associated with the thread that made the 
> original call is therefore *not the one you want*; you want a threadstate 
> associated with the thread the callback is run on.

Ah, right, I had overlooked that mention.  This does complicate things
a bit.  In that case you would want to pass the interpreter state and
then use this particular interpreter's mapping of OS thread to
threadstate.

(assuming that per-interpreter mapping exists, which is another
question; but it will have to exist at some point for PEP 554)

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S5RDSBW7NZ3RAIHCFHQYQW4X6JD4N3G5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: thes mapping data type and thescfg cfg file module - review and suggestion request

[Guido van Rossum]

It won't be easy to add this to the stdlib. One thing that bugs me in
particular is that you can access key via '.attr' notation (the example
shows t['a']['b']['c']['d'] as equivalent to t.a.b.c.d). This feels
problematic: What should happen if the key happens to be the name of a
method (e.g. .keys or .update)? The choices you have are either to mask the
method or to mask the key. Neither solution seems ideal. JavaScript has
this equivalence and it makes me very uncomfortable.

I'm not denying that Thesaurus[Cfg] looks useful. But, like Tal, I must
stress that that's not enough to consider inclusion in the stdlib.

On Mon, Nov 18, 2019 at 4:25 PM Dave Cinege  wrote:

> Hello Tal,
>
> Yes I understand that. I posted this here because it's been suggested by
> those less in the know that Thesaurus and ThesaurusCfg (or some of the
> the concepts contained in them) might have a place in the future mainline.
>
> Remember that Thesaurus is a data type and ThesaurusCfg is a alternative
> to configparser. Additionally I could bring this full circle to
> something that integrates with or provides an alternative to argparse.
> (providing an integrated cfg file and command line argument solution
> which currently does not exist.)
>
> I hope with this in mind this thread is considered on-topic and I'd
> highly value any feedback from the 'serious people' on this list if they
> should take a look.
>
> Dave
>
>
>
> On 2019/11/18 08:51, Tal Einat wrote:
> > Hi Dave,
> >
> > Thesaurus looks interesting and it is obvious that you've put a lot of
> > effort into it!
> >
> > This list is for the discussion of the development *of* Python itself,
> > however, rather than development *with* Python, so it's not an
> > appropriate place for such posts.
> >
> > I suggest you post this on python-list and/or python-announce, to get
> > this in front of a wider audience.
> >
> > - Tal Einat
> >
> > On Mon, Nov 18, 2019 at 7:17 AM Dave Cinege  > > wrote:
> >
> > If you are not aware:
> >
> >- Thesaurus is a mapping data type with recursive keypath map
> > and attribute aliasing. It is a subclass of dict() and is mostly
> > compatible as a general use dictionary replacement.
> >
> >- ThesaurusExtended is a subclass of Thesaurus providing
> additional
> > usability methods such as recursive key and value searching.
> >
> >- ThesaurusCfg is a subclass of ThesaurusExtended providing a
> nested
> > key configuration file parser and per key data coercion methods.
> >
> > The README.rsl will give a better idea:
> > https://git.cinege.com/thesaurus/
> >
> >
> > After 7 years I might have reached the point of 'interesting' with
> > my Thesaurus and ThesaurusCfg modules.
> >
> > To anyone that is overly bored, I'd appreciate your terse review and
> > comments on how mundane and worthless they still actually are. :-)
> >
> > I'm primarily interested in suggestions to what I've done
> conceptually
> > here. While I'm not completely ashamed of the state of the Thesaurus
> > code, ThesaurusCfg is not far beyond the original few hours I
> > slapped it
> > together one day in frustration.
> >
> > After considering suggestions I intend to make changes towards a
> formal
> > release.
> >
> > Thanks in advance,
> >
> > Dave
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > 
> > To unsubscribe send an email to python-dev-le...@python.org
> > 
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> >
> https://mail.python.org/archives/list/python-dev@python.org/message/CTXKJUTTJKOS47T6XI2O6WU7EYVEQQ3N/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/VXA7EZYVADF3I3OV7WADMMD5PO4YB2OU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UHO7UEJUKFXFYFOBIEAX6AI4DOSGYARQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Dangerous default value for reuse_address in asyncio loop.create_datagram_endpoint()

[vaizki]

When creating UDP servers with asyncio's create_datagram_endpoint(), the 
default value for reuse_address = True, resulting in a dangerous (and currently 
incorrectly documented) situation. I have proposed changing the default value 
but understandably such a change for a core library function parameter is not 
to be taken lightly. Thus I put this up for discussion on the list.

As background, when creating TCP servers on UNIX-like systems, it is almost 
boilerplate to set SO_REUSEADDR for all server sockets to make sure that a 
restarting server can immediately bind the socket again. Without the 
SO_REUSEADDR sockopt, the kernel will hold the addr:port in a TIME_WAIT state 
for a while, preventing reuse. Thus, when creating TCP servers with 
loop.create_server(), the parameter reuse_address has a very reasonable default 
value of True.

However things are very different in UDP-land. The kernel does not hold UDP 
server ports in a waiting state so the SO_REUSEADDR sockopt was repurposed in 
Linux (and *BSD afaik) to allow multiple processes to bind the SAME addr:port 
for a UDP server. The kernel will then feed incoming UDP packets to all such 
processes in a semi-fair-roundrobin manner. This is very useful in some 
scenarios, for example I've used it myself in C++ projects to allow UDP servers 
to be scaled easily and rolling upgrades to be implemented without separate 
load-balancing. But for this to be the default behaviour is quite dangerous.

I discovered this default behaviour accidentally by having 2 separate Python 
programs (both doing SIP over UDP) accidentally configured to use the same UDP 
port. The result was that my 2 processes were indeed "sharing the load" - 
neither of them threw an exception at startup about the port being already in 
use and both started getting ~half of the incoming packets. So off to the docs 
I went and discovered that the documentation for create_datagram_endpoint() 
does not mention this behaviour at all, instead it mistakenly refers to the TCP 
protocol use of SO_REUSEADDR: "reuse_address tells the kernel to reuse a local 
socket in TIME_WAIT state, without waiting for its natural timeout to expire. 
If not specified will automatically be set to True on Unix."
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.create_datagram_endpoint

What makes this default especially dangerous is,
- Most people are not aware of this special mode that Linux allows for UDP 
sockets
- Even if it was documented to be the default, many people would miss it unless 
a big warning was slapped on the docs
- The problems are unlikely to appear in test scenarios and much more likely to 
pop up in production months or years after rolling out the code
- If you have never used it on purpose, it is very confusing to debug, causing 
you to doubt your own and the kernel's sanity
- The behaviour changes again if you happen to use a multicast address...

Thus, my proposal is to change the default value for UDP to False or deprecate 
the function and introduce a new one as suggested by Yuri in my original bug 
report at:  https://bugs.python.org/issue37228
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TK2NTPWID7RBUUNUU5JYAZHR6FKEABRU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Dangerous default value for reuse_address in asyncio loop.create_datagram_endpoint()

[Jukka Vaisanen]

When creating UDP servers with asyncio's create_datagram_endpoint(), the 
default value for reuse_address = True, resulting in a dangerous (and currently 
incorrectly documented) situation. I have proposed changing the default value 
but understandably such a change for a core library function parameter is not 
to be taken lightly. Thus I put this up for discussion on the list.

As background, when creating TCP servers on UNIX-like systems, it is almost 
boilerplate to set SO_REUSEADDR for all server sockets to make sure that a 
restarting server can immediately bind the socket again. Without the 
SO_REUSEADDR sockopt, the kernel will hold the addr:port in a TIME_WAIT state 
for a while, preventing reuse. Thus, when creating TCP servers with 
loop.create_server(), the parameter reuse_address has a very reasonable default 
value of True.

However things are very different in UDP-land. The kernel does not hold UDP 
server ports in a waiting state so the SO_REUSEADDR sockopt was repurposed in 
Linux (and *BSD afaik) to allow multiple processes to bind the SAME addr:port 
for a UDP server. The kernel will then feed incoming UDP packets to all such 
processes in a semi-fair-roundrobin manner. This is very useful in some 
scenarios, for example I've used it myself in C++ projects to allow UDP servers 
to be scaled easily and rolling upgrades to be implemented without separate 
load-balancing. But for this to be the default behaviour is quite dangerous.

I discovered this default behaviour accidentally by having 2 separate Python 
programs (both doing SIP over UDP) accidentally configured to use the same UDP 
port. The result was that my 2 processes were indeed "sharing the load" - 
neither of them threw an exception at startup about the port being already in 
use and both started getting ~half of the incoming packets. So off to the docs 
I went and discovered that the documentation for create_datagram_endpoint() 
does not mention this behaviour at all, instead it mistakenly refers to the TCP 
protocol use of SO_REUSEADDR: "reuse_address tells the kernel to reuse a local 
socket in TIME_WAIT state, without waiting for its natural timeout to expire. 
If not specified will automatically be set to True on Unix."
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.create_datagram_endpoint

What makes this default especially dangerous is,
- Most people are not aware of this special mode that Linux allows for UDP 
sockets
- Even if it was documented to be the default, many people would miss it unless 
a big warning was slapped on the docs
- The problems are unlikely to appear in test scenarios and much more likely to 
pop up in production months or years after rolling out the code
- If you have never used it on purpose, it is very confusing to debug, causing 
you to doubt your own and the kernel's sanity
- The behaviour changes again if you happen to use a multicast address...

Thus, my proposal is to change the default value for UDP to False or deprecate 
the function and introduce a new one as suggested by Yuri in my original bug 
report at:  https://bugs.python.org/issue37228
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YTWM2NRTUEKFSU4I3VHPR7JGMW6N5POI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: thes mapping data type and thescfg cfg file module - review and suggestion request

[Dave Cinege]

Tal,

On 2019/11/18 10:59, Tal Einat wrote:


These days, thanks to pip and PyPI, anyone can publish libraries and it 
is easy for developers to find and use them if they like. There is no 
longer a need to add things to the stdlib just to make them widely 
available.


Fair enough, only that you possibly neglected to remember the benefits 
to my ego if I get something accepted into mainline. :-D


Framing this a little more specifically:

Aside from my original Thesaurus release in Dec 2012, there have been 
many many attempts at extended/recursive dictionary objects. The use 
case for such a thing is well established. (JSON maybe enough to mention)


I would argue that a new standardized mapping datatype, possibly (some 
form of) Thesaurus, could be warranted in stdlib.


At this stage I consider Thesaurus 'serious' and 'interesting' because:

- I've been using some version of it for 7 years in my own
production code; many of the concepts in it are mature.

- It's tightly wrapped around and highly compatible with dict.

- I've paid close attention to performance

- The recent extended features for tree searches and the slicing
you can do with 'keypaths' (my own term) appears novel.

I am also sure that some will consider things I currently do in 
Thesaurus complete heresy. For now I'll point to ThesaurusCfg as an 
actually use case. (But still consider my use of coercion methods really 
cool :-)


My goal is to attempt to take Thesaurus toward a standardized or de 
facto recursive mapping object and I welcome any dialog that results.


Dave

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QLO2QDSAD3A3NUDJOGIJA2NUHD3N2GGN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Pass the Python thread state to internal C functions

[Nick Coghlan]

On Mon., 18 Nov. 2019, 8:19 am Nathaniel Smith,  wrote:

>
> > - Eventually make it easier for embedding applications to control which
> Python code runs in which thread state by moving the thread state
> activation dance out of the application and into the CPython shared library
>
> That seems like a good goal, but I don't understand how it's related
> to passing threadstate explicitly as a function argument. If the plan
> is to move towards passing threadstates both implicitly AND explicitly
> everywhere, that seems like it would make things more error-prone, not
> less, because the two states could get out of sync. Could you
> elaborate?
>

What I said my original reply: if an API that accepts an explicit thread
state ever calls an API that expects an implicit one, we'll need to
internally implement the dance to activate the supplied thread state before
making that call.

At the moment, we expect callers of the public API to do that dance, and
it's tricky to get it right in all cases.

My hope (and it's a subjective hope, not an objective fact) is that
implementing the dance more often ourselves will help us identify future
abstractions that will make the public API easier to use correctly in
multi-threaded applications.

Cheers,
Nick.



> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W3DJG5NTRCQJ45SYGYSJLGWC5AM2Z3W5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Dangerous default value for reuse_address in asyncio loop.create_datagram_endpoint()

[Nick Coghlan]

On Tue., 19 Nov. 2019, 6:46 am ,  wrote:

>
> Thus, my proposal is to change the default value for UDP to False or
> deprecate the function and introduce a new one as suggested by Yuri in my
> original bug report at:  https://bugs.python.org/issue37228


Guido's suggestion on the ticket sounds like a sensible solution to me:

* in 3.9, emit DeprecationWarning for cases where the default would
currently be "True"
* in 3.10, change the default to always be False

It would also make sense to update the 3.8 documentation to mention the
problematic default and its upcoming deprecation.

Cheers,
Nick.



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O6WHPMK6FPAWJJLG4VDWZHXSLSPPA6X2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: thes mapping data type and thescfg cfg file module - review and suggestion request

[Nick Coghlan]

On Tue., 19 Nov. 2019, 7:32 am Dave Cinege,  wrote:

> Tal,
>
> On 2019/11/18 10:59, Tal Einat wrote:
> >
> > These days, thanks to pip and PyPI, anyone can publish libraries and it
> > is easy for developers to find and use them if they like. There is no
> > longer a need to add things to the stdlib just to make them widely
> > available.
>
> Fair enough, only that you possibly neglected to remember the benefits
> to my ego if I get something accepted into mainline. :-D
>
> Framing this a little more specifically:
>
> Aside from my original Thesaurus release in Dec 2012, there have been
> many many attempts at extended/recursive dictionary objects. The use
> case for such a thing is well established. (JSON maybe enough to mention)
>
> I would argue that a new standardized mapping datatype, possibly (some
> form of) Thesaurus, could be warranted in stdlib.
>
> At this stage I consider Thesaurus 'serious' and 'interesting' because:
>
> - I've been using some version of it for 7 years in my own
> production code; many of the concepts in it are mature.
>
> - It's tightly wrapped around and highly compatible with dict.
>
> - I've paid close attention to performance
>
> - The recent extended features for tree searches and the slicing
> you can do with 'keypaths' (my own term) appears novel.
>


I think there's one potential avenue for making the case that a new
JSON-centric data type would be worth including: framing it as an
alternative to the None-aware operator proposal in PEP 505.

It may still not be successful, but ad hoc data manipulation has been the
most compelling use case presented for that PEP so far, and a standard
library addition would be a less intrusive change than a syntax update.

So if you wanted to pursue this idea further, my suggestions would be:

* post a thread on python-ideas seeking advice on building a credible case
for improving arbitrarily nested data container support in the standard
library, as we currently make it easy to build them, but not so easy to
destructure them later. (Replies to my suggestions below would also be
better directed to python-ideas)
* review PEP 505 and consider how the library might help with the use cases
mentioned there
* review other APIs like https://glom.readthedocs.io/en/latest/ and
consider the trade-offs between a data manipulation library that can work
with existing libraries that produce plain dictionaries and one that
requires a new data structure
* in the same vein, review whether it might be possible to use a "dict
wrapper" approach over a "dict subclass" approach (many potential users
will balk at a data copy operation, but would be amenable to a ChainMap
style wrapper)
* consider your options for tackling the naming conflict with
https://libraries.io/pypi/thesaurus (I must admit, "nested data structure
manipulation" is not what springs to mind when I hear "thesaurus". I'm far
more likely to think of synonyms, antonyms, and natural language processing)

Cheers,
Nick.




>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VAZOMJXUQCC6E57U656JQPNPVVZBXQ2Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: thes mapping data type and thescfg cfg file module - review and suggestion request

[Dave Cinege]

Guido,

Thank you for your generous comments. All of them.

On 2019/11/18 15:33, Guido van Rossum wrote:

It won't be easy to add this to the stdlib.


Maybe there are other existing threads that considered a new recursive 
mapping object for stblib that someone can point me to?


One thing that bugs me in 
particular is that you can access key via '.attr' notation (the example 
shows t['a']['b']['c']['d'] as equivalent to t.a.b.c.d). This feels 
problematic: What should happen if the key happens to be the name of a 
method (e.g. .keys or .update)? The choices you have are either to mask 
the method or to mask the key. Neither solution seems ideal. JavaScript 
has this equivalence and it makes me very uncomfortable.


Well, let's see:

>>> t.set_path('a.b.c.update', 'Hello')
>>> t.a.b.c.update

>>> t.a.b.c['update']
'Hello'

Attribute aliasing has limitations the coder must understand. Thesaurus 
attempts to deal with them gracefully. When in doubt normal key names 
should always work (including keys with dots in them) and if they don't 
I have something to fix. With the focus on being a datatype primitive I 
consider anything more to be the job of higher level modules. 
(ThesaurusCfg being an example)


The benefit of course is dramatically cleaner code for heavily nested 
objects and recursive string replacement compatibility. (f-string, 
printf, format, template)


And with that said, if the only obstacle to a new recursive mapping 
object for stblib is attribute aliasing, then it can be removed.


Dave

I'm not denying that Thesaurus[Cfg] looks useful. But, like Tal, I must 
stress that that's not enough to consider inclusion in the stdlib.


On Mon, Nov 18, 2019 at 4:25 PM Dave Cinege > wrote:


Hello Tal,

Yes I understand that. I posted this here because it's been
suggested by
those less in the know that Thesaurus and ThesaurusCfg (or some of the
the concepts contained in them) might have a place in the future
mainline.

Remember that Thesaurus is a data type and ThesaurusCfg is a
alternative
to configparser. Additionally I could bring this full circle to
something that integrates with or provides an alternative to argparse.
(providing an integrated cfg file and command line argument solution
which currently does not exist.)

I hope with this in mind this thread is considered on-topic and I'd
highly value any feedback from the 'serious people' on this list if
they
should take a look.

Dave



On 2019/11/18 08:51, Tal Einat wrote:
 > Hi Dave,
 >
 > Thesaurus looks interesting and it is obvious that you've put a
lot of
 > effort into it!
 >
 > This list is for the discussion of the development *of* Python
itself,
 > however, rather than development *with* Python, so it's not an
 > appropriate place for such posts.
 >
 > I suggest you post this on python-list and/or python-announce, to
get
 > this in front of a wider audience.
 >
 > - Tal Einat
 >
 > On Mon, Nov 18, 2019 at 7:17 AM Dave Cinege mailto:d...@cinege.com>
 > >> wrote:
 >
 >     If you are not aware:
 >
 >        - Thesaurus is a mapping data type with recursive keypath map
 >     and attribute aliasing. It is a subclass of dict() and is mostly
 >     compatible as a general use dictionary replacement.
 >
 >        - ThesaurusExtended is a subclass of Thesaurus providing
additional
 >     usability methods such as recursive key and value searching.
 >
 >        - ThesaurusCfg is a subclass of ThesaurusExtended
providing a nested
 >     key configuration file parser and per key data coercion methods.
 >
 >     The README.rsl will give a better idea:
 > https://git.cinege.com/thesaurus/
 >
 >
 >     After 7 years I might have reached the point of 'interesting'
with
 >     my Thesaurus and ThesaurusCfg modules.
 >
 >     To anyone that is overly bored, I'd appreciate your terse
review and
 >     comments on how mundane and worthless they still actually
are. :-)
 >
 >     I'm primarily interested in suggestions to what I've done
conceptually
 >     here. While I'm not completely ashamed of the state of the
Thesaurus
 >     code, ThesaurusCfg is not far beyond the original few hours I
 >     slapped it
 >     together one day in frustration.
 >
 >     After considering suggestions I intend to make changes
towards a formal
 >     release.
 >
 >     Thanks in advance,
 >
 >     Dave
 >     ___
 >     Python-Dev mailing list -- python-dev@python.org

 >     >
 >     To unsubscribe send an email to python-dev-le..

[Python-Dev] Exposing Tools/parser/unparse.py in the stdlib?

[Pablo Galindo Salgado]

Hi,

What do people feel about exposing Tools/parser/unparse.py in the standard
library? Here is my initial rationale:

* The tool already needs to be maintained and updated as is tested as part
of the test suite.
* I have used the tool almost all the time I needed to deal with AST
transformations.
* The public interface will have a very low surface API, keeping
maintaining it (the public interface) a very small burden IMHO.

We could add the public interface to the ast.py module or a new one if
people feel strongly about it.

Does anyone feel strongly against this or have any objection that I am not
contemplating?

Regards from rainy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JAQDBMC23HW2PQ27HQNJ7G244T423IDD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

[Mahmoud Hashemi]

>From the first time I found it years ago, I've often wondered why it wasn't
exposed. I presumed due to certain API shifts I wasn't paying close
attention to, that maybe the ast module was buffering. But I've definitely
wanted to import it in the past.

If the API is as stable as you say, then I guess the only question I'd have
is: will the (potential) upcoming shift to PEG [1] change that? :P

[1]: https://www.youtube.com/watch?v=QppWTvh7_sI

On Mon, Nov 18, 2019 at 4:48 PM Pablo Galindo Salgado 
wrote:

> Hi,
>
> What do people feel about exposing Tools/parser/unparse.py in the standard
> library? Here is my initial rationale:
>
> * The tool already needs to be maintained and updated as is tested as part
> of the test suite.
> * I have used the tool almost all the time I needed to deal with AST
> transformations.
> * The public interface will have a very low surface API, keeping
> maintaining it (the public interface) a very small burden IMHO.
>
> We could add the public interface to the ast.py module or a new one if
> people feel strongly about it.
>
> Does anyone feel strongly against this or have any objection that I am not
> contemplating?
>
> Regards from rainy London,
> Pablo Galindo Salgado
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/JAQDBMC23HW2PQ27HQNJ7G244T423IDD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WYVOBWFZKPKDCWGUWDRK26HK5PVPFGEC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

[Guido van Rossum]

I don't think the PEG parser switch would change anything -- it produces
the same AST as the old parser and ast.parse().

It looks a reasonable thing to move into the stdlib (maybe as
ast.unparse()?), assuming it's complete. Tests and doc need to be written.

On Tue, Nov 19, 2019 at 1:08 AM Mahmoud Hashemi  wrote:

> From the first time I found it years ago, I've often wondered why it
> wasn't exposed. I presumed due to certain API shifts I wasn't paying close
> attention to, that maybe the ast module was buffering. But I've definitely
> wanted to import it in the past.
>
> If the API is as stable as you say, then I guess the only question I'd
> have is: will the (potential) upcoming shift to PEG [1] change that? :P
>
> [1]: https://www.youtube.com/watch?v=QppWTvh7_sI
>
> On Mon, Nov 18, 2019 at 4:48 PM Pablo Galindo Salgado 
> wrote:
>
>> Hi,
>>
>> What do people feel about exposing Tools/parser/unparse.py in the
>> standard library? Here is my initial rationale:
>>
>> * The tool already needs to be maintained and updated as is tested as
>> part of the test suite.
>> * I have used the tool almost all the time I needed to deal with AST
>> transformations.
>> * The public interface will have a very low surface API, keeping
>> maintaining it (the public interface) a very small burden IMHO.
>>
>> We could add the public interface to the ast.py module or a new one if
>> people feel strongly about it.
>>
>> Does anyone feel strongly against this or have any objection that I am
>> not contemplating?
>>
>> Regards from rainy London,
>> Pablo Galindo Salgado
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/JAQDBMC23HW2PQ27HQNJ7G244T423IDD/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WYVOBWFZKPKDCWGUWDRK26HK5PVPFGEC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3ZI6YWVHX4ESSZNM3GFYRTYTHX4YHUNU/
Code of Conduct: http://python.org/psf/codeofconduct/