[Python-Dev] Re: Increase of Spammy PRs and PR reviews

[Martin Dengler]

Hi,

On Tue, Feb 01, 2022 at 10:19:12AM -0500, Éric Araujo wrote:

 Hi,

Le 01/02/2022 à 07:25, Martin Dengler a écrit :

The fact people are assuming bad faith and spilling ink about those getting
involved with that extremely-underserved part when contributors are sorely > 
needed is counter-productive.


 I disagree, the original message was a good-faith question about how 
to interpret a pattern.  It is healthy to check with the group to 
discuss if there is a problem and how to deal with it.


I understand what you're saying; if we meet in person I'm happy to discuss 
further -- but as my original point was that long messages about small, 
probably-in-good-faith-but-could-be-in-bad-faith actions (as these turned out 
to be) are (further) wastes of time (net-net), I won't continue.


[snip]

 Regards


Regards,
Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/66GTVP7HIGQ6WT52PCGAFPW2L3LZ22W7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Increase of Spammy PRs and PR reviews

[Martin Dengler]

On Tue, Feb 01, 2022 at 12:35:02AM -0500, Terry Reedy wrote:

On 1/31/2022 7:31 PM, Nikita Sobolev wrote:

Hi, my name is Nikita and I think that I am the person behind these spammy PRs.
Link: https://github.com/python/cpython/pulls/sobolevn


I also encouraged multiple easily reviewable PRs from you.  Please
continue.


Preening a large codebase is not only a good thing, but encourages familiarity:
not only with the code, but with the opaque and worst-documentend parts of the
software development processes: the maintenance lifecycle.  The fact people are
assuming bad faith and spilling ink about those getting involved with that
extremely-underserved part when contributors are sorely needed is
counter-productive.

It would be better if more people contributed small changes and thus became
low-friction contributors who don't require a lot of mentoring.


Terry Jan Reedy


Martin

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IK2GOZ3R67QGZJH2CQUVWHYQPUDEOU6B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] bpo-41544: Add missing parameters to dummy.DummyProcess and dummy.Pool

[Martin Schröder]

Hi,

could someone please review my pull request?

https://github.com/python/cpython/pull/21869

It adds missing parameters to multiprocessing.dummy.DummyProcess in 
order to match the API of multiprocessing.Process.


Best regards,
Martin Schröder
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NVNRA57DDTBMO5FQR7D3ELB5436XZDLE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Dropping out of this list

[Martin Dengler]

On Wed, Aug 18, 2021 at 10:36:00PM -0400, Jonathan Goble wrote:

On Wed, Aug 18, 2021 at 10:22 PM Terry Reedy  wrote:


On 8/18/2021 9:37 PM, Edwin Zimmerman wrote:
> On 8/18/21 9:18 PM, Jonathan Goble wrote:
>> I am mostly a lurker, but I am also considering unsubscribing if
someone doesn't step in and stop the mess
>
> +1

Both the email and newsreader parts of Thunderbird have an option called
Ignore Thread.  Do your readers have such?



I consume my email from a variety of disparate devices, including a Debian
laptop, a Windows desktop, a Chromebook, an Android phone, and various
university-managed computer labs. As such, and as I rely heavily on Gmail's
filtering and labeling system to manage my email, I rely solely on Google's
standard web interface and Android app for Gmail for the sake of a
consistent experience across devices. I don't know if Google offers such an
option. It takes me about three seconds to click a thread, see the arguing,
and click Archive, but I'm getting tired of doing that repeatedly.


https://gsuitetips.com/tips/gmail/how-to-mute-and-find-email-conversations/


To mute an email thread, simply:
- Within Gmail:
- Open the email conversation you want to mute
- Click more and in the drop down menu select mute
[...]

When a conversation is muted, any new message will bypass your inbox so
that the conversation stays archived. Muted conversations will reappear in
your inbox if a new message in the conversation is addressed to you and no
one else, or if you're added to the "To" or "Cc" line in a new message.


Does that fix your problem?

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HTKWGADUUAIPYSJVHCSQ6GKLPICM6Z56/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Delayed evaluation of f-strings?

[Martin (gzlist) via Python-Dev]

On Thu, 24 Jun 2021 at 17:37, Martin (gzlist)  wrote:
>
> >>> d = dict(a=1)
> >>> f'{d["a"]}'
> '1'
> >>> str(DelayedFString('{d["a"]}'))
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "", line 5, in __str__
> KeyError: '"a"'

And the other side of the attribute lookup:

>>> d = dict(a=1)
>>> str(DelayedFString('{d[a]}'))
'1'
>>> f'{d[a]}'
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'a' is not defined

Yes, having three different ways of doing string interpolation (not
counting other things you can import, like string.Template) is a bit
confusing.

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A7FQUWQND56VGHCTTCFW6XDNCWP5MVNM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Delayed evaluation of f-strings?

[Martin (gzlist) via Python-Dev]

On Thu, 24 Jun 2021 at 17:25, Eric Nieuwland  wrote:
>
> class DelayedFString(str):
> def __str__(self):
> vars = inspect.currentframe().f_back.f_globals.copy()
> vars.update(inspect.currentframe().f_back.f_locals)
> return self.format(**vars)

This isn't quite right as the semantics between f-strings and
str.format() are not actually the same (though this isn't well
documented):

>>> f'{1 + 2}'
'3'
>>> str(DelayedFString('{1 + 2}'))
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 5, in __str__
KeyError: '1 + 2'

>>> d = dict(a=1)
>>> f'{d["a"]}'
'1'
>>> str(DelayedFString('{d["a"]}'))
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 5, in __str__
KeyError: '"a"'

Basically, f-strings rely on eval-like semantics.

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HGITYZPPRXJM4Y7YVJKUSVXUB75W5Z2L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Help with the grail browser

[Martin Panter]

On 30/05/2021, Prakhar Sehgal  wrote:
> The latest version of Grail is 0.6 which was released in 1999 was made for
> Python 1 but I want to make one compatible with Python 3. But I need help
> for that. Currently me and Guido van Rossum are working on this. Any help
> will be welcomed. This is the link for the GitHub repo:
> https://github.com/PrakharPythonProgrammer/Grail-0.6

Quite a while ago I got Grail working with Python 3. That was in the
era of 3.3; there might be more deprecated modules since (I'm thinking
the "formatter" module at least). I'm pretty sure I put all the
important changes on the "py3" branch on my Git Hub repository, at
<https://github.com/vadmium/grailbrowser/tree/py3>. It may be useful
to you.

-Martin.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RQTC4E3Q3I6PHXCQOX23AV6K2FTQ47EG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: On the migration from master to main

[Martin (gzlist) via Python-Dev]

On Tue, 23 Mar 2021 at 21:39, Python Steering Council
 wrote:
>
> This isn’t happening because GitHub/Microsoft made a political decision. It’s 
> happening because it is incredibly easy to make this move, many projects have 
> already done this, and it reflects badly on any project not making this 
> change.

"We do this not because it's right, but because it's easy."

Great to see Python adopting the motto of our new century.

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DDYQXHYATCMPD3W2OHETZLTVN5DJ4VIO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Steering Council reply regarding conduct (was Re: Steering Council update for February)

[Martin Dengler]

On Tue, Mar 23, 2021 at 12:02:38PM -0700, Python Steering Council wrote:

From Thomas Wouters, on behalf of and with full support of the Python Steering
Council:
[use of SC power; specifically, PEP-0013.Powers.2: 'Enforce ... code of 
conduct']


From PEP 13[^1]

To use its powers, the council votes.

[...]

Whenever possible, the council's deliberations and votes shall be held in 
public.


Please share the deliberations and votes.

Martin

[^1]: https://github.com/python/peps/blob/master/pep-0013.rst
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WWVV7ONKG7GD73JJ7YBPHTSRH2PCRY3U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Integer concatenation to byte string

[Martin (gzlist) via Python-Dev]

On Tue, 2 Mar 2021 at 21:44, Memz  wrote:
>
> foo+= 255 # Works the same as 
> bytesvariable+=b"ÿ"

foo = b"%b%d" % (foo, 255)

> foo+= a"\x255\x00"  # Concatenation with itself

foo = b"%b%b" % (foo, foo)

See PEP461:

Adding % formatting to bytes and bytearray
<https://www.python.org/dev/peps/pep-0461/>
"With Python 3 and the split between str and bytes, one small but
important area of programming became slightly more difficult, and much
more painful -- wire format protocols."

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TM2XN4P6JMOFP3RUSFJLFPTDC5J77J7O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Martin Teichmann]

Hi Greg, Hi List,

Greg wrote:

>  class MyClass(B, Mixin):
> >"whatever"
> >
> > this leads to an MRO of MyClass -> B -> Mixin -> A -> object.
>
> If you do the tp_new stuff correctly at the C level, you can still
> create such a class. The only limitation is that if Mixin has a __new__
> method written in Python, it won't get called.


Yes, it is indeed possible to write a class like this. But that would be,
as mentioned in the comment, silly. Now there is nothing bad with being
silly once in a while, especially while programming that cannot be avoided
at times, I am just warning to use such a construct in a wider context.

Because the limitation is actually much more general: if the Mixin has any
method written in Python, it won't get called, unless you do a lot of heavy
lifting. Now why do you write something in C at all? Two reasons: once for
speed, which we can effectively exclude here because the super() construct
is slow, or because you want to interact with a library written in another
language. Unless your library knows about cooperative inheritance, which is
very unlikely, the code from class B is not aware that it is not supposed
to call the method inherited from A, but the one from the Mixin instead.

So by writing

   class MyClass(B, Mixin):
"whatever"

I have major problems to understand what you actually want to achieve, what
you could not achieve by writing

class MyClass(Mixin, B):
"whatever"

Since one cannot achieve anything with the former that could not be
achieved by the latter, just with much less hassle, I would call the former
code silly.

That said, I might be wrong and there is indeed something one can achieve
with the former but not with the latter, I would be very pleased to hear
about this, so please come forward! (For sure, all is based on B and A
being written in C, not Python)

Cheers

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VQPMUJKMZ44MLUPULFERPLMM5IXEKFYE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Understanding "is not safe" in typeobject.c

[Martin Teichmann]

Hi Phil, Hi List,

unfortunately you do not give enough code to reproduce what you are doing,
but just guessing roughly:

you say that you have a hierarchy like B -> A -> object, with B and A
implemented in C, and then want to use B with a mixin. Programmers with a
non-python background then often write

class MyClass(B, Mixin):
  "whatever"

this leads to an MRO of MyClass -> B -> Mixin -> A -> object. This is
horror if B and A are written in C, because suddenly B needs to do
something with Python code if it wants to have to do something with its
superclass Mixin, like creating a new object. I am just guessing that this
is what your code tries to do. And this is what the comment considers silly.

With

class MyClass(Mixin, B):
 "whatever"

there is no problem at all. We get an MRO of MyClass -> Mixin -> B -> A ->
object. There is no need for B to do anything special, being written in C
it already knows by itself how to construct A as well, no need to fiddle
with Python at all. In general, it is usually not necessary to deal with
super() in C code at all.

The statement that there is only single inheritance on the C level becomes
obvious once you look at the MRO: that one is always linear, it is always
effectively a single inheritance. This is also why you have to call super()
only once even if you have multiple superclasses: super() just follows the
effective single inheritance of the MRO.

Hope that helps.

Cheers

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WHTRIPKQMVCQPOYBFWRW6HI6KOBUYJU3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 638: Syntactic macros

[Martin (gzlist) via Python-Dev]

On Fri, 16 Oct 2020 at 23:22, Guido van Rossum  wrote:
>
> Dima,
>
> Do you have a link to "babel macros"? Searching for that brought up several 
> different things; not being a frequent JS user I don't know how to filter 
> these.

These links should help:

https://babeljs.io/blog/2017/09/11/zero-config-with-babel-macros
https://github.com/kentcdodds/babel-plugin-macros
https://github.com/jgierer12/awesome-babel-macros

That's a general intro, the code repo for the macro plugin, and a repo
that lists implemented macros.

Martin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HJR6UF2XPOC6UUY6QRLKYCYZOWK2BYDD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-ideas] Re: Amend PEP-8 to require clear, understandable comments instead of Strunk & White Standard English comments

[Martin Dengler]

On Sat, Jul 04, 2020 at 05:51:04PM +0100, MRAB wrote:
I'd also add: Try to avoid regionalisms; aim for a 
broadly "international" form of the language. Some 


How do you spell "regionalism"?

Martin

PS: Irony intended
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3L7UEATIHJ4JV5UDXLMKRUMC3H2EXWLY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Something wrong with the bug tracker?

[Martin (gzlist) via Python-Dev]

Logging in with openid using launchpad just worked for me, so
transient or related to google's openid support presumably.

Martin

On Sun, 22 Dec 2019 at 15:13, Mark Shannon  wrote:
>
> Hi,
>
> For some reason I can't log into the bug tracker with Google.
> I just get the error message:
>
> """
> An error has occurred
>
> A problem was encountered processing your request.
> The tracker maintainers have been notified of the problem.
> """
>
> Whether I'm logged in or out of Google, or using a private tab, it makes
> no difference.
>
> Anyone else having problems?
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/BXWXJYB2NDDRCM7CDWL2X3IWVDW7ZE7D/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KADR37G3JOAJWNB6VOJQDFNHSTZ2PH7Q/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-Dev] git history conundrum

[Martin Panter]

On Sat, 27 Apr 2019 at 19:07, Chris Withers  wrote:
> Right, so I've merged up to 15f44ab043, what comes next?
>
> $ git log --oneline  --no-merges 15f44ab043.. -- Lib/unittest/mock.py
> Lib/unittest/test/testmock/ | tail -n 3

This Git command line means list all the revisions except 15f44ab043
and those leading up to it.

> 161a4dd495 Issue #28919: Simplify _copy_func_details() in unittest.mock
> ac5084b6c7 Fixes issue28380: unittest.mock Mock autospec functions now
> properly support assert_called, assert_not_called, and assert_called_once.
> 0be894b2f6 Issue #27895:  Spelling fixes (Contributed by Ville Skyttä).
>
> Okay, no idea why 0be894b2f6 is there, appears to be a totally identical
> commit to 15f44ab043

Git revision 15f44ab043 is the original spelling fixes, which were
pushed to the Mercurial “default” branch (= Git master) for Python 3.6
by Raymond. Revision 0be894b2f6 is my backport to the 3.5 branch, done
about a week later. The backport is probably a subset of the original,
rather than identical (e.g. the datetime.rst change was not applicable
to 3.5).

The convention at the time was to keep the 3.5 branch merged into
Default (Master). That is why my 3.5 backport appears in your history
of Master.

> so let's skip it:
>
> $ git log --oneline  --no-merges 0be894b2f6.. -- Lib/unittest/mock.py
> Lib/unittest/test/testmock/ | tail -n 3
> 161a4dd495 Issue #28919: Simplify _copy_func_details() in unittest.mock
> ac5084b6c7 Fixes issue28380: unittest.mock Mock autospec functions now
> properly support assert_called, assert_not_called, and assert_called_once.
> 15f44ab043 Issue #27895:  Spelling fixes (Contributed by Ville Skyttä).
>
> Wat?! Why is 15f44ab043 showing up again?!

Because you are asked for all the revisions except my backport and its
ancestors. As far as Git is concerned, the original spelling fixes are
not an ancestor of my backport.

I don’t have a copy of the Git repository to try, but I suggest the
following command is what you want:

git log --oneline  --no-merges HEAD ^15f44ab043 ^0be894b2f6 --
Lib/unittest/mock.py Lib/unittest/test/testmock/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Replacement for array.array('u')?

[Martin (gzlist) via Python-Dev]

On Fri, 22 Mar 2019 at 16:12, Steve Dower  wrote:
>
> On 22Mar2019 0433, Antoine Pitrou wrote:
> > The question is: why would you use a array.array() with a Windows C API?
>
> I started replying to this with a whole lot of examples, and eventually
> convinced myself that you wouldn't (or shouldn't).
>
> That said, I see value in having a type for array/struct/memoryview that
> "is the same size as wchar_t", since that will avoid people having to
> guess (the docs for array in particular are deliberately vague about the
> actual size of the various types).

This is pretty much what ctypes provides for dealing with unicode?

https://docs.python.org/3/library/ctypes.html#ctypes.create_unicode_buffer

Seems a fine place to have things that help with win32 api interactions.

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] bpo-34203: FAQ now recommends python 3.x over 2.x (GH-9796)

[Martin Panter]

On 12/10/2018, Eric V. Smith  wrote:
> On 10/12/2018 5:17 AM, Tal Einat wrote:
>
>>   The latest stable releases can always be found on the `Python download page
>> -`_.  There are two recommended 
>> production-ready
>> -versions at this point in time, because at the moment there are two 
>> branches of
>> -stable releases: 2.x and 3.x.  Python 3.x may be less useful than 2.x, since
>> -currently there is more third party software available for Python 2 than for
>> -Python 3.  Python 2 code will generally not run unchanged in Python 3.
>> +`_.  There are two production-ready 
>> version
>> +of Python: 2.x and 3.x, but the recommended one at this times is Python 3.x.
>
> This should be "time", not "times". I'd fix it, but I'm unsure if this
> is being backported or not, and I don't want to mess up any merges
> before they're done.

Or just remove “at this time[s]”; it doesn’t add much.

Also, “two . . . version” should be changed back to plural: “two . . .
versions”.

> I do think this should backported to 3.7, 3.6, and 2.7.
>
>> +Although Python 2.x is still widely used, `it will not be
>> +maintained after January 1, 2020
>> `_.
>> +Python 2.x was known for having more third-party libraries available,
>> however,
>> +by the time of this writing, most of the widely used libraries support
>> Python 3.x,
>
> Should probably be "at the time of this writing".
>
>> +and some are even dropping the Python 2.x support.
>
> And this would read better as "are even dropping support for Python 2.x".
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Microsoft to acquire GitHub for $7.5 b

[Martin Gainty]

who owns the Data hosted on Github?

Github Author?
Microsoft?


Martin
__





From: Python-Dev  on behalf 
of M.-A. Lemburg 
Sent: Tuesday, June 5, 2018 7:54 AM
To: Antoine Pitrou; python-dev@python.org
Subject: Re: [Python-Dev] Microsoft to acquire GitHub for $7.5 billion

Something that may change is the way they treat Github
accounts, after all, MS is very much a sales driven company.

But then there's always the possibility to move to Gitlab
as alternative (hosted or run on PSF VMs), so I would
worry too much.

Do note, however, that the value in Github is not so much with
the products they have, but with the data. Their databases
know more about IT developer than anyone else and given
that Github is put under the AI umbrella in MS should tell
us something :-)


On 04.06.2018 19:02, Antoine Pitrou wrote:
>
> That's true, but Microsoft has a lot of stakes in the ecosystem.
> For example, since it has its own CI service that it tries to promote
> (VSTS), is it in Microsoft's best interest to polish and improve
> integrations with other CI services?
>
> Regards
>
> Antoine.
>
>
> On Mon, 4 Jun 2018 09:06:28 -0700
> Guido van Rossum  wrote:
>> On Mon, Jun 4, 2018 at 8:40 AM, Antoine Pitrou  wrote:
>>
>>>
>>> On Mon, 4 Jun 2018 17:03:27 +0200
>>> Victor Stinner  wrote:
>>>>
>>>> At this point, I have no opinion about the event :-) I just guess that
>>>> it should make GitHub more sustainable since Microsoft is a big
>>>> company with money and interest in GitHub. I'm also confident that
>>>> nothing will change soon. IMHO there is no need to worry about
>>>> anything.
>>>
>>> It does spell uncertainty on the long term.  While there is no need to
>>> worry for now, I think it gives a different colour to the debate about
>>> moving issues to Github.
>>>
>>
>> I don't see how this *increases* the uncertainty. Surely if GitHub had
>> remained independent there would have been be similar concerns about how it
>> would make enough money to stay in business.
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
Python-Dev Info Page<https://mail.python.org/mailman/listinfo/python-dev>
mail.python.org
Do not post general Python questions to this list. For help with Python please 
see the Python help page.. On this list the key Python developers discuss the 
future of the language and its implementation.



> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
>

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jun 05 2018)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/mgainty%40hotmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 572: Why not := as standard assignment operator?

[Martin Teichmann]

Hi list,

when reading PEP 572 I actually liked it a lot - I think it's actually
a cool idea. I think it's actually that cool an idea that it should be
made the default way of doing an assignment, over time phasing out the
good ole =.

This would have several benefits:

- people wouldn't have to worry about two different options
- different things would have a different look: assignment is :=,
keyword args is =, while comparison is ==. Especially beginners would
benefit from this clarity.

in this case, for sure, we should make it possible to chain :=s, for
example by making it bind right-to-left, so that  a := b := 3 would be
a := (b := 3)

I'm sorry if somebody brought that up already, but the discussion has
grown so huge that I couldn't read through it entirely.

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Basic test to validate win10 install?

[Martin Gainty]

 Please feel free to reply to
python-dev@python.org rather than me personally to take this back to
the list.

On Tue, Apr 17, 2018 at 1:20 PM, Martin Gainty <mgai...@hotmail.com> wrote:
> I'll need a few specifics before i wandering into "dll hell"
>
> .NetFramework version?
> RT Library version? in the old days we used to call this msvcrt.dll
>
> msvc or mingw compiler?
> if msvc which version msvc?
> if mingw which version msvc ?

Have a look at https://devguide.python.org/ to get started
Python Developer’s Guide — Python Developer's 
Guide<https://devguide.python.org/>
devguide.python.org
Contributing¶. We encourage everyone to contribute to Python and that’s why we 
have put up this developer’s guide. If you still have questions after reviewing 
the material in this guide, then the Core Python Mentorship group is available 
to help guide new contributors through the process.



contributing to CPython, and in particular
https://devguide.python.org/setup/#windows for getting set up on
Windows.

To answer your questions in brief, though: we use the latest MSVC
compiler available with Visual Studio 2017, which links to the
"universal CRT" and avoids most of the old headaches of
incompatibilities between the runtimes used by different compiler
versions.

You may also be interested in the core-mentorship mailing list,
details about which can be found at
https://mail.python.org/mm3/mailman3/lists/core-mentorship.python.org/

Regards,
--
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Basic test to validate win10 install?

[Martin Gainty]

Martin Gainty has shared a OneDrive file with you. To view it, click the link 
below.


<https://1drv.ms/u/s!AkpuiYcNg4cShQZgKKBOeW9mOW0X>
[https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<https://1drv.ms/u/s!AkpuiYcNg4cShQZgKKBOeW9mOW0X>

errors 3.lst<https://1drv.ms/u/s!AkpuiYcNg4cShQZgKKBOeW9mOW0X>





i ran python test suite here and attaching output for your purview..

displaying output from stderr


Python36>python -m test
== CPython 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit 
(AMD64)]
== Windows-10-10.0.16299-SP0 little-endian
== cwd: .\test_python_6296
== CPU count: 4
== encodings: locale=cp1252, FS=utf-8
Run tests sequentially
0:00:00 [  1/406] test_grammar
0:00:00 [  2/406] test_opcodes
0:00:00 [  3/406] test_dict
0:00:00 [  4/406] test_builtin
0:00:00 [  5/406] test_exceptions
0:00:02 [  6/406] test_types
0:00:02 [  7/406] test_unittest
0:00:04 [  8/406] test_doctest
0:05:41 [  9/406] test_doctest2 -- test_doctest passed in 5 min 37 sec
0:05:41 [ 10/406] test_support
0:05:41 [ 11/406] test___all__
0:05:57 [ 12/406] test___future__
0:07:30 [ 13/406] test__locale -- test___future__ passed in 1 min 34 sec
0:07:30 [ 14/406] test__opcode
0:07:31 [ 15/406] test__osx_support
0:07:31 [ 16/406] test_abc
0:07:31 [ 17/406] test_abstract_numbers
0:07:31 [ 18/406] test_aifc
0:07:31 [ 19/406] test_argparse
0:07:33 [ 20/406] test_array
0:07:37 [ 21/406] test_asdl_parser
test_asdl_parser skipped -- test irrelevant for an installed Python
0:07:37 [ 22/406] test_ast -- test_asdl_parser skipped
0:07:39 [ 23/406] test_asyncgen
0:08:01 [ 24/406] test_asynchat
0:08:03 [ 25/406] test_asyncio
0:08:53 [ 26/406] test_asyncore -- test_asyncio passed in 50 sec
0:11:38 [ 27/406] test_atexit -- test_asyncore passed in 2 min 45 sec
0:11:39 [ 28/406] test_audioop
0:11:39 [ 29/406] test_augassign
0:11:39 [ 30/406] test_base64
0:11:40 [ 31/406] test_baseexception
0:11:40 [ 32/406] test_bigaddrspace
0:11:40 [ 33/406] test_bigmem
0:11:40 [ 34/406] test_binascii
0:11:40 [ 35/406] test_binhex
0:11:40 [ 36/406] test_binop
0:11:40 [ 37/406] test_bisect
0:11:40 [ 38/406] test_bool
0:11:40 [ 39/406] test_buffer
0:11:47 [ 40/406] test_bufio
0:11:52 [ 41/406] test_bytes
0:11:54 [ 42/406] test_bz2
0:12:01 [ 43/406] test_calendar
0:12:05 [ 44/406] test_call
0:12:05 [ 45/406] test_capi
0:12:09 [ 46/406] test_cgi
0:12:09 [ 47/406] test_cgitb
0:12:10 [ 48/406] test_charmapcodec
0:12:10 [ 49/406] test_class
0:12:10 [ 50/406] test_cmath
0:12:10 [ 51/406] test_cmd
0:12:10 [ 52/406] test_cmd_line
0:12:14 [ 53/406] test_cmd_line_script
0:12:20 [ 54/406] test_code
0:12:20 [ 55/406] test_code_module
0:12:20 [ 56/406] test_codeccallbacks
0:12:20 [ 57/406] test_codecencodings_cn
0:12:21 [ 58/406] test_codecencodings_hk
0:12:21 [ 59/406] test_codecencodings_iso2022
0:12:21 [ 60/406] test_codecencodings_jp
0:12:22 [ 61/406] test_codecencodings_kr
0:12:22 [ 62/406] test_codecencodings_tw
0:12:22 [ 63/406] test_codecmaps_cn
0:12:24 [ 64/406] test_codecmaps_hk
0:12:25 [ 65/406] test_codecmaps_jp
0:12:28 [ 66/406] test_codecmaps_kr
0:12:30 [ 67/406] test_codecmaps_tw
0:12:31 [ 68/406] test_codecs
0:12:33 [ 69/406] test_codeop
0:12:33 [ 70/406] test_collections
0:12:34 [ 71/406] test_colorsys
0:12:34 [ 72/406] test_compare
0:12:34 [ 73/406] test_compile
0:12:34 [ 74/406] test_compileall
0:12:44 [ 75/406] test_complex
0:12:44 [ 76/406] test_concurrent_futures
0:14:00 [ 77/406] test_configparser -- test_concurrent_futures passed in 1 min 
16 sec
0:14:01 [ 78/406] test_contains
0:14:01 [ 79/406] test_contextlib
0:14:01 [ 80/406] test_copy
0:14:02 [ 81/406] test_copyreg
0:14:02 [ 82/406] test_coroutines
0:14:02 [ 83/406] test_cprofile
0:14:03 [ 84/406] test_crashers
0:14:03 [ 85/406] test_crypt
test_crypt skipped -- No module named '_crypt'
0:14:03 [ 86/406] test_csv -- test_crypt skipped
0:14:03 [ 87/406] test_ctypes
0:14:05 [ 88/406] test_curses
test_curses skipped -- Use of the 'curses' resource not enabled
0:14:05 [ 89/406] test_datetime -- test_curses skipped (resource denied)
0:14:06 [ 90/406] test_dbm
0:14:06 [ 91/406] test_dbm_dumb
0:14:07 [ 92/406] test_dbm_gnu
test_dbm_gnu skipped -- No module named '_gdbm'
0:14:07 [ 93/406] test_dbm_ndbm -- test_dbm_gnu skipped
test_dbm_ndbm skipped -- No module named '_dbm'
0:14:07 [ 94/406] test_decimal -- test_dbm_ndbm skipped
0:14:14 [ 95/406] test_decorators
0:14:14 [ 96/406] test_defaultdict
0:14:14 [ 97/406] test_deque
0:14:17 [ 98/406] test_descr
0:14:19 [ 99/406] test_descrtut
0:14:19 [100/406] test_devpoll
test_devpoll skipped -- test works only on Solaris OS family
0:14:19 [101/406] test_dict_version -- test_devpoll skipped
0:14:19 [102/406] test_dictcomps
0:14:19 [103/406] test_dictviews
0:14:20 [104/406] test_difflib
0:14:20 [105/406] test_dis
0:14:21 [106/406] test_distutils

.\test_python_6296>exit 1

.\test_python_6296>exit 0
Warning -- files was modified by test_distutils
  Before: []
  After:  ['_configtest.c']
test test_distutil

Re: [Python-Dev] Is object the most base type? (bpo-20285)

[Martin Panter]

> On Fri, Feb 02, 2018 at 01:53:00AM -0500, Terry Reedy wrote:
>> >>> object.__doc__
>> 'The most base type'
> [...]
>> I have suggested
>> "The superclass for all Python classes."
>> "The starting base class of all types and classes other than itself."
>>
>> I intended to pick the second, but Serhiy Storchake wants more opinions.

On 2 February 2018 at 07:25, Steven D'Aprano  wrote:
> "the base class of the class heirarchy"
>
> "the root of the class heirarchy"

The RST documentation

says

Object is a base for all classes.

which is better than “the most base type” IMO, although I would write
_the_ base rather than _a_ base; there is only one object class. In my
patch for the RST documentation in
 I proposed

This is the ultimate base class of all other classes.

which I still think is okay (though other parts of my patch doesn’t
seem very good).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)

[Martin Teichmann]

>> I do worry that things like your autoslots decorator example might be
>> problematic because they create a new class, throwing away a lot of work
>> that was already done. But perhaps the right way to address this would be
>> to move the decision about the instance layout to a later phase? (Not sure
>> if that makes sense though.)
>>
>> --Guido
>
>
> Just FYI, recreating the class with slots runs into problems with regards to
> PEP 3135 (New Super). In attrs we resort to black magic to update the
> __class__ cell in existing methods. Being able to add slotness later would
> be good, but not that useful for us since we have to support down to 2.7.

You're both bringing up an important point here: while in function
decorators it is normal to return a completely new function (albeit
one that wraps the original), this is close to impossible for classes.
You cannot just wrap a class in another one. You may inherit from it,
but that's already often not what one wants.

While I am not worried about the poor computers having to do a lot of
work creating a throwaway class, I do see the problem with the new
super. What I would like to see is something like the @wraps decorator
for classes, such that you could write something like:

def class_decorator(cls):
@wraps_class
class MyNewCoolClass:
  """my cool functionality here"""
 return MyNewCoolClass

wraps_class would then copy over everything such that the new class gets it.

Unfortunately, this won't work, because of the new super. The new
super is about the only thing that cannot be dynamically changed in
Python. While it is no problem to make a function a method of a random
class (just use __get__), it is not possible to move a function from
one class to another, because you cannot change its binding to
__class__, which is used by super(). And even if we could, the method
whose __class__ we want to change might hide in a wrapper.

The current behavior of __class__ is weird, it is set to the class
that type.__new__ creates. So even if another metaclasses __new__ or a
decorator returns another class, the method's __class__ would still
point to the original class, which might even not exist anymore.

One might argue that this is due to the fact that it is not
well-defined what __class__ should be set to. But this is not true, it
is crystal clear: __class__ should be set to the class from whose
__dict__ the method was drawn. I thought about a new three-parameter
__get__(self, instance, owner, supplier), which would then set
__class__ to the supplier. This is a beautiful concept, that is
unfortunately not so simple when it comes to methods hidden in
wrappers.

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)

[Martin Teichmann]

> Things that will not work if Enum does not have a metaclass:
>
> list(EnumClass) -> list of enum members
> dir(EnumClass)  -> custom list of "interesting" items
> len(EnumClass)  -> number of members
> member in EnumClass -> True or False
>
> - protection from adding, deleting, and changing members
> - guards against reusing the same name twice
> - possible to have properties and members with the same name (i.e. "value"
> and "name")

In current Python this is true. But if we would go down the route of
PEP 560 (which I just found, I wasn't involved in its discussion),
then we could just add all the needed functionality to classes.

I would do it slightly different than proposed in PEP 560:
classmethods are very similar to methods on a metaclass. They are just
not called by the special method machinery. I propose that the
following is possible:

>>> class Spam:
 ...   @classmethod
 ...   def __getitem__(self, item):
 ...   return "Ham"

>>> Spam[3]
Ham

this should solve most of your usecases.

When thinking about how an automatic metaclass combiner would look
like, I realized that it should ideally just reproduce the class mro,
just with metaclasses. So if a class has an mro of [A, B, C, object],
its metaclass should have an mro of unique_everseen([type(A), type(B),
type(C), type]). But in this case, why add this layer at all? Just
give the class the ability to do everything a metaclass could do,
using mechanisms like @classmethod, and we're done.

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)

[Martin Teichmann]

> Metaclasses currently tend to serve two distinct purposes:
>
> 1. Actually altering the runtime behaviour of a class and its children in
> non-standard ways (e.g. enums, ABCs, ORMs)
> 2. Boilerplate reduction in class definitions, reducing the amount of code
> you need to write as the author of that class
>
> Nobody has a problem with using metaclasses for the first purpose - that's
> what they're for.

I am that nobody. The examples you give would be much nicer solved
with decorators. Especially for ABCs it would be much better, because
the fact that a class is an ABC is explicitly not inherited - its
entire reason of existence is to be inherited with all the
abstractness going away. And yet, currently the concrete class will
still inherit from ABC. The following way of writing ABCs looks much
nicer to me:

@abstractclass
class Spam:
@abstractmethod
def ham(self):
 ...

The same holds for enums. Inheriting from enums is possible, but
weird, given that you cannot add new enums to it. So, especially when
comparing to the dataclasses, the following looks appealing to me:

@enum
class Breakfast:
spam = 0
ham = 1

I'm not an expert on ORMs, but they seem to me a lot like data classes
in the context of this discussion.

I am aware that currently some things are easier done with
metaclasses. But given that decorators can create an entirely new
class if they like, they have all the power to do what they want, and
even in a way much easier understood by people.

As an example, here the autoslot decorator:

def autoslot(cls):
"""turn all class variables into slots"""
cls.__slots__ = list(cls.__dict__)
return type(cls.__name__, cls.__bases__, class.__dict__)

So I am personally more and more leaning towards a metaclass-free future.

Cheers

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Martin Teichmann]

Hi list,

first, a big thanks to the authors of PEP 557! Great idea!

For me, the dataclasses were a typical example for inheritance, to be
more precise, for metaclasses. I was astonished to see them
implemented using decorators, and I was not the only one, citing
Guido:

> I think it would be useful to write 1-2 sentences about the problem with
> inheritance -- in that case you pretty much have to use a metaclass, and the
> use of a metaclass makes life harder for people who want to use their own
> metaclass (since metaclasses don't combine without some manual
> intervention).

Python is at a weird point here. At about every new release of Python,
a new idea shows up that could be easily solved using metaclasses, yet
every time we hesitate to use them, because of said necessary manual
intervention for metaclass combination.

So I think we have two options now: We could deprecate metaclasses,
going down routes like PEP 487's __init_subclass__. Unfortunately, for
data classes __init_subclass__ it is too late in the class creation
process for it to influence the __slots__ mechanism. A
__new_subclass__, that acts earlier, could do the job, but to me that
simply sounds like reinventing the wheel of metaclasses.

The other option would be to simply make metaclasses work properly. We
would just have to define a way to automatically combine metaclasses.
Guido once mention once (here:
https://mail.python.org/pipermail/python-dev/2017-June/148501.html)
that he left out automatic synthesis of combined metaclasses on
purpose, but given that this seems to be a major problem, I think it
is about time to overthink this decision.

So I propose to add such an automatic synthesis. My idea is that a
metaclass author can define the __or__ and __ror__ methods for
automatic metaclass synthesis. Then if a class C inherits from two
classes A and B with metaclasses MetaA and MetaB, the metaclass would
be MetaA | MetaB.

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Evil reference cycles caused Exception.__traceback__

[Martin (gzlist) via Python-Dev]

Thanks for working on this and writing up the details Victor.

For those confused about why this matters, routinely having every
object in your application participating in one (or more) giant
reference cycles makes reasoning about and fixing resource issues very
difficult.

For instance, a while back I made some changes in Bazaar so that each
TestCase instance was dereferenced after being run:

<https://bugs.launchpad.net/bzr/+bug/613247>

Which mattered when running ~25000 tests, to keep peak memory usage
sane. Also there were various other object lifetime issues, and
tracking down which specific test failed to join a thread, or close a
file on Windows was much simpler after making sure cleanup actually
happened at the time a test was deleted.

Keeping more stuff alive for longer periods also makes peak memory
requirements higher, and the gc has to do more work each time.

On 18/09/2017, Victor Stinner <victor.stin...@gmail.com> wrote:
>
> Ideally, CPython 3.x should never create reference cycles. Removing
> Exception.__traceback__ is the obvious "fix" for the issue. But I
> expect that slowly, a lot of code started to rely on the attribute,
> maybe even for good reasons :-)
>
> A more practical solution would be to log a warning. Maybe the garbage
> collector can emit a warning if it detects an exception part of a
> reference cycle? Or maybe detect frames?

Logging a warning is unlikely to be practical. I had an optional test
flag that complained about reference cycles, and it produced a lot of
output.

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Evil reference cycles caused Exception.__traceback__

[Martin Panter]

On 18 September 2017 at 09:31, Victor Stinner  wrote:
> Last years, I fixed many reference cycles in various parts of the
> Python 3 standard library. Sometimes, it takes years to become aware
> of the reference cycle and finally fix it.
>
> For example, recently, I worked on fixing all "dangling threads"
> leaked by tests of the Python test suite, and I found and fixed many
> reference cycles which probably existed since Python 3 was created
> (forked from Python 2)

My instinct is to suggest to make the “dangling threads” test tolerate
Thread objects that are no longer running. Last time I looked, it
tests a list of weak references to Thread objects. But maybe it should
check “Thread.is_alive” or use “threading.enumerate”.

> . . .
>
> Ideally, CPython 3.x should never create reference cycles. Removing
> Exception.__traceback__ is the obvious "fix" for the issue. But I
> expect that slowly, a lot of code started to rely on the attribute,
> maybe even for good reasons :-)
>
> A more practical solution would be to log a warning. Maybe the garbage
> collector can emit a warning if it detects an exception part of a
> reference cycle? Or maybe detect frames?

The “gc” module can do some of this already. In the past (when __del__
used to prevent garbage collection) I used DEBUG_SAVEALL to and
graphed the result to figure out where code was creating reference
cycles. Or maybe if you turned on DEBUG_COLLECTABLE and filtered the
objects printed out, or played with the new “gc.callbacks” API.

> If the GC cannot do it, maybe we might use a debug thread (enabled
> manually) which checks manually if an exception is part of a reference
> cycle using gc.get_objects(): check if an exception remains alive
> longer than X seconds? I had the same idea for asyncio, to detect
> reference cycles or if a task is never "awaited", but I never
> implemented the idea.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] socketserver ForkingMixin waiting for child processes

[Martin Panter]

> On Fri, Aug 11, 2017 at 6:46 AM Victor Stinner 
> wrote:
>> => http://bugs.python.org/issue31151
>>
>> I changed the code to call waitpid() in blocking mode on each child
>> process on server_close(), to ensure that all children completed when
>> on server close:
>>
>> https://github.com/python/cpython/commit/aa8ec34ad52bb3b274ce91169e1bc4a598655049
>>
>> After pushing my change, I'm not sure anymore if it's a good idea.
>> There is a risk that server_close() blocks if a child is stuck on a
>> socket recv() or send() for some reasons.

I agree that this could be an unwanted change in behaviour. For
example, a web browser could be holding a HTTP 1.1 persistent
connection open.

>> Should we relax the code by waiting a few seconds (problem: hardcoded
>> timeouts are always a bad idea), or *terminate* processes (SIGKILL on
>> UNIX) if they don't complete fast enough?

It does seem reasonable to have an option to clean up background
forks, whether by blocking, or terminating them immediately.

On 11 August 2017 at 22:34, Ryan Smith-Roberts  wrote:
> Since ThreadingMixIn also leaks threads,
> server_close() could grow a timeout flag (following the socket module
> timeout convention) and maybe a terminate boolean. ThreadingMixIn could then
> also be fixed.

You could do a blocking join on each background thread. But I suspect
there is no perfect way to terminate the threads without waiting.
Using ThreadingMixIn.daemon_threads and exiting the interpreter might
have to be good enough.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale

[Martin (gzlist) via Python-Dev]

Thanks for replying to my points!

On 12/06/2017, Nick Coghlan <ncogh...@gmail.com> wrote:
>
> `PYTHONIOENCODING=:strict` remains the preferred way of forcing strict
> encoding checks on the standard streams, regardless of locale.

Then the user of my script has to care that it's written in Python and
set that specifically in their crontab or so on...

> In addition to providing a reliable escape hatch with no other
> potentially unwanted side effects (for when folks actually want the
> current behaviour), the entry for the off switch in the CLI usage docs
> also provides us with a convenient place to document the *default*
> behaviour.

The documentation aspect is an interesting consideration.

Having thought about it a bit more, my preferred option is having the
disable be if either LC_ALL or LC_CTYPE vars are exactly 'C', then
don't override. Otherwise (including for LANG=C), force C.UTF-8. The
CLI usage docs could have a LC_CTYPE entry that goes into details of
why.

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handle errors in cleanup code

[Martin (gzlist) via Python-Dev]

On 12/06/2017, Serhiy Storchaka <storch...@gmail.com> wrote:
>
> But if an error is raised when execute undo_something(), it replaces the
> original exception which become chaining as the __context__ attribute.
> The problem is that this can change the type of the exception. If
> do_something_other() raises SystemExit and undo_something() raises
> KeyError, the final exception has type KeyError.

For Python 2.7, I've used the following idiom, which always masks
errors from the undo:

  do_something()
  try:
  do_something_other()
  except:
  try:
  undo_something()
  finally:
  raise

Unfortunately, that breaks down on Python 3, as the new exception
propogates with the original as context..

> Does it mean that we should rewrite every chunk of code similar to the
> above? And if such cumbersome code is necessary and become common, maybe
> it needs syntax support in the language? Or changing the semantic of
> exceptions raised in error handlers and finally blocks?

What I want is a way to chain exceptions in the other direction,
raising the original error, but attaching a related one. Unfortunately
neither __cause__ or __context__ really help.

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale

[Martin (gzlist) via Python-Dev]

On 09/05/2017, Nick Coghlan <ncogh...@gmail.com> wrote:
>
> Enough changes have accumulated in PEP 538 since the start of the
> previous thread that it seems sensible to me to start a new thread
> specifically covering the current design (which aims to address all
> the concerns raised in the previous thread).
>
> I haven't requoted the PEP in full since it's so long, but will
> instead refer readers to the web version:
> https://www.python.org/dev/peps/pep-0538/

I did try to follow along via the mailing list threads, and have now
read over the PEP again. Responding now as I'm actually touching code
relevent to this again.

Broadly the proposal looks good to me. It does help one of the two
cases I care about, and does no serious harm.

For a command line Python script, making sure Python itself uses UTF-8
for the C locale is sufficient, and setting LC_CTYPE so spawned
processes that aren't Python have a chance at doing the right thing
too is a reasonable upgrade. This is probably good enough to drop one
hack[1] rather than porting it to Python 3.

For hosted Python code this does nothing (apart from print to stderr),
so mod_wsgi for instance is still going to need the same kind dance to
get users to set LANG as configuration themselves. Ideally this PEP
would have a C api or something so I could file bugs to make it just
do the right thing.

A few notes on specifics, I'll try not to stray too much into choices
already made.

The PEP doesn't persuade me that Py_Initialize actually is too late to
switch *specifically* from ascii to utf-8. Any preceeding operations
that operate on unicode would have been a safe subset. There might be
issues with other internals, or surrogateescape, or it's just a pain?

I don't like the side effect of changing the standard stream error
handler to surrogateescape if LANG=C.UTF-8 is actually set. Obviously
bad data vs exception is a trade off anyway, but means to get a Python
script that will always output valid data or exit, you have to set an
arbitrary language like en_US. Yes, that's true of the change as
implemented in 3.5 anyway.

Not setting LANG and only setting LC_CTYPE seems fine. Obviously,
things can go wrong based on odd behaviours of spawned processes, but
it works for the normal idioms.

I'm not sold on adding the PYTHONCOERCECLOCALE runtime configuration.
All it really does is turn off stderr kipple if you must use the C
locale for other reasons? Anyone with the ability to set that variable
could just set LANG instead. I was going to suggest just documenting
LC_ALL=C as the override instead of adding a python specific variable,
but note looking around that Debian discourage that[3].

That's all, though I will also grumble a bit about how long the PEP is.

Martin


[1] Override Py_FileSystemDefaultEncoding to utf-8 from ascii for the bzr script
<https://code.launchpad.net/~gz/bzr/filesystem_default_encoding_794353/+merge/85170>
[2] WSGIDaemonProcess lang and locale options
<https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html>
[3] "Using LC_ALL is strongly discouraged as it overrides everything"
<https://wiki.debian.org/Locale#Configuration>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Snap Python for simple distribution across multiple Linux distros

[Martin Wimpress]

Hi,

On 23/05/17 17:54, Guido van Rossum wrote:

I think the I inevitable conclusion is"thanks, but no thanks."


Can I ask why this the inevitable conclusion? The Python Foundation make 
packages for Windows and macOS, why not snaps for Linux?


On May 23, 2017 8:05 AM, "Martin Wimpress" 
<martin.wimpr...@canonical.com <mailto:martin.wimpr...@canonical.com>> 
wrote:


Hi,

On 23/05/17 00:56, Matt Billenstein wrote:

On Tue, May 16, 2017 at 11:31:42AM +0100, Martin Wimpress wrote:

Is there someone here who'd be interested in doing the same
for Python?


Do snaps support Windows and/or MacOS?


You can't install snaps on Windows or macOS. Snaps are currently
targeting various Linux distros, you can create Linux snaps on
Windows using WSL and we're working on improving the tooling for
snap creation on macOS and Windows.

m


    -- 
--

Regards, Martin.
___
Python-Dev mailing list
Python-Dev@python.org <mailto:Python-Dev@python.org>
https://mail.python.org/mailman/listinfo/python-dev
<https://mail.python.org/mailman/listinfo/python-dev>
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org 
<https://mail.python.org/mailman/options/python-dev/guido%40python.org>



--
--
Regards, Martin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Snap Python for simple distribution across multiple Linux distros

[Martin Wimpress]

Hi,

On 23/05/17 00:56, Matt Billenstein wrote:

On Tue, May 16, 2017 at 11:31:42AM +0100, Martin Wimpress wrote:

Is there someone here who'd be interested in doing the same for Python?


Do snaps support Windows and/or MacOS?


You can't install snaps on Windows or macOS. Snaps are currently 
targeting various Linux distros, you can create Linux snaps on Windows 
using WSL and we're working on improving the tooling for snap creation 
on macOS and Windows.



m



--
--
Regards, Martin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Snap Python for simple distribution across multiple Linux distros

[Martin Wimpress]

On 23/05/17 09:27, Petr Viktorin wrote:

On 05/23/2017 01:34 AM, Brett Cannon wrote:


On Tue, 16 May 2017 at 08:08 Martin Wimpress 
<martin.wimpr...@canonical.com <mailto:martin.wimpr...@canonical.com>> 
wrote:


Hi all,

I work at Canonical as part of the engineering team developing Ubuntu
and Snapcraft [1] and I'm a long time Python fan :-)

We've created snaps, a platform that enables projects to directly
control delivery of software updates to users. This video of a
lightning talk by dlang developers at DConf2017 [2] shows how they've
made good use of snaps to distribute their compiler. They found the
release channels particularly useful so their users can track a
specific release.

Is there someone here who'd be interested in doing the same for 
Python?



So the problem with adding Snap is it's yet one more thing for us to 
create at release time. And if we do this for Snap are there 
competitors from e.g. Fedora that people would want supported?


Not Fedora per se, Flatpak [0] (formerly `xdg-app`) is distro-agnostic [1].


Snaps are supported on Fedora 24, 25 and 26 [1] as well as other distros 
[2].





[0] http://flatpak.org/
[1] http://flatpak.org/faq.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/martin.wimpress%40canonical.com 



[1] 
https://insights.ubuntu.com/2017/04/11/snap-support-lands-in-fedora-24-25-26/

[2] https://snapcraft.io/docs/core/install

--
Regards, Martin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Snap Python for simple distribution across multiple Linux distros

[Martin Wimpress]

Hi all,

I work at Canonical as part of the engineering team developing Ubuntu
and Snapcraft [1] and I'm a long time Python fan :-)

We've created snaps, a platform that enables projects to directly
control delivery of software updates to users. This video of a
lightning talk by dlang developers at DConf2017 [2] shows how they've
made good use of snaps to distribute their compiler. They found the
release channels particularly useful so their users can track a
specific release.

Is there someone here who'd be interested in doing the same for Python?

[1] https://snapcraft.io/
[2] https://www.youtube.com/watch?v=M-bDzr4gYUU
[3] https://snapcraft.io/docs/core/install
[4] https://build.snapcraft.io/

-- 
Regards, Martin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Escaping docs markup in NEWS entries?

[Martin Panter]

On 1 May 2017 at 06:37, Nick Coghlan  wrote:
> Hi folks,
>
> I'm trying to write a NEWS entry that explains that the
> ":func:`bytes`" cross-references have changed to refer to the type
> descriptions by default (matching other builtin container types), so
> you now need to use ``:ref:`func-bytes`" to refer to the old target in
> the list of builtin functions (if you really want that for some
> reason).
>
> Unfortunately, my first two attempts both cause warnings in "make
> suspicious" with the following output:

What is the full output? Usually it includes instructions to add false
positives to Doc/tools/susp-ignored.csv; maybe that is all you have to
do?

> 
> WARNING: [whatsnew/changelog:986] ":func" found in "\:func\:\`bytes\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:func\:\`bytes\`"
> WARNING: [whatsnew/changelog:986] ":func" found in "\:func\:\`bytearray\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:func\:\`bytearray\`"
> WARNING: [whatsnew/changelog:986] ":ref" found in "\:ref\:\`func-bytes\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:ref\:\`func-bytes\`"
> WARNING: [whatsnew/changelog:986] ":ref" found in "\:ref\:\`func-bytes\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:ref\:\`func-bytes\`"
> 
>
> My first attempt just escaped the nested backticks:
>
> - Issue #30052: the link targets for ``:func:\`bytes\``` and
>   ``:func:\`bytearray\``` are now their respective type definitions, rather
>   than the corresponding builtin function entries. Use ``:ref:\`func-bytes\```
>   and ``:ref:\`func-bytes\``` to reference the latter.
>
> My second attempt escaped the colons as well:
>
> - Issue #30052: the link targets for ``\:func\:\`bytes\``` and
>   ``\:func\:\`bytearray\``` are now their respective type definitions, rather
>   than the corresponding builtin function entries. Use 
> ``\:ref\:\`func-bytes\```
>   and ``\:ref\:\`func-bytes\``` to reference the latter.
>
> My fallback plan is to just include the unescaped markup in the NEWS
> entry (rather than trying to make it readable even in rendered form),
> but I figured I'd ask for advice here first.

I thought the NEWS file was mainly regarded as plain text, so it would
be important to avoid ugly RST markup like backslashes if possible.

I am no RST expert, but perhaps it would be clearer to a human RST
parser if you added a space among the last three backticks, where the
underscore is in ``:func:`bytes`_``.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Martin Panter]

On 31 March 2017 at 05:22, Nathaniel Smith  wrote:
>>> On Mon, Mar 20, 2017 at 11:30 AM, Oleg Nesterov  wrote:
>>> > [Aborting "yield" in a "for" loop leaves a sub-generator open, but 
>>> > aborting "yield from" cleans up the sub-generator]
>
> In any case the short answer to your original question is that PEP 342
> says that generator finalization calls the generator's close() method,
> which throws a GeneratorExit into the generator, and PEP 380 says that
> as a special case, when a GeneratorExit is thrown into a yield from,
> then this is propagated by calling .close() on the yielded-from
> iterator (if such a method exists) and then re-raised in the original
> generator.

I think the Python documentation could be improved regarding this.
When I wrote the documentation for coroutine methods
,
I included details about the "close" and "throw" methods delegating to
inner iterators. I thought I was going to propose similar updates to
the generator documentation
,
but it seems I never got around to it. (In the mean time,

was added, to which this may also be relevant, but that is too
complicated for me.)

There is a parallel with another annoyance with Python generator
cleanup: . There are two ways you
can require generators to be used. With a simple generator, you can
partially iterate it and then throw it away without any special
cleaning up. But with more complex generators that "own" expensive
resources, it would be nice to produce a ResourceWarning I you forget
to clean them up.

With the "for / yield" case, the sub-generator is not cleaned up, so
if a resource-intensive sub-generator has to be cleaned up you have to
do it yourself. With "yield from", the cleanup is implicit and
unavoidable, which means you can't use it if you want keep the
sub-generator alive for later. But the automatic cleanup may be useful
in other cases.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Martin Panter]

On 30 March 2017 at 15:31, Nick Coghlan  wrote:
> On 29 March 2017 at 02:18, Paul Moore  wrote:
>> On 28 March 2017 at 12:24, Miro Hrončok  wrote:
>>> I'd like some clarification on what ABI compatibility we can expect.
>>>  * Should the ABI be stable across patch releases (so calling
>>> PySlice_AdjustIndices from an existing macro would be a bug)?
>>>  * Should the ABI be forward-compatible within a minor release (so modules
>>> built for 3.6.0 should be usable with 3.6.1, but not vice versa)?
>>>  * Or should we expect the ABI to change even across patch releases?
>>
>> Given that binary wheels are built against a specific minor version
>> (3.6, 3.5, ...) I would expect the ABI to be consistent over a minor
>> release. That would fit with my expectations of the compatibility
>> guarantees on patch releases.
>>
>> So I from what you describe, I'd consider this as a bug. Certainly, if
>> someone built a C extension as a wheel using Python 3.6.1, it would be
>> tagged as compatible with cp36, and pip would happily use it when
>> installing to a Python 3.6.0 system, where it would fail.
>
> Right, this is the main problem - while "build against the X.Y.0
> headers" is useful advice, it's not something we've ever explicitly
> stated, and it's not something we can reasonably expect all providers
> of pre-built binary modules to do.
>
> Instead, it makes sense to explicitly strengthen the ABI guarantees
> within CPython maintenance releases, and add some automated testing to
> avoid accidental changes and oversights (similar to the pending test
> to ensure magic number stability for cached bytecode files)

There's a website that has nice reports of ABI compatibility of
various packages and might useful for testing. It shows up the added
PySlice_AdjustIndices function in 3.6.1, along with PySlice_Unpack
(and some changes to internal names, so probably not such a problem).
https://abi-laboratory.pro/tracker/compat_report/python/3.6.0/3.6.1/496a4/abi_compat_report.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue with _thread.interrupt_main (29926)

[Martin Panter]

On 29 March 2017 at 00:40, Terry Reedy  wrote:
> [. . .] Eryk Sun suggested a patch for Windows, (and
> the possibility of using pthread_kill).  Can you possibly do one for *nix?
> This is out of my ballpark, but the bug (relative to console behavior) is a
> nuisance.

I'll try to find time, but no promises.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue with _thread.interrupt_main (29926)

[Martin Panter]

On 28 March 2017 at 03:11, Steven D'Aprano  wrote:
> On Mon, Mar 27, 2017 at 10:33:44PM -0400, Terry Reedy wrote:
>> https://bugs.python.org/issue29926 was opened as an IDLE issue, which
>> means that most watching the new issues list would ignore it.  But I
>> think it is an issue with _thread.interrupt_main (which IDLE calls in
>> respond to ^C) not interrupting time.sleep(n) in main thread*.  I tested
>> on Windows, don't know yet about OP.  Since there is no Expert's Index
>> listing for _thread (or threading), I am asking here for someone who
>> knows anything to take a look.
>>
>> *
>> >>> time.sleep(10)
>> 
>> 
>> <... remainder of 10 seconds pass>
>> KeyboardInterrupt
>
>
> I get similar behaviour under Linux. I don't have the debug print, but
> the KeyboardInterrupt doesn't interrupt the sleep until the 10 seconds
> are up.

Looking at the implementation, _thread.interrupt_main just calls
PyErr_SetInterrupt. It doesn’t appear to send a signal. I played with
“strace” and couldn’t see any evidence of a signal. I guess it just
sets a flag that will be polled. To actually interrupt the “sleep”
call, you might need to use “pthread_kill” or similar (at least on
Unix).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Regarding writing tests for module tabnanny

[Martin Panter]

On 13 March 2017 at 11:56, Jaysinh Shukla  wrote:
> Respected Members,
>
> I identified the standard module 'tabnanny' is having 16.66% of code
> coverage (Source:
> https://codecov.io/gh/python/cpython/src/master/Lib/tabnanny.py). I am
> interested to write tests for this module. Before starting, I would like to
> get help from any core developer on this.
>
>
> 1. Is community expecting to have tests for this module?
> 2. Any module specific guidelines?
>
> I am waiting for green signal from any core developer. Thanks!

Try the people involved in the existing patches at
 and
.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory)

[Martin Panter]

On 24 February 2017 at 07:51, Benjamin Peterson  wrote:
> As for this, particular issue, we should determine if there's a tracker
> issue yet and continue discussion there.

That would be .
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] adding threaded tests to the test suite

[Martin Panter]

> Le dim. 22 janv. 2017 à 21:04, Ethan Furman  a écrit :
>> Question:  I need to add a threaded test to the enum test module [1] -- is
>> there anything extra I
>> need to worry about besides the test itself?  Setting or resetting or
>> using a tool library, etc?
>>
>> threads = []
>> for i in range(8):
>> threads.append(threading.Thread(target=cycle_enum))
>> for t in threads:
>> t.start()
>> for t in threads:
>> t.join()

On 22 January 2017 at 20:17, Victor Stinner  wrote:
> There is @support.reap_thread which can help.

As I understand, @reap_threads basically does a join() on each
background thread, with a total timeout of 1 s. So since your test is
unlikely to fail between starting threads and joining them, I don’t
think you need to use @reap_threads.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Update the porting HOWTO

[Martin Panter]

On 17 December 2016 at 20:39, brett.cannon  wrote:
> https://hg.python.org/cpython/rev/287d4290b1b4
> changeset:   105714:287d4290b1b4
> branch:  2.7
> parent:  105677:eb02db65e148
> user:Brett Cannon 
> date:Sat Dec 17 12:38:54 2016 -0800
> summary:
>   Update the porting HOWTO
>
> diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst
> --- a/Doc/howto/pyporting.rst
> +++ b/Doc/howto/pyporting.rst
> . . .
>  Have good test coverage
>  ---
>
> @@ -106,10 +107,11 @@
>  thumb is that if you want to be confident enough in your test suite that any
>  failures that appear after having tools rewrite your code are actual bugs in 
> the
>  tools and not in your code. If you want a number to aim for, try to get over 
> 80%
> -coverage (and don't feel bad if you can't easily get past 90%). If you
> +coverage (and don't feel bad if you can't easily get passed 90%). If you
>  don't already have a tool to measure test coverage then coverage.py_ is
>  recommended.

Hi Brett, why did you make the above change (get past → get passed)?
To me, “get past 90%” means achieving over 90%, but “you can’t get
passed 90%” would mean that 90% cannot be given (passed) to you. The
original made more sense. Another option would be “get over 90%”,
consistent with the previous sentence.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implementation difference of audioop.lin2lin in Python2 and Python3

[Martin Panter]

On 13 December 2016 at 13:37, MRAB  wrote:
> On 2016-12-13 11:31, KH Luke Kim wrote:
>>
>> Hello,
>> recently there had been some issues in audioread and librosa that 3-byte
>> samples can be loaded in Python 3 but 2.
>>
>> The documentation says that the audioop.lin2lin function in Python 3
>> support 1-, 2-, 3-, 4-byte samples but only 1-, 2-, 4-byte samples in
>> Python 2.
>>
>> I wonder why 3-byte support is not implemented in Python 2. If there is
>> any previous thread or history regarding this issue, could you refer it
>> to me?
>>
> The Python docs say that support for 3-byte (24-bit) samples was added in
> Python 3.4, so anyone using a version before that one is out of luck!

The 3.4 reference leads you to What’s New, which leads to discussion
in the bug tracker:
https://docs.python.org/3/whatsnew/3.4.html#audioop
https://bugs.python.org/issue12866
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Revert unintended merge

[Martin Panter]

On 3 December 2016 at 22:15, Steve Dower  wrote:
> On 03Dec2016 1312, Serhiy Storchaka wrote:
>>
>> On 03.12.16 22:13, steve.dower wrote:
>>>
>>> https://hg.python.org/cpython/rev/a60767015bed
>>> changeset:   105436:a60767015bed
>>> user:Steve Dower 
>>> date:Sat Dec 03 12:12:23 2016 -0800
>>> summary:
>>>   Revert unintended merge
>>
>>
>> I suppose it should be reverted in the 3.6 branch too.
>
>
> Maybe, but I didn't make the change in the 3.6 branch, so I have no idea
> whether it is meant to be there or not. But it wasn't part of the change I
> made, so I didn't want to merge it forward.

This change comes from Matthias (Doko), and was originally only in the
3.5 branch:
https://hg.python.org/cpython/rev/14c80065c36e

I presume it was left unmerged until there is a branch for 3.6.1 to
merge into. So Steve, when you did your first 3.5 → 3.6 merge, you
merged Doko’s change into 3.6:

https://hg.python.org/cpython/rev/5171bd86a36f

> Someone who actually understands the implications of changing these files
> (config.guess, config.sub) can figure it out :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New problem accessing tracker with Firefox 50.0

[Martin Panter]

On 18/11/2016, Terry Reedy  wrote:
> Starting today, while reviewing the "Summary of Python tracker issues",
> I get the following for about half the clicks.
>
> """
> Secure Connection Failed
>
> An error occurred during a connection to bugs.python.org. A PKCS #11
> module returned CKR_DEVICE_ERROR, indicating that a problem has occurred
> with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR
>
>  The page you are trying to view cannot be shown because the
> authenticity of the received data could not be verified.
>  Please contact the website owners to inform them of this problem.
>
> [   Try Again   ]
> """
>
> So far, the Try Again button has always worked.
>
> I believe I upgraded Firefox (to 50.0) just last night, or possibly the
> night before.

Yesterday I started seeing this behaviour too. I have Firefox 43 and
haven’t updated any relevant libraries recently, so I assumed it was
the server’s fault :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (merge 3.5 -> default): Null merge.

[Martin Panter]

On 29 September 2016 at 04:26, Zachary Ware
 wrote:
> On Wed, Sep 28, 2016 at 10:04 PM, Terry Reedy  wrote:
>> On 9/28/2016 9:57 PM, terry.reedy wrote:
>>> https://hg.python.org/cpython/rev/02eb35b79af0
>>
>>
>> (2nd try) I mistakenly null merged from 3.5 to default.
>> Should a now do a proper null merge from 3.5 to 3.6 to default?
>> Should I revert this null merge?

FYI I committed some merges (04060fa4428d and ae0c983d3c65) which
should have fixed this all up.

> You aren't the only one who's missed 3.6 since it was branched :).  If
> there are changes in 3.5 that should not be in 3.6, you should go
> ahead and do a null merge from 3.5 -> 3.6 -> default.  If the changes
> in 3.5 are already in 3.6, I'd just leave it as is; it will clear up
> when somebody next merges something.

In this case, my automatic merge process gave conflicts and spooky
“ambiguous merge” warnings, so in this case I think it was good to
deal with it sooner rather than later.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (merge 3.5 -> default): Null merge.

[Martin Panter]

On 29 September 2016 at 03:04, Terry Reedy  wrote:
> On 9/28/2016 9:57 PM, terry.reedy wrote:
>> https://hg.python.org/cpython/rev/02eb35b79af0
>
> (2nd try) I mistakenly null merged from 3.5 to default.
> Should a now do a proper null merge from 3.5 to 3.6 to default?

Yes, I think 3.5 needs to be merged into 3.6, and the result needs to
be merged into default. I guess they are null merges because the
entries were already present in 3.6b1.

> Should I revert this null merge?

I don’t think there is much point in reverting a null merge, if there
are no actual changes to revert.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Fixes test_getargs2 to get the buildbots working again.

[Martin Panter]

On 12 September 2016 at 02:48, Steve Dower  wrote:
>>   Fixes test_getargs2 to get the buildbots working again.
>
> I'm not sure this is the fix we want to keep here, but it was sufficient to
> get the test going and unblock all the buildbots.
>
> I'm not entirely sure when the break appeared (essentially we seem to not be
> copying *args into a new tuple), but I'd guess it's to do with the fast
> calling improvements.

That seems to be everyone else’s guess too. See
https://bugs.python.org/issue28086 (bug about this failure)
https://bugs.python.org/issue27213 (bisected cause)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Martin Panter]

Thank you very much Benjamin.

On 7 September 2016 at 17:56, Benjamin Peterson  wrote:
> To conclude our discussion about using C99 features, I've updated PEP 7
> to allow the following features:
> - Standard integer types in  and 

Perhaps PEP 7 should clarify if the optional types like uint32_t are
allowed, or only C99 mandatory types like uint_fast32_t etc. I think
more people will be familiar with the fixed-width uint32_t etc. I know
they are mandatory in Posix, and presumably also Windows, so they may
be okay.

> - ``static inline`` functions
> - designated initializers
> - intermingled declarations
> - booleans
>
> I've been adding examples of these to 3.6 over the last few days to make
> sure the buildbots will like it.
>
> https://github.com/python/peps/commit/b6efe6e06fa70e8933440da26474a804fb3edb6e
>
> Enjoy.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Martin Panter]

On 5 September 2016 at 21:40, eryk sun  wrote:
> On Mon, Sep 5, 2016 at 7:54 PM, Steve Dower  wrote:
>> On 05Sep2016 1234, eryk sun wrote:
>>> It would probably be simpler to use UTF-16 in the main pipeline and
>>> implement Martin's suggestion to mix in a UTF-8 buffer. The UTF-16
>>> buffer could be renamed as "wbuffer", for expert use. However, if
>>> you're fully committed to transcoding in the raw layer, I'm certain
>>> that these problems can be addressed with small buffers and using
>>> Python's codec machinery for a flexible mix of "surrogatepass" and
>>> "replace" error handling.
>>
>> I don't think it actually makes things simpler. Having two buffers is
>> generally a bad idea unless they are perfectly synced, which would be
>> impossible here without data corruption (if you read half a utf-8 character
>> sequence and then read the wide buffer, do you get that character or not?).
>
> Martin's idea, as I understand it, is a UTF-8 buffer that reads from
> and writes to the text wrapper.

Yes, that was basically it. Though I had only thought as far as simple
encodings like ASCII, where one byte corresponds to one character. I
wonder if you really need UTF-8 support. Are the encoding values
currently encountered for Windows consoles all single-byte encodings
or are they more complicated?

> It necessarily consumes at least one
> character and buffers it to allow reading per byte. Likewise for
> writing, it buffers bytes until it can write a character to the text
> wrapper. ISTM, it has to look for incomplete lead-continuation byte
> sequences at the tail end, to hold them until the sequence is
> complete, at which time it either decodes to a valid character or the
> U+FFFD replacement character.

This buffering behaviour would be necessary for a multi-byte encodings
like UTF-8.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Where are the list and array.array implementations in CPython source?

[Martin Panter]

On 5 September 2016 at 23:45, Jonathan Goble  wrote:
> I'd like to study the CPython implementations of lists and array.array
> instances for a personal project of mine, but I've very unfamiliar
> with the Python source code as it pertains to internals like this.
> Which files would I need to look at to do this,

Built-in objects are usually in the Objects/ directory, with a
corresponding include file in the Include/ directory:
https://hg.python.org/cpython/file/default/Objects/listobject.c
https://hg.python.org/cpython/file/default/Include/listobject.h

Modules implemented in C are usually in the Modules/ directory:
https://hg.python.org/cpython/file/default/Modules/arraymodule.c

> and are there a few
> particular functions/structures I should pay attention to? I'm just
> looking for a brief pointer in the right direction here, not a full
> explanation of how it works -- I'll get that from studying the source
> code. :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Martin Panter]

On 5 September 2016 at 09:10, Paul Moore  wrote:
> On 5 September 2016 at 06:54, Steve Dower  wrote:
>> +Using the raw object with small buffers
>> +---
>> +
>> +Code that uses the raw IO object and attempts to read less than four 
>> characters
>> +will now receive an error. Because it's possible that any single character 
>> may
>> +require up to four bytes when represented in utf-8, requests must fail.
>
> I'm very concerned about this statement. It's clearly not true that
> the request *must* fail, as reading 1 byte from a UTF-8 enabled Linux
> console stream currently works (at least I believe it does). And there
> is code in the wild that works by doing a test that "there's input
> available" (using kbhit on Windows and select on Unix) and then doing
> read(1) to ensure a non-blocking read (the pyinvoke code I referenced
> earlier). If we're going to break this behaviour, I'd argue that we
> need to provide a working alternative.
>
> At a minimum, can the PEP include a recommended cross-platform means
> of implementing a non-blocking read from standard input, to replace
> the current approach? (If the recommendation is to read a larger
> 4-byte buffer and manage the process of retaining unused bytes
> yourself, then that's quite a major change to at least the code I'm
> thinking of in invoke, and I'm not sure read(4) guarantees that it
> *won't* block if only 1 byte is available without blocking...)

FWIW, on Linux and Unix in general, if select() or similar indicates
that some read data is available, calling raw read() with any buffer
size should return at least one byte, whatever is available, without
blocking. If the user has only typed one byte, read(4) would return
that one byte immediately.

But if you are using a BufferedReader (stdin.buffer rather than
stdin.buffer.raw), then this guarantee is off and read(4) will block
until it gets 4 bytes, or until EOF.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Martin Panter]

On 1 September 2016 at 23:28, Random832  wrote:
> On Thu, Sep 1, 2016, at 18:28, Steve Dower wrote:
>> This is a raw (bytes) IO class that requires text to be passed encoded
>> with utf-8, which will be decoded to utf-16-le and passed to the Windows 
>> APIs.
>> Similarly, bytes read from the class will be provided by the operating
>> system as utf-16-le and converted into utf-8 when returned to Python.
>
> What happens if a character is broken across a buffer boundary? e.g. if
> someone tries to read or write one byte at a time (you can't do a
> partial read of zero bytes, there's no way to distinguish that from an
> EOF.)
>
> Is there going to be a higher-level text I/O class that bypasses the
> UTF-8 encoding step when the underlying bytes stream is a console? What
> if we did that but left the encoding as mbcs? I.e. the console is text
> stream that can magically handle characters that aren't representable in
> its encoding. Note that if anything does os.read/write to the console's
> file descriptors, they're gonna get MBCS and there's nothing we can do
> about it.

Maybe it is too complicated and impractical, but I have imagined that
the sys.stdin/stdout/stderr could be custom TextIOBase objects. They
would not be wrappers or do much encoding (other than maybe newline
encoding). To solve the compatibility problems with code that uses
stdout.buffer or whatever, you could add a custom “buffer” object,
something like my AsciiBufferMixin class:
https://gist.github.com/vadmium/d1b07d771fbf4347683c005c40991c02

Just putting this idea out there, but maybe Steve’s UTF-8 encoding
solution is good enough.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Martin Panter]

On 1 September 2016 at 19:36, Ethan Furman  wrote:
> Deprecation of current "zero-initialised sequence" behaviour without removal
> 
>
> Currently, the ``bytes`` and ``bytearray`` constructors accept an integer
> argument and interpret it as meaning to create a zero-initialised sequence
> of the given size::
>
> >>> bytes(3)
> b'\x00\x00\x00'
> >>> bytearray(3)
> bytearray(b'\x00\x00\x00')
>
> This PEP proposes to deprecate that behaviour in Python 3.6, but to leave
> it in place for at least as long as Python 2.7 is supported, possibly
> indefinitely.

Can you clarify what “deprecate” means? Just add a note in the
documentation, or make calls trigger a DeprecationWarning as well?
Having bytearray(n) trigger a DeprecationWarning would be a minor
annoyance for code being compatible with Python 2 and 3, since
bytearray(n) is supported in Python 2.

> Addition of "getbyte" method to retrieve a single byte
> --
>
> This PEP proposes that ``bytes`` and ``bytearray`` gain the method
> ``getbyte``
> which will always return ``bytes``::

Should getbyte() handle negative indexes? E.g. getbyte(-1) returning
the last byte.

> Open Questions
> ==
>
> Do we add ``iterbytes`` to ``memoryview``, or modify
> ``memoryview.cast()`` to accept ``'s'`` as a single-byte interpretation?  Or
> do we ignore memory for now and add it later?

Apparently memoryview.cast('s') comes from Nick Coghlan:
.
However, since 3.5 (https://bugs.python.org/issue15944) you can call
cast("c") on most memoryviews, which I think already does what you
want:

>>> tuple(memoryview(b"ABC").cast("c"))
(b'A', b'B', b'C')
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Martin Panter]

> Le samedi 3 septembre 2016, Random832  a écrit :
>> On Fri, Sep 2, 2016, at 19:44, Ethan Furman wrote:
>> > The problem with only having `bchr` is that it doesn't help with
>> > `bytearray`;
>>
>> What is the use case for bytearray.fromord? Even in the rare case
>> someone needs it, why not bytearray(bchr(...))?

On 3 September 2016 at 08:47, Victor Stinner  wrote:
> Yes, this was my point: I don't think that we need a bytearray method to
> create a mutable string from a single byte.

I agree with the above. Having an easy way to turn an int into a bytes
object is good. But I think the built-in bchr() function on its own is
enough. Just like we have bytes object literals, but the closest we
have for a bytearray literal is bytearray(b". . .").
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Martin Panter]

On 2 September 2016 at 17:54, Koos Zevenhoven  wrote:
> On Thu, Sep 1, 2016 at 10:36 PM, Ethan Furman  wrote:
>> * Deprecate passing single integer values to ``bytes`` and ``bytearray``
>> * Add ``bytes.fromsize`` and ``bytearray.fromsize`` alternative
>> constructors
>> * Add ``bytes.fromord`` and ``bytearray.fromord`` alternative constructors
>> * Add ``bytes.getbyte`` and ``bytearray.getbyte`` byte retrieval methods
>> * Add ``bytes.iterbytes`` and ``bytearray.iterbytes`` alternative
>> iterators
>
> I wonder if from_something with an underscore is more consistent (according
> to a quick search perhaps yes).

That would not be too inconsistent with the sister constructor bytes.fromhex().
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Lib/http/client.py: could it return an OSError with the current response?

[Martin Panter]

On 30 August 2016 at 13:41, Ivo Bellin Salarin
 wrote:
> While using requests to tunnel a request via a proxy requiring user
> authentication, I have seen that httplib
> (https://hg.python.org/cpython/file/3.5/Lib/http/client.py#l831) raises the
> message returned by the proxy, along with its status code (407) without
> including the proxy response. This one could be very interesting to the
> consumer, since it could contain some useful headers (like the supported
> authentication schemes).

Here are some existing bug threads which may be relevant:

https://bugs.python.org/issue7291 (urllib.request support for handling
tunnel authentication)
https://bugs.python.org/issue24964 (get tunnel response header fields
in http.client)

> Would it be possible to change the http/client.py behavior in order to raise
> an exception including the whole response?

That would be one way, and might be good enough for getting a
Proxy-Authenticate value. Although there might be other ways that
also:

* Allow reading the body (e.g. HTML page) of the error response. IMO
an exception instance is the wrong place for this;
urllib.error.HTTPError is a bad example.
* Allow the tunnel response fields to be obtained even when the
request was successful

> If you don't see any problem with my proposal, how can I propose a pull
> request? :-)

Perhaps you can use one of the patches at one of the above bug reports
as a starting point. What you want seems to be a prerequisite for
Issue 7291 (urllib.request), so maybe we can discuss it there. Or open
a new bug to focus on the http.client-only aspect.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] socket.setsockopt() with optval=NULL

[Martin Panter]

On 21 August 2016 at 12:37, Christian Heimes  wrote:
> the socket.setsockopt(level, optname, value) method has two calling
> variants. When it is called with a buffer-like object as value, it calls
> the C API function setsockopt() with optval=buffer.buf and
> optlen=buffer.len. When value is an integer, setsockopt() packs it as
> int32 and sends it with optlen=4.
>
> ---
> # example.py
> import socket
> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,
> b'\x00\x00\x00\x00')
> sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> ---
>
> $ strace -e setsockopt ./python example.py
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
>
>
> For AF_ALG (Linux Kernel crypto) I need a way to call the C API function
> setsockopt() with optval=NULL and optlen as any arbitrary number. I have
> been playing with multiple ideas. So far I liked the idea of
> value=(None, int) most.
>
> setsockopt(socket.SOL_ALG, socket.ALG_SET_AEAD_AUTHSIZE, (None, taglen))

Would this new functionality be open-ended? What would happen if you did

sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, (None, 4))

On Linux, this seems to fail with errno = EFAULT, which would be fine.
But this does not seem to be specified by Posix, and I imagine other
platforms might crash. On the other hand, these sort of holes are
already present in the socket module. E.g. with MSG_TRUNC
 you can apparently get Python to
copy from unallocated memory.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Method signatures in the datetime module documentation

[Martin Panter]

On 4 August 2016 at 00:24, Alexander Belopolsky
 wrote:
> On Fri, Jul 29, 2016 at 12:55 PM, Alexander Belopolsky
>  wrote:
>>
>> How should I present the signature of the new replace method in the
>> documentation?
>
> Having not seeing any replies, let me make a proposal:
>
> datetime.replace(hour=self.hour, minute=self.minute, second=self.second,
> microsecond=self.microsecond, tzinfo=self.tzinfo, *, fold=self.fold)
>
> while not a valid method signature, this seems to correctly reflect
> what replace() does.

Does that mean replace() accepts most of its arguments as either
positional or keyword arguments? IMO when the documentation uses the
“name=default” notation, it implies keyword argument support, while
the [name] notation implies only supporting a positional argument. If
replace() actually supported these keywords all along, then I am happy
with this proposal.

There are a few bug reports that may be relevant:

https://bugs.python.org/issue13386: notation where there is no simple
default value
https://bugs.python.org/issue23738: notation for defaults for
positional-only parameters
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] stuck issue 26826

[Martin Panter]

On 03/08/2016, Marcos Dione  wrote:
> Hi pythonistas. A couple of moths ago I opened an issue in the bug
> tracker for adding a new syscall to the os module. It's based on new
> developments in the Linux kernel. Here's the link:
>
> https://bugs.python.org/issue26826

To give more context, this is about adding support for Linux’s
copy_file_range() system call.

> After two months and a half I managed to create a nice patch with
> unit tests and documentation (yay!), but then the issue went cold. I
> would like to know how to proceed to get the issue going again.

I thought the main problem remaining was getting concensus about
adding it, since a couple people mentioned waiting for glibc support.
I don’t have much personal opinion on this, but FWIW I don’t see much
disadvantage in adding it now.

I did mean to look over your latest patch, but that has drifted
towards the bottom of my list of things to do :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

Hi Sylvain,

thanks for the example, it's a great example to illustrate PEP 487 and
its design decisions.

> What it does is setting some class attributes that are required for certain
> types of descriptors to be able to initialize themselves.
>
> class MetaHasTraits(MetaHasDescriptors):
>
> """A metaclass for HasTraits."""
> def setup_class(cls, classdict):
> cls._trait_default_generators = {}
> super(MetaHasTraits, cls).setup_class(classdict)

While in a metaclass solution this does need that the metaclass needs
to execute code on the first class it is used for, in a PEP 487
solution this is not the case. A PEP 487 class HasTraits (no Meta
here) would not have Traits-descriptors itself, the classes inheriting
from it would have traits to be initialized. The PEP 487 HasTraits
takes the role of the metaclass.

A completely different problem shows up here. In your example,
HasTraits needs to initialize things on the class before the
Descriptors are run. This is not possible with PEP 487, where the
descriptors are initialized before __init_subclass__ is even called.

There are two ways to mitigate that problem:

- the first initialized descriptor could do the necessary initialization, or
- the descriptors are initialized from within __init_subclass__

At first sight, the first option seems hackish and the second option
looks much saner. Nonetheless PEP 487 proposes the second solution.
The rationale behind that is that people tend to forget to call
super(), and suddenly descriptors don't work anymore.

I realized later there is another benefit to this: if the first
initialized descriptor is doing the class initialization, often
__init_subclass__ isn't needed at all anymore, which means that those
kind of descriptors can be used on any class, without the need to tell
users that they have to inherit from a certain base class for the
descriptors to work. Only if some finalizing code needs to run after
the last descriptor is initialized one needs to write an
__init_subclass__. This is unavoidable as the last descriptor doesn't
know that it is the last.

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

Hello,

there has been quite some discussion on why PEP 487's
__init_subclass__  initializes subclasses, and not the class itself. I
think the most important details have been already thoroughly
discussed here.

One thing I was missing in the discussion is practical examples. I
have been using PEP 487-like metaclasses since several years now, and
I have never come across an application where it would have even been
nice to initialize the class itself. Also, while researching other
people's code when I wrote PEP 487, I couldn't find any such code
elsewhere, yet I found a lot of code where people took the wildest
measure to prevent a metaclass in doing its job on the first class it
is used for. One example is enum.EnumMeta, which contains code not to
make enum.Enum an enum (I do not try to propose that the enum module
should use PEP 487, it's way to complicated for that).

So once we have a practical example, we could start discussing how to
mitigate the problem.

Btw, everyone is still invited to review the patch for PEP 487 at
http://bugs.python.org/issue27366. Many thanks to Nick who already
reviewed, and also to Guido who left helpful comments!

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

Hi list, Hi Nick,

Sorry for my delayed response, it is summer here...

> However, phrasing it that way suggest that it's possible we *did* miss
> something in the PEP: we haven't specified whether or not __set_name__
> should be called when someone does someone does "cls.attr = descr".
> Given the name, I think we *should* call it in that case, and then the
> semantics during class creation are approximately what would happen if
> we actually built up the class attributes as:
>
> for attr, value in cls_ns.items():
> setattr(cls, attr, value)

That's a very good point and actually easy to solve: we would just
need to override type.__setattr__ to do so. Actually, it is already
overridden, so we just need to add code to type.__setattr__ to also
call __set_name__.

One could be of the other standpoint that in your above example it's
the duty of the caller of setattr to also call __set_name__. It would
be pretty easy to add a line in the loop that also calls
__set_owner__.

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should we fix these errors?

[Martin Panter]

FYI there is also a bug tracker report about this:
https://bugs.python.org/issue27587

On 23 July 2016 at 13:22, Christian Heimes  wrote:
> On 2016-07-22 16:36, Guido van Rossum wrote:
>> Somebody did some research and found some bugs in CPython (IIUC). The
>> published some questionable fragments. If there's a volunteer we could
>> probably easily fix these. (I know we already have occasional Coverity
>> scans and there are other tools too (anybody try lgtm yet?) But this
>> seems honest research (also Python leaves Ruby in the dust :-):
>>
>> http://www.viva64.com/en/b/0414/
>
> I had a closer look at the report. About half of the bugs, maybe more
> are not in the C code of CPython but in OpenSSL code. I really mean
> OpenSSL code, not _ssl.c and _hashopenssl.c. It's safe to assume that
> they forgot to exclude external dependencies.
>
> The issues in ASN1_PRINTABLE_type() [N2], BN_mask_bits() [N4 bn_lib.c,
> digest.c, evp_enc.c], dh_cms_set_peerkey() [N5, dh_ameth.c] and
> cms_env_set_version() [N6, cms_env.c] are all OpenSSL issues and should
> be reported to OpenSSL.
>
> Guido, did the company contact you or do you have Pavel Belikov's email
> address?

Perhaps you can contact him via the email address at
.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fun with ExitStack

[Martin Panter]

On 18 July 2016 at 23:40, Barry Warsaw  wrote:
> I was trying to debug a problem in some work code and I ran into some
> interesting oddities with contextlib.ExitStack and other context managers in
> Python 3.5.
>
> This program creates a temporary directory, and I wanted to give it a --keep
> flag to not automatically delete the tempdir at program exit.  I was using an
> ExitStack to manage a bunch of resources, and the temporary directory is the
> first thing pushed into the ExitStack.  At that point in the program, I check
> the value of --keep and if it's set, I use ExitStack.pop_all() to clear the
> stack, and thus, presumably, prevent the temporary directory from being
> deleted.
>
> There's this relevant quote in the contextlib documentation:
>
> """
> Each instance [of an ExitStack] maintains a stack of registered callbacks that
> are called in reverse order when the instance is closed (either explicitly or
> implicitly at the end of a with statement). Note that callbacks are not
> invoked implicitly when the context stack instance is garbage collected.
> """
>
> However if I didn't save the reference to the pop_all'd ExitStack, the tempdir
> would be immediately deleted.  If I did save a reference to the pop_all'd
> ExitStack, the tempdir would live until the saved reference went out of scope
> and got refcounted away.
>
> As best I can tell this happens because TemporaryDirectory.__init__() creates
> a weakref finalizer which ends up calling the _cleanup() function.  Although
> it's rather difficult to trace, it does appear that when the ExitStack is
> gc'd, this finalizer gets triggered (via weakref.detach()), thus causing the
> _cleanup() method to be called and the tmpdir to get deleted.

This seems to be missing from the documentation, but when you delete a
TemporaryDirectory instance without using a context manager or
cleanup(), it complains, and then cleans up anyway:

>>> d = TemporaryDirectory()
>>> del d
/usr/lib/python3.5/tempfile.py:797: ResourceWarning: Implicitly
cleaning up 
  _warnings.warn(warn_message, ResourceWarning)

Perhaps you will have to use the lower-level mkdtemp() function
instead if you want the option of making the “temporary” directory
long-lived.

> I "fix" this by
> doing:
>
> def __init__(self):
> tmpdir = TemporaryDirectory()
> self._tmpdir = (tmpdir.name if keep
> else self.resources.enter_context(tmpdir))
>
> There must be more to the story because when __init__() exits in the --keep
> case, tmpdir should have gotten refcounted away and the directory deleted, but
> it doesn't.  I haven't dug down deep enough to figure that out.
>
> Now, while I was debugging that behavior, I ran across more interesting bits.
> I put this in a file to drive some tests:
>
> --snip snip-
> with ExitStack() as resources:
> print('enter context')
> tmpdir = resources.enter_context(X())
> resources.pop_all()
> print('exit context')
> --snip snip-
>
> Let's say X is:
>
> class X:
> def __enter__(self):
> print('enter Foo')
> return self
>
> def __exit__(self, *args, **kws):
> print('exit Foo')
> return False
>
> the output is:
>
> enter context
> enter Foo
> exit context
>
> So far so good.  A fairly standard context manager class doesn't get its
> __exit__() called even when the program exits.  Let's try this:
>
> @contextmanager
> def X():
> print('enter bar')
> yield
> print('exit bar')
>
> still good:
>
> enter context
> enter bar
> exit context
>
> Let's modify X a little bit to be a more common idiom:
>
> @contextmanager
> def X():
> print('enter foo')
> try:
> yield
> finally:
> print('exit foo')
>
> enter context
> enter foo
> exit foo
> exit context
>
> Ah, the try-finally changes the behavior!  There's probably some documentation
> somewhere that defines how a generator gets finalized, and that triggers the
> finally clause, whereas in the previous example, nothing after the yield gets
> run.  I just can't find that anything that would describe the observed
> behavior.

I suspect the documentation doesn’t spell everything out, but my
understanding is that garbage collection of a generator instance
effectively calls its close() method, triggering any “finally” and
__exit__() handlers.

IMO, in some cases if a generator would execute these handlers and
gets garbage collected, it is a programming error because the
programmer should have explicitly called generator.close(). In these
cases, it would be nice to emit a ResourceWarning, just like
forgetting to close a file, or delete your temporay directory above.

But maybe there are other cases where there is no valid reason to emit
a warning. I have hesitated in suggesting this change in the past, but
I don’t remember why. One reason is it might be an annoyance with code
that also wants to handle non-generator iterators that don’t have a
close() method. In a 

Re: [Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

> This PEP is now accepted for inclusion in Python 3.6. Martin,
> congratulations!

Thank you very much! What a great news!

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __qualname__ exposed as a local variable: standard?

[Martin Teichmann]

Hi,

so I did quite some research on this topic. And what I found out is
that __qualname__ needs to exist in the namespace. Not necessarily
because it should be used, but because it may be modified.

The story goes as follows: the compiler sets the __qualname__ at the
beginning of the class body. Within the class body, it may be modified
as needed. Then type.__new__ takes it and uses it.

Now one could think that instead of setting the __qualname__ at the
beginning of the class body, we could do so at the end as to not
clutter the namespace, and only if the __qualname__ has been set in
the class body we would use the user-supplied version. But this is
forgetting __prepare__: unfortunately, we have no good way to find out
whether something has been set in a class body, because we have no
guarantee that the object returned by __prepare__ doesn't do something
weird, as autogenerating values for all requested keys.

> To Martin: it would be easier for people (even myself, who implemented
> this super() hack eons ago) to review your patch if you were able to
> explain the current and proposed behavior more precisely.

I tried to give some context on my issue
(http://bugs.python.org/issue23722). Hope that helps.

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

Hi Guido, Hi Nick, Hi list,

so I just updated PEP 487, you can find it here:
https://github.com/python/peps/pull/57 if it hasn't already been
merged. There are no substantial updates there, I only updated the
wording as suggested, and added some words about backwards
compatibility as hinted by Nick.

Greetings

Martin

2016-07-14 17:47 GMT+02:00 Guido van Rossum <gu...@python.org>:
> I just reviewed the changes you made, I like __set_name__(). I'll just
> wait for your next update, incorporating Nick's suggestions. Regarding
> who merges PRs to the PEPs repo, since you are the author the people
> who merge don't pass any judgment on the changes (unless it doesn't
> build cleanly or maybe if they see a typo). If you intend a PR as a
> base for discussion you can add a comment saying e.g. "Don't merge
> yet". If you call out @gvanrossum, GitHub will make sure I get a
> message about it.
>
> I think the substantial discussion about the PEP should remain here in
> python-dev; comments about typos, grammar and other minor editorial
> issues can go on GitHub. Hope this part of the process makes sense!
>
> On Thu, Jul 14, 2016 at 6:50 AM, Martin Teichmann
> <lkb.teichm...@gmail.com> wrote:
>> Hi Guido, Hi list,
>>
>> Thanks for the nice review! I applied followed up your ideas and put
>> it into a github pull request: https://github.com/python/peps/pull/53
>>
>> Soon we'll be working there, until then, some responses to your comments:
>>
>>> I wonder if this should be renamed to __set_name__ or something else
>>> that clarifies we're passing it the name of the attribute? The method
>>> name __set_owner__ made me assume this is about the owning object
>>> (which is often a useful term in other discussions about objects),
>>> whereas it is really about telling the descriptor the name of the
>>> attribute for which it applies.
>>
>> The name for this has been discussed a bit already, __set_owner__ was
>> Nick's idea, and indeed, the owner is also set. Technically,
>> __set_owner_and_name__ would be correct, but actually I like your idea
>> of __set_name__.
>>
>>> That (inheriting type from type, and object from object) is very
>>> confusing. Why not just define new classes e.g. NewType and NewObject
>>> here, since it's just pseudo code anyway?
>>
>> Actually, it's real code. If you drop those lines at the beginning of
>> the tests for the implementation (as I have done here:
>> https://github.com/tecki/cpython/blob/pep487b/Lib/test/test_subclassinit.py),
>> the test runs on older Pythons.
>>
>> But I see that my idea to formulate things here in Python was a bad
>> idea, I will put the explanation first and turn the code into
>> pseudo-code.
>>
>>>> def __init__(self, name, bases, ns, **kwargs):
>>>> super().__init__(name, bases, ns)
>>>
>>> What does this definition of __init__ add?
>>
>> It removes the keyword arguments. I describe that in prose a bit down.
>>
>>>> class object:
>>>> @classmethod
>>>> def __init_subclass__(cls):
>>>> pass
>>>>
>>>> class object(object, metaclass=type):
>>>> pass
>>>
>>> Eek! Too many things named object.
>>
>> Well, I had to do that to make the tests run... I'll take that out.
>>
>>>> In the new code, it is not ``__init__`` that complains about keyword 
>>>> arguments,
>>>> but ``__init_subclass__``, whose default implementation takes no 
>>>> arguments. In
>>>> a classical inheritance scheme using the method resolution order, each
>>>> ``__init_subclass__`` may take out it's keyword arguments until none are 
>>>> left,
>>>> which is checked by the default implementation of ``__init_subclass__``.
>>>
>>> I called this out previously, and I am still a bit uncomfortable with
>>> the backwards incompatibility here. But I believe what you describe
>>> here is the compromise proposed by Nick, and if that's the case I have
>>> peace with it.
>>
>> No, this is not Nick's compromise, this is my original. Nick just sent
>> another mail to this list where he goes a bit more into the details,
>> I'll respond to that about this topic.
>>
>> Greetings
>>
>> Martin
>>
>> P.S.: I just realized that my changes to the PEP were accepted by
>> someone else than Guido. I am a bit surprised about that, but I guess
>> this is how it works?
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

Hi Nick, hi list,

> It would be worth spelling out the end result of the new behaviour in
> the PEP to make sure it's what we want. Trying to reason about how
> that code works is difficult, but looking at some class definition
> scenarios and seeing how they behave with the old semantics and the
> new semantics should be relatively straightforward (and they can
> become test cases for the revised implementation).

I agree with that. All the examples that Nick gives should work the
same way as they used to. I'll turn them into tests. I hope it is fine
if the error messages change slightly, as long as the error type stays
the same?

Let's look at an example that won't work anymore if my proposal goes throught:

class MyType(type):
def __new__(cls, name, bases, namespace):
return super().__new__(cls, name=name, bases=bases, dict=namespace)

I guess this kind of code is pretty rare. Note that I need to call the
third parameter dict, as this is how it is named. Even if there is
code out there like that, it would be pretty easy to change.

Just in case someone wondered:

def __new__(cls, **kwargs):
return super().__new__(cls, **kwargs)

doesn't work now and won't work afterwards, as the interpreter calls
the metaclass with positional arguments.

That said, it would be possible, at the cost of quite some lines of
code, to make it fully backwards compatible. If the consensus is that
this is needed, I'll change the PEP and code accordingly.

My proposal also has the advantage that name, bases and dict may be
used as class keyword arguments. At least for name I see a usecase:

class MyMangledClass(BaseClass, name="Nice class name"):
 pass

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

Hi Guido, Hi list,

Thanks for the nice review! I applied followed up your ideas and put
it into a github pull request: https://github.com/python/peps/pull/53

Soon we'll be working there, until then, some responses to your comments:

> I wonder if this should be renamed to __set_name__ or something else
> that clarifies we're passing it the name of the attribute? The method
> name __set_owner__ made me assume this is about the owning object
> (which is often a useful term in other discussions about objects),
> whereas it is really about telling the descriptor the name of the
> attribute for which it applies.

The name for this has been discussed a bit already, __set_owner__ was
Nick's idea, and indeed, the owner is also set. Technically,
__set_owner_and_name__ would be correct, but actually I like your idea
of __set_name__.

> That (inheriting type from type, and object from object) is very
> confusing. Why not just define new classes e.g. NewType and NewObject
> here, since it's just pseudo code anyway?

Actually, it's real code. If you drop those lines at the beginning of
the tests for the implementation (as I have done here:
https://github.com/tecki/cpython/blob/pep487b/Lib/test/test_subclassinit.py),
the test runs on older Pythons.

But I see that my idea to formulate things here in Python was a bad
idea, I will put the explanation first and turn the code into
pseudo-code.

>> def __init__(self, name, bases, ns, **kwargs):
>> super().__init__(name, bases, ns)
>
> What does this definition of __init__ add?

It removes the keyword arguments. I describe that in prose a bit down.

>> class object:
>> @classmethod
>> def __init_subclass__(cls):
>> pass
>>
>> class object(object, metaclass=type):
>> pass
>
> Eek! Too many things named object.

Well, I had to do that to make the tests run... I'll take that out.

>> In the new code, it is not ``__init__`` that complains about keyword 
>> arguments,
>> but ``__init_subclass__``, whose default implementation takes no arguments. 
>> In
>> a classical inheritance scheme using the method resolution order, each
>> ``__init_subclass__`` may take out it's keyword arguments until none are 
>> left,
>> which is checked by the default implementation of ``__init_subclass__``.
>
> I called this out previously, and I am still a bit uncomfortable with
> the backwards incompatibility here. But I believe what you describe
> here is the compromise proposed by Nick, and if that's the case I have
> peace with it.

No, this is not Nick's compromise, this is my original. Nick just sent
another mail to this list where he goes a bit more into the details,
I'll respond to that about this topic.

Greetings

Martin

P.S.: I just realized that my changes to the PEP were accepted by
someone else than Guido. I am a bit surprised about that, but I guess
this is how it works?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of Python 3.6 PEPs?

[Martin Teichmann]

Hi,

> "PEP 487 -- Simpler customisation of class creation"
> https://www.python.org/dev/peps/pep-0487/
> => draft

I would like to get that into Python 3.6. It's already implemented,
including documentation and tests (http://bugs.python.org/issue27366).

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

Hi list,

another round for PEP 487, is there any chance it still makes it into
Python 3.6?

The PEP should be effectively done, I updated the examples in it,
given that I implemented the PEP I could actually test the examples,
so now they work.

The implementation is at http://bugs.python.org/issue27366, including
documentation and tests. Unfortunately nobody has reviewed the patch
yet.

The new version of the PEP is attached.

Greetings

Martin

PEP: 487
Title: Simpler customisation of class creation
Version: $Revision$
Last-Modified: $Date$
Author: Martin Teichmann <lkb.teichm...@gmail.com>,
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Feb-2015
Python-Version: 3.6
Post-History: 27-Feb-2015, 5-Feb-2016, 24-Jun-2016, 2-Jul-2016, 13-Jul-2016
Replaces: 422


Abstract


Currently, customising class creation requires the use of a custom metaclass.
This custom metaclass then persists for the entire lifecycle of the class,
creating the potential for spurious metaclass conflicts.

This PEP proposes to instead support a wide range of customisation
scenarios through a new ``__init_subclass__`` hook in the class body,
and a hook to initialize attributes.

The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power of Python's metaclass machinery.


Background
==

Metaclasses are a powerful tool to customize class creation. They have,
however, the problem that there is no automatic way to combine metaclasses.
If one wants to use two metaclasses for a class, a new metaclass combining
those two needs to be created, typically manually.

This need often occurs as a surprise to a user: inheriting from two base
classes coming from two different libraries suddenly raises the necessity
to manually create a combined metaclass, where typically one is not
interested in those details about the libraries at all. This becomes
even worse if one library starts to make use of a metaclass which it
has not done before. While the library itself continues to work perfectly,
suddenly every code combining those classes with classes from another library
fails.

Proposal


While there are many possible ways to use a metaclass, the vast majority
of use cases falls into just three categories: some initialization code
running after class creation, the initalization of descriptors and
keeping the order in which class attributes were defined.

The first two categories can easily be achieved by having simple hooks
into the class creation:

1. An ``__init_subclass__`` hook that initializes
   all subclasses of a given class.
2. upon class creation, a ``__set_owner__`` hook is called on all the
   attribute (descriptors) defined in the class, and

The third category is the topic of another PEP 520.

As an example, the first use case looks as follows::

   >>> class QuestBase:
   ...# this is implicitly a @classmethod
   ...def __init_subclass__(cls, swallow, **kwargs):
   ...cls.swallow = swallow
   ...super().__init_subclass__(**kwargs)

   >>> class Quest(QuestBase, swallow="african"):
   ...pass

   >>> Quest.swallow
   'african'

The base class ``object`` contains an empty ``__init_subclass__``
method which serves as an endpoint for cooperative multiple inheritance.
Note that this method has no keyword arguments, meaning that all
methods which are more specialized have to process all keyword
arguments.

This general proposal is not a new idea (it was first suggested for
inclusion in the language definition `more than 10 years ago`_, and a
similar mechanism has long been supported by `Zope's ExtensionClass`_),
but the situation has changed sufficiently in recent years that
the idea is worth reconsidering for inclusion.

The second part of the proposal adds an ``__set_owner__``
initializer for class attributes, especially if they are descriptors.
Descriptors are defined in the body of a
class, but they do not know anything about that class, they do not
even know the name they are accessed with. They do get to know their
owner once ``__get__`` is called, but still they do not know their
name. This is unfortunate, for example they cannot put their
associated value into their object's ``__dict__`` under their name,
since they do not know that name.  This problem has been solved many
times, and is one of the most important reasons to have a metaclass in
a library. While it would be easy to implement such a mechanism using
the first part of the proposal, it makes sense to have one solution
for this problem for everyone.

To give an example of its usage, imagine a descriptor representing weak
referenced values::

import weakref

class WeakAttribute:
def __get__(self, instance, owner):
return instance.__dict__[self.name]()

def __set__(self, instance, value):
insta

Re: [Python-Dev] __qualname__ exposed as a local variable: standard?

[Martin Teichmann]

Hi list,

> I noticed __qualname__ is exposed by locals() while defining a class. This
> is handy but I'm not sure about its status: is it standard or just an
> artifact of the current implementation? (btw, the pycodestyle linter -former
> pep8- rejects its usage). I was unable to find any reference to this
> behavior in PEP 3155 nor in the language reference.

I would like to underline the importance of this question, and give
some background, as it happens to resonate with my work on PEP 487.

The __qualname__ of a class is originally determined by the compiler.
One might now think that it would be easiest to simply set the
__qualname__ of a class once the class is created, but this is not as
easy as it sounds. The class is created by its metaclass, so possibly
by user code, which might create whatever it wants, including
something which is not even a class. So the decision had been taken to
sneak the __qualname__ through user code, and pass it to the
metaclasses __new__ method as part of the namespace, where it is
deleted from the namespace again. This has weird side effects, as the
namespace may be user code as well, leading to the funniest possible
abuses, too obscene to publish on a public mailing list.

A very different approach has been taken for super(). It has similar
problems: the zero argument version of super looks in the surrounding
scope for __class__ for the containing class. This does not exist yet
at the time of creation of the methods, so a PyCell is put into the
function's scope, which will later be filled. It is actually filled
with whatever the metaclasses __new__ returns, which may, as already
be said, anything (some sanity checks are done to avoid crashing the
interpreter).

I personally prefer the first way of doing things like for
__qualname__, even at the cost of adding things to the classes
namespace. It could be moved after the end of the class definition,
such that it doesn't show up while the class body is executed. We
might also rename it to __@qualname__, this way it cannot be accessed
by users in the class body, unless they look into locals().

This has the large advange that super() would work immediately after
the class has been defined, i.e. already in the __new__ of the
metaclass after it has called type.__new__.

All of this changes the behavior of the interpreter, but we are
talking about undocumented behavior.

The changes necessary to make super() work earlier are store in
http://bugs.python.org/issue23722

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] devguide: Star self as idlelib expert. Mark other 2 as inactive.

[Martin Panter]

On 12 July 2016 at 04:14, Martin Panter <vadmium...@gmail.com> wrote:
> On 12 July 2016 at 03:47, terry.reedy <python-check...@python.org> wrote:
>> https://hg.python.org/devguide/rev/cc1c0dd798e7
>
> Terry it looks like you accidentally added Christian back (undoing
> <https://hg.python.org/devguide/rev/cc1c0dd798e7>)

Sorry, you can ignore that, I see you already reverted it :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] devguide: Star self as idlelib expert. Mark other 2 as inactive.

[Martin Panter]

On 12 July 2016 at 03:47, terry.reedy  wrote:
> https://hg.python.org/devguide/rev/cc1c0dd798e7

Terry it looks like you accidentally added Christian back (undoing
)

> -xml.parsers.expat
> -xml.sax
> +xml.parsers.expat christian.heimes
> +xml.sax   christian.heimes
>  xml.sax.handler
>  xml.sax.saxutils
>  xml.sax.xmlreader
> @@ -319,7 +319,7 @@
>  bytecodebenjamin.peterson, pitrou, georg.brandl, yselivanov
>  context managersncoghlan
>  coverity scan   christian.heimes, brett.cannon, twouters
> -cryptographygregory.p.smith, dstufft
> +cryptographychristian.heimes, gregory.p.smith, dstufft
>  data formatsmark.dickinson, georg.brandl
>  databaselemburg
>  devguidencoghlan, eric.araujo, ezio.melotti, willingc
> @@ -341,7 +341,7 @@
>  georg.brandl
>  str.format  eric.smith
>  testing michael.foord, pitrou, ezio.melotti
> -test coverage   giampaolo.rodola
> +test coverage   giampaolo.rodola, christian.heimes
>  threads pitrou
>  time and dates  lemburg, belopolsky
>  unicode lemburg, ezio.melotti, haypo, benjamin.peterson, pitrou
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

> This is an area of exceeding subtlety (and also not very well
> documented/specified, probably). I'd worry that changing anything here
> might break some code. When a metaclass overrides neither __init__ nor
> __new__, keyword args will not work because type.__init__ forbids
> them. However when a metaclass overrides them and calls them using
> super(), it's quite possible that someone ended up calling
> super().__init__() with three positional args but super().__new__()
> with keyword args, since the call sites are distinct (in the overrides
> for __init__ and __new__ respectively).
>
> What's your argument for changing this, apart from a desire for more 
> regularity?

The implementation gets much simpler if __new__ doesn't take keyword
arguments. It's simply that if it does, I have to filter out __new__'s
three arguments.
That's easily done in Python, unfortunately not so much in C.

So we have two options: either type.__new__ is limited to accepting positional
arguments only, possibly breaking some code, but which could be changed
easily. This leads to a pretty simple implementation: pass over
keyword arguments
to __init_subclass__, that's it.

The other option is: filter out name, bases and dict from the keyword arguments
If people think that backwards compatibility is that important, I
could do that. But
that just leaves quite some code in places where there is already a lot of
complicated code.

Nick proposed a compromise, just don't filter for name, bases and dict, and
pass them over to __init_subclass__. Then the default implementation of
__init_subclass__ must support those three keyword arguments and do
nothing with them.

I'm fine with all three solutions, although I have a preference for the first.
I think passing keyword arguments to type.__new__ is already really rare
and if it does exist, it's super easy to fix.

> I'm confused. In the above example it would seem that the keyword args
> {'a': 1, 'b': 2} are passed right on to super9).__init_subclass__().
> Do you mean that it ignores all keyword args? Or that it has no
> positional args? (Both of which would be consistent with the example.)

The example is just wrong. I'll fix it.

> Can you state exactly at which point during class initialization
> __init_class__() is called? (Surely by now, having implemented it, you
> know exactly where. :-)

Further down in the PEP I give the exact

> [This is as far as I got reviewing when the weekend activities
> interrupted me. In the light of ongoing discussion I'm posting this
> now -- I'll continue later.]

I hope you had a good weekend not thinking too much about
metaclasses...

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 487: Simpler customization of class creation

[Martin Teichmann]

Hi Guido,

sorry I missed your post...

>> One of the big issues that makes library authors reluctant to use
>> metaclasses
>> (even when they would be appropriate) is the risk of metaclass conflicts.
>
> Really? I've written and reviewed a lot of metaclasses and this has never
> worried me. The problem is limited to multiple inheritance, right? I worry a
> lot about MI being imposed on classes that weren't written with MI in mind,
> but I've never particularly worried about the special case of metaclasses.

Yes, the problem only arises with MI. Unfortunately, that's not
uncommon: if you want to implement an ABC with a class from a
framework which uses metaclasses, you have a metaclass conflict. So
then you start making MyFrameworkABCMeta-classes.

The worst is if you already have a framework with users out there. No
way you add  a metaclass to your class, however convenient it would
be. Because you never now if some user out there had gotten the idea
to implement an ABC with it. Sure, you could let your metaclass
inherit from ABCMeta, but is this really how it should be done?

(This has already been mentioned by others over at python-ideas:
https://mail.python.org/pipermail/python-ideas/2016-February/038506.html)

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

Hi Nick,

thanks for the nice review!

> I think making __init_subclass__ implicitly a class method is still
> the right thing to do if this proposal gets accepted, we'll just want
> to see if we can do something to tidy up that aspect of the
> documentation at the same time.

I could write some documentation, I just don't know where to put it.
I personally have no strong feelings whether __init_subclass__ should
be implicitly a @classmethod or not - but as the general consensus here
seemed to hint making it implicit is better, this is how I wrote it.

>> While implementing PEP 487 I realized that there is and oddity in the
>> type base class: type.__init__ forbids to use keyword arguments, even
>> for the usual three arguments it has (name, base and dict), while
>> type.__new__ allows for keyword arguments. As I plan to forward any
>> keyword arguments to the new __init_subclass__, I stumbled over that.
>> As I write in the PEP, I think it would be a good idea to forbid using
>> keyword arguments for type.__new__ as well. But if people think this
>> would be to big of a change, it would be possible to do it
>> differently.
>
> [some discussion cut out]
>
> I think the PEP could be accepted without cleaning this up, though -
> it would just mean __init_subclass__ would see the "name", "bases" and
> "dict" keys when someone attempted to use keyword arguments with the
> dynamic type creation APIs.

Yes, this would be possible, albeit a bit ugly. I'm not so sure whether
backwards compatibility is so important in this case. It is very
easy to change the code to the fully cleaned up version


Looking through old stuff I found http://bugs.python.org/issue23722,
which describes the following problem: at the time __init_subclass__ is
called, super() doesn't work yet for the new class. It does work for
__init_subclass__, because it is called on the base class, but not for
calls to other classmethods it does. This is a pity especially because
also the two argument form of super() cannot be used as the new
class has no name yet.

The problem is solvable though. The initializations necessary for
super() to work properly simply should be moved before the call
to __init_subclass__. I implemented that by putting a new attribute
into the class's namespace to keep the cell which will later be used
by super(). This new attribute would be remove by type.__new__
again, but transiently it would be visible. This technique has already
been used for __qualname__.

The issue contains a patch that fixes that behavior, and back in the
day you proposed I add the problem to the PEP. Should I?

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP487: Simpler customization of class creation

[Martin Teichmann]

Hi list,

so this is the next round for PEP 487. During the last round, most of
the comments were in the direction that a two step approach for
integrating into Python, first in pure Python, later in C, was not a
great idea and everything should be in C directly. So I implemented it
in C, put it onto the issue tracker here:
http://bugs.python.org/issue27366, and also modified the PEP
accordingly.

For those who had not been in the discussion, PEP 487 proposes to add
two hooks, __init_subclass__ which is a classmethod called whenever a
class is subclassed, and __set_owner__, a hook in descriptors which
gets called once the class the descriptor is part of is created.

While implementing PEP 487 I realized that there is and oddity in the
type base class: type.__init__ forbids to use keyword arguments, even
for the usual three arguments it has (name, base and dict), while
type.__new__ allows for keyword arguments. As I plan to forward any
keyword arguments to the new __init_subclass__, I stumbled over that.
As I write in the PEP, I think it would be a good idea to forbid using
keyword arguments for type.__new__ as well. But if people think this
would be to big of a change, it would be possible to do it
differently.

Hoping for good comments

Greetings

Martin

The PEP follows:

PEP: 487
Title: Simpler customisation of class creation
Version: $Revision$
Last-Modified: $Date$
Author: Martin Teichmann <lkb.teichm...@gmail.com>,
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Feb-2015
Python-Version: 3.6
Post-History: 27-Feb-2015, 5-Feb-2016, 24-Jun-2016, 2-Jul-2016
Replaces: 422


Abstract


Currently, customising class creation requires the use of a custom metaclass.
This custom metaclass then persists for the entire lifecycle of the class,
creating the potential for spurious metaclass conflicts.

This PEP proposes to instead support a wide range of customisation
scenarios through a new ``__init_subclass__`` hook in the class body,
and a hook to initialize attributes.

The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power Python's metaclass machinery.


Background
==

Metaclasses are a powerful tool to customize class creation. They have,
however, the problem that there is no automatic way to combine metaclasses.
If one wants to use two metaclasses for a class, a new metaclass combining
those two needs to be created, typically manually.

This need often occurs as a surprise to a user: inheriting from two base
classes coming from two different libraries suddenly raises the necessity
to manually create a combined metaclass, where typically one is not
interested in those details about the libraries at all. This becomes
even worse if one library starts to make use of a metaclass which it
has not done before. While the library itself continues to work perfectly,
suddenly every code combining those classes with classes from another library
fails.

Proposal


While there are many possible ways to use a metaclass, the vast majority
of use cases falls into just three categories: some initialization code
running after class creation, the initalization of descriptors and
keeping the order in which class attributes were defined.

The first two categories can easily be achieved by having simple hooks
into the class creation:

1. An ``__init_subclass__`` hook that initializes
   all subclasses of a given class.
2. upon class creation, a ``__set_owner__`` hook is called on all the
   attribute (descriptors) defined in the class, and

The third category is the topic of another PEP 520.

As an example, the first use case looks as follows::

   >>> class SpamBase:
   ...# this is implicitly a @classmethod
   ...def __init_subclass__(cls, **kwargs):
   ...cls.class_args = kwargs
   ...super().__init_subclass__(cls, **kwargs)

   >>> class Spam(SpamBase, a=1, b="b"):
   ...pass

   >>> Spam.class_args
   {'a': 1, 'b': 'b'}

The base class ``object`` contains an empty ``__init_subclass__``
method which serves as an endpoint for cooperative multiple inheritance.
Note that this method has no keyword arguments, meaning that all
methods which are more specialized have to process all keyword
arguments.

This general proposal is not a new idea (it was first suggested for
inclusion in the language definition `more than 10 years ago`_, and a
similar mechanism has long been supported by `Zope's ExtensionClass`_),
but the situation has changed sufficiently in recent years that
the idea is worth reconsidering for inclusion.

The second part of the proposal adds an ``__set_owner__``
initializer for class attributes, especially if they are descriptors.
Descriptors are defined in the body of a
class, but they do not know anything about that class, they do not
even know the name they are accessed with. They do get to 

Re: [Python-Dev] PEP 487: Simpler customization of class creation

[Martin Teichmann]

Hi Nick, hi List,

thanks for the good comments! I'll update the PEP and the
implementation soon, as discussed at the end, I'll
do it in C this time. For now just some quick responses:

> This part isn't entirely clear to me, so you may want to give some
> Python pseudo-code that:

I will actually give the actual code, it's just 10 lines, that should
be understandable.

> - is explicit regarding exactly when this new code runs in the type
> creation process
> - whether the __set_owner__ hooks are called before or after
> __init_subclass__ runs, or only when the subclass calls up to
> super().__init_subclass__, and the implications of each choice (either
> descriptors see a partially initialised class object, or init_subclass
> sees partially initialised descriptor objects, or that choice is
> delegated to individual subclasses)
> - how the list of objects to be checked for "__set_owner__" methods is
> retrieved (presumably via "ns.items()" on the class definition
> namespace, but the PEP should be explicit)
>
> For the second point, my personal preference would be for descriptors
> to have their owner set first and independently of __init_subclass__
> super calls (as it seems more likely that __init_subclass__ will
> depend on having access to fully initialised descriptors than the
> other way around).

I intuitively programmed it the other way around, but I get your point
and will change it. I agree that it should not be done in
super().__init_subclass__
as people often forget to call it, or do weird things.

> Honestly though, I'm not sure this additional user-visible complexity
> is worth it - "The default type metaclass has this new behaviour" is a
> lot easier to document and explain than "We added a new opt-in
> alternate metaclass that you can use if you want, and in the next
> version that will just become an alias for the builtin types again".
> We'd also end up being stuck with types.Type and types.Object as
> aliases for the type and object builtins forever (with the associated
> "How does 'class name:' or 'class name(object)' differ from 'class
> name(types.Object)'?" question and "It doesn't, unless you're using
> Python 3.6" answer for folks learning the language for the first
> time).

My idea with a stepped approach was to have a chance to look how
people use the feature, so that when we make it standard eventually
we actually get it right. In the end, this is a maintainability question.
I am fully OK with following experienced core developers here, so
if the general idea here is that a two-step approach is not needed,
then no problem, let's do it in the Python core directly!

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 487: Simpler customization of class creation

[Martin Teichmann]

Hi list,

just recently, I posted about the implementation of PEP 487. The
discussion quickly diverted to PEP 520, which happened to be
strongly related.

Hoping to get some comments about the rest of PEP 487, I took
out the part that is also in PEP 520. I attached the new version of
the PEP. The implementation can be found on the Python issue tracker:
http://bugs.python.org/issue27366

So PEP 487 is about simplifying the customization of class creation.
Currently, this is done via metaclasses, which are very powerful, but
often inflexible, as it is hard to combine two metaclasses. PEP 487
proposes a new metaclass which calls a method on all newly created
subclasses. This way, in order to customize the creation of subclasses,
one just needs to write a simple method.

An absolutely classic example for metaclasses is the need to tell descriptors
who they belong to. There are many large frameworks out there, e.g.
enthought's traits, IPython's traitlets, Django's forms and many more.
Their problem is: they're all fully incompatible. It's really hard to inherit
from two classes which have different metaclasses.

PEP 487 proposes to have one simple metaclass, which can do all those
frameworks need, making them all compatible. As an example, imagine
the framework has a generic descriptor called Integer, which describes,
well, an integer. Typically you use it like that:

class MyClass(FrameworkBaseClass):
my_value = Integer()

how does my_value know how it's called, how it should put its data into the
object's __dict__? Well, this is what the framework's metaclass is for.
With PEP 487, a framework doesn't need to declare an own metaclass
anymore, but simply uses types.Object of PEP 487 as a base class:

class FrameworkBaseClass(types.Object):
def __init_subclass__(cls):
super().__init_subclass__()
for k, v in cls.__dict__.items():
if isinstance(v, FrameworkDescriptorBase):
v.__set_owner__(cls, name)

and all the framework's descriptors know their name. And if another framework
should be used as well: no problem, they just work together easily.

Actually, the above example is just that common, that PEP 487 includes
it directly:
a method __set_owner__ is called for every descriptor. That could make most
descriptors in frameworks work out of the box.

So now I am hoping for comments!

Greetings

Martin

New version of the PEP follows:

PEP: 487
Title: Simpler customisation of class creation
Version: $Revision$
Last-Modified: $Date$
Author: Martin Teichmann <lkb.teichm...@gmail.com>,
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Feb-2015
Python-Version: 3.6
Post-History: 27-Feb-2015, 5-Feb-2016, 24-Jun-2016
Replaces: 422


Abstract


Currently, customising class creation requires the use of a custom metaclass.
This custom metaclass then persists for the entire lifecycle of the class,
creating the potential for spurious metaclass conflicts.

This PEP proposes to instead support a wide range of customisation
scenarios through a new ``__init_subclass__`` hook in the class body,
a hook to initialize attributes.

Those hooks should at first be defined in a metaclass in the standard
library, with the option that this metaclass eventually becomes the
default ``type`` metaclass.

The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power Python's metaclass machinery.


Background
==

Metaclasses are a powerful tool to customize class creation. They have,
however, the problem that there is no automatic way to combine metaclasses.
If one wants to use two metaclasses for a class, a new metaclass combining
those two needs to be created, typically manually.

This need often occurs as a surprise to a user: inheriting from two base
classes coming from two different libraries suddenly raises the necessity
to manually create a combined metaclass, where typically one is not
interested in those details about the libraries at all. This becomes
even worse if one library starts to make use of a metaclass which it
has not done before. While the library itself continues to work perfectly,
suddenly every code combining those classes with classes from another library
fails.

Proposal


While there are many possible ways to use a metaclass, the vast majority
of use cases falls into just three categories: some initialization code
running after class creation, the initalization of descriptors and
keeping the order in which class attributes were defined.

Those three use cases can easily be performed by just one metaclass. If
this metaclass is put into the standard library, and all libraries that
wish to customize class creation use this very metaclass, no combination
of metaclasses is necessary anymore. Said metaclass should live in the
``types`` module under the name ``Type``. This should hint th

Re: [Python-Dev] PEP 487: Simpler customization of class creation

[Martin Teichmann]

Hi Eric, hi List,

> I'd be glad to give feedback on this, probably later today or
> tomorrow.  In particular, I'd like to help resolve the intersection
> with PEP 520. :)

Thanks in advance! Let me already elaborate on the differences, so
that others can follow:

You chose the name "__definition_order__", I chose
"__attribute_order__", I am fine with either, what are other people's
opinions?

The bigger difference is actually the path to inclusion into Python:
my idea is to first make it a standard library feature, with the later
option to put it into the C core, while you want to put the feature
directly into the C core. Again I'm fine with either, as long as the
feature is eventually in.

As a side note, you propose to use OrderedDict as the class definition
namespace, and this is exactly how I implemented it. Nonetheless, I
would like to keep this fact as an implementation detail, such that
other implementations of Python (PyPy comes to mind) or even CPython
at a later time may switch to a different way to implement this
feature. I am thinking especially about the option to determine the
-_order__ already at compile time. Sure, this would mean that someone
could trick us by dynamically changing the order of attribute
definition, but I would document that as an abuse of the functionality
with undocumented outcome.

Greetings

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 487: Simpler customization of class creation

[Martin Teichmann]

Hi list,

using metaclasses in Python is a very flexible method of customizing
class creation, yet this customization comes at a cost: once you want
to combine two classes with different metaclasses, you run into
problems.

This is why I proposed PEP 487 (see
https://github.com/tecki/peps/blob/pep487/pep-0487.txt, which I also
attached here for ease of discussion), proposing a simple hook into
class creation, with which one can override in subclasses such that
sub-subclasses get customized accordingly. Otherwise, the standard
Python inheritance rules apply (super() and the MRO).

I also proposed to store the order in which attributes in classes are
defined. This is exactly the same as PEP 520, discussed here recently,
just that unfortunately we chose different names, but I am open for
suggestions for better names.

After having gotten good feedback on python-ideas (see
https://mail.python.org/pipermail/python-ideas/2016-February/038305.html)
and from IPython traitlets as a potential user of the feature (see
https://mail.scipy.org/pipermail/ipython-dev/2016-February/017066.html,
and the code at https://github.com/tecki/traitlets/commits/pep487) I
implemented a pure Python version of this PEP, to be introduced into
the standard library. I also wrote a proof-of-concept for another
potential user of this feature, django forms, at
https://github.com/tecki/django/commits/no-metaclass.

The code to be introduced into the standard library can be found at
https://github.com/tecki/cpython/commits/pep487
(sorry for using github, I'll submit something using hg once I
understand that toolchain). It introduces a new metaclass types.Type
which contains the machinery, and a new base class types.Object which
uses said metaclass. The naming was chosen to clarify the intention
that eventually those classes may be implemented in C and replace type
and object. As above, I am open to better naming.

As a second step, I let abc.ABCMeta inherit from said types.Type, such
that an ABC may also use the features of my metaclass, without the
need to define a new mixing metaclass.

I am looking forward to a lot of comments on this!

Greetings

Martin

The proposed PEP for discussion:

PEP: 487
Title: Simpler customisation of class creation
Version: $Revision$
Last-Modified: $Date$
Author: Martin Teichmann <lkb.teichm...@gmail.com>,
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Feb-2015
Python-Version: 3.6
Post-History: 27-Feb-2015, 5-Feb-2016
Replaces: 422


Abstract


Currently, customising class creation requires the use of a custom metaclass.
This custom metaclass then persists for the entire lifecycle of the class,
creating the potential for spurious metaclass conflicts.

This PEP proposes to instead support a wide range of customisation
scenarios through a new ``__init_subclass__`` hook in the class body,
a hook to initialize attributes, and a way to keep the order in which
attributes are defined.

Those hooks should at first be defined in a metaclass in the standard
library, with the option that this metaclass eventually becomes the
default ``type`` metaclass.

The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power Python's metaclass machinery.


Background
==

Metaclasses are a powerful tool to customize class creation. They have,
however, the problem that there is no automatic way to combine metaclasses.
If one wants to use two metaclasses for a class, a new metaclass combining
those two needs to be created, typically manually.

This need often occurs as a surprise to a user: inheriting from two base
classes coming from two different libraries suddenly raises the necessity
to manually create a combined metaclass, where typically one is not
interested in those details about the libraries at all. This becomes
even worse if one library starts to make use of a metaclass which it
has not done before. While the library itself continues to work perfectly,
suddenly every code combining those classes with classes from another library
fails.

Proposal


While there are many possible ways to use a metaclass, the vast majority
of use cases falls into just three categories: some initialization code
running after class creation, the initalization of descriptors and
keeping the order in which class attributes were defined.

Those three use cases can easily be performed by just one metaclass. If
this metaclass is put into the standard library, and all libraries that
wish to customize class creation use this very metaclass, no combination
of metaclasses is necessary anymore. Said metaclass should live in the
``types`` module under the name ``Type``. This should hint the user that
in the future, this metaclass may become the default metaclass ``type``.

The three use cases are achieved as follows:

1. The metaclass contains an ``__init_subclass__`` hook that initializes
   all subc

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Martin Panter]

> On 06/09/2016 08:52 AM, Guido van Rossum wrote:
> That leaves direct calls to os.urandom(). I don't think this should block
> either.

On 9 June 2016 at 22:22, Larry Hastings  wrote:
> Then it's you and me against the rest of the world ;-)
>
>
> Okay, it's decided: os.urandom() must be changed for 3.5.2 to never block on
> a getrandom() call.  It's permissible to take advantage of
> getrandom(GRND_NONBLOCK), but if it returns EAGAIN we must read from
> /dev/urandom.

So assuming this is the “final” decision, where to from here? I think
the latest change by Colm and committed by Victor already implements
this decision:

https://hg.python.org/cpython/rev/9de508dc4837

Getrandom() is still called, but if it would block, we fall back to
trying the less-secure Linux /dev/urandom, or fail if /dev/urandom is
missing. The Python hash seed is still set using this code. And
os.urandom() calls this code. Random.seed() and SystemRandom still use
os.urandom(), as documented.

So I suggest we close the original mega bug thread
 as fixed. Unless people think
they can change Larry or Guido’s mind, we should focus further
discussion on any changes for 3.6.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: Minor API improvements to bytes, bytearray, and memoryview

[Martin Panter]

On 7 June 2016 at 21:56, Nick Coghlan  wrote:
> On 7 June 2016 at 14:33, Paul Sokolovsky  wrote:
>> Ethan Furman  wrote:
>>> Deprecation of current "zero-initialised sequence" behaviour
>>> 
>>>
>>> Currently, the ``bytes`` and ``bytearray`` constructors accept an
>>> integer argument and interpret it as meaning to create a
>>> zero-initialised sequence of the given size::
>>>
>>>  >>> bytes(3)
>>>  b'\x00\x00\x00'
>>>  >>> bytearray(3)
>>>  bytearray(b'\x00\x00\x00')
>>>
>>> This PEP proposes to deprecate that behaviour in Python 3.6, and
>>> remove it entirely in Python 3.7.
>>
>> Why the desire to break applications of thousands and thousands of
>> people?
>
> Same argument as any deprecation: to make existing and future defects
> easier to find or easier to debug.
>
> That said, this is the main part I was referring to in the other
> thread when I mentioned some of the constructor changes were
> potentially controversial and probably not worth the hassle - it's the
> only one with the potential to break currently working code, while the
> others are just a matter of choosing suitable names.

An argument against deprecating bytearray(n) in particular is that
this is supported in Python 2. I think I have (ab)used this fact to
work around the problem with bytes(n) in Python 2 & 3 compatible code.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C99

[Martin Panter]

On 4 June 2016 at 06:11, Benjamin Peterson  wrote:
> PEP 7 requires CPython to use C code conforming to the venerable C89
> standard. Traditionally, we've been stuck with C89 due to poor C support
> in MSVC. However, MSVC 2013 and 2015 implement the key features of C99.
> C99 does not offer anything earth-shattering; here are the features I
> think we'd find most interesting:
> - Variable declarations can be on any line: removes possibly the most
> annoying limitation of C89.
> - Inline functions: We can make Py_DECREF and Py_INCREF inline functions
> rather than unpleasant macros.
> - C++-style line comments: Not an killer feature but commonly used.
> - Booleans

My most-missed C99 feature would be designated initializers. Does MSVC
support them? It might allow you to do away with those giant pasted
slot tables, and just write the slots you need:

PyTypeObject PyUnicodeIter_Type = {
PyVarObject_HEAD_INIT(_Type, 0)
.tp_name = "str_iterator",
.tp_basicsize = sizeof(unicodeiterobject),
.tp_dealloc = unicodeiter_dealloc,
.tp_getattro = PyObject_GenericGetAttr,
.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
.tp_traverse = unicodeiter_traverse,
.tp_iter = PyObject_SelfIter,
.tp_iternext = unicodeiter_next,
.tp_methods = unicodeiter_methods,
};

> So, what say you to updating PEP 7 to allow C99 features for Python 3.6
> (in so much as GCC and MSVC support them)?

Sounds good for features that are well-supported by compilers that
people use. (Are there other compilers used than just GCC and MSVC?)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ctypes compatibility with 2.3

[Martin Panter]

On 12 May 2016 at 01:05, Brett Cannon  wrote:
>
>
> On Wed, 11 May 2016 at 09:28 Brett Cannon  wrote:
>>
>> On Wed, 11 May 2016 at 09:07 Thomas Heller  wrote:
>>>
>>> Am 11.05.2016 um 18:04 schrieb Brett Cannon:
>>> >
>>> >
>>> > On Wed, 11 May 2016 at 04:35 Thomas Heller >> > > wrote:
>>> > For me it is totally ok to lift this restriction.
>>> >
>>> >
>>> > Great! I'll also update PEP 291.
>>>
>>> Cool.  While you're at it, the compatibility restriction for
>>> modulefinder could also be lifted.
>>
>>
>> Will do.
>
>
> PEP 291 no longer lists any restrictions on ctypes or modulefinder.

Thanks everyone for your responses. I will look at removing the
notices in the code when I get a chance. That would probably involve
reverting

https://hg.python.org/cpython/rev/381a72ab5fb8
And also the modulefinder.py comment

There are also these commits that could be backported
https://hg.python.org/cpython/rev/0980034adaa7 (ctypes)
https://hg.python.org/cpython/diff/627db59031be/Lib/modulefinder.py
but it might be safer just to leave the compatibility code there,
perhaps with a clarifying comment.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] ctypes compatibility with 2.3

[Martin Panter]

I am working on <https://bugs.python.org/issue22636>, to fix shell
injection problems with ctypes.util.find_library(). The proposal for
Python 3 is to change os.popen(shell-script) calls to use
subprocess.Popen().

However the Python 2.7 version of the module has a comment which says
“This file should be kept compatible with Python 2.3, see PEP 291.”
Looking at <https://www.python.org/dev/peps/pep-0291/>, it is not
clear why we have to maintain this compatibility. My best guess is
that there may be an external ctypes package that people want(ed) to
keep compatible with 2.3, and also keep synchronized with 2.7.

I would like to lift this restriction to at least 2.4, because that is
when the “subprocess” module was added. I notice that there is already
code that relies on the list.sort(key=...) feature, which was added in
2.4. Ideally I would prefer to drop the restriction and only require
2.7 compatibility. Would either of these options be a problem?

If it is a problem, I think it is still possible to avoid the shell by
passing a sequence of program arguments to os.popen(). But I prefer
not to do this, because the function is marked as deprecated, and the
code would be substantially different to Python 3.

-Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker Etiquette

[Martin Panter]

On 7 May 2016 at 23:23, Senthil Kumaran  wrote:
>
> On Sat, May 7, 2016 at 4:17 PM, MRAB  wrote:
>>
>> I think you shouldn't delete them. It would be better just to say that
>> you've changed your mind and explain why.
>
>
> I support this. Please leave your new comments correcting previous one and
> support your current stance. I think, it is alright to make revisions in
> comments.

Taking a middle ground between this and what Brett said, I would say
it is okay to delete stuff if it is fresh and hasn’t been replied to
(but maybe leave a message if it is not obvious why you delete it). On
the other hand, avoid deleting stuff if it would harm the overall
thread.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Convert int() to size_t in Python/C

[Martin Panter]

On 29 April 2016 at 18:11, Marcos Dione  wrote:
> On Fri, Apr 29, 2016 at 12:18:46PM -0400, Random832 wrote:
>> On Fri, Apr 29, 2016, at 10:45, Marcos Dione wrote:
>> > One possible solution hat was suggested to me in the #python IRC
>> > channel was to use that, then test if the resulting value is negative,
>> > and adjust accordingly, but I wonder if there is a cleaner, more general
>> > solution (for instance, what if the type was something else, like loff_t,
>> > although for that one in particular there *is* a convertion
>> > function/macro).
>>
>> In principle, you could just use PyLong_AsUnsignedLong (or LongLong),
>> and raise OverflowError manually if the value happens to be out of
>> size_t's range. (99% sure that on every linux platform unsigned long is
>> the same size as size_t.
>>
>> But it's not like it'd be the first function in OS to call a system call
>> that takes a size_t. Read just uses Py_ssize_t. Write uses the buffer
>> protocol, which uses Py_ssize_t. How concerned are you really about the
>> lost range here? What does the system call return (its return type is
>> ssize_t) if it writes more than SSIZE_MAX bytes? (This shouldn't be hard
>> to test, just try copying a >2GB file on a 32-bit system)

I would probably just use Py_ssize_t, since that is what the return
value is. Otherwise, a large positive count input could return a
negative value, which would be inconsistent, and could be mistaken as
an error.

> It's a very good point, but I don't have any 32 bits systems around
> with a kernel-4.5. I'll try to figure it out and/or ask in the kernel ML.

Maybe you can compile a 32-bit program and run it on a 64-bit computer
(gcc -m32).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Convert int() to size_t in Python/C

[Martin Panter]

On 29 April 2016 at 18:25, Random832  wrote:
> On Fri, Apr 29, 2016, at 14:11, Marcos Dione wrote:
>> These are not output parameters, even if they're pointers. they'r
>> using the NULL pointer to signal that the current offsets should not be
>> touched, to differentiate from a offset of 0. Something that in Python we
>> would use None.
>
> That's not actually true according to the documentation. (And if it
> were, they could simply use -1 rather than a null pointer)
> . . .
>*  If off_in is not NULL, then off_in must point to a buffer that
>   specifies the starting offset where bytes from fd_in will be
>   read.
>   The file offset of fd_in is not changed, >>>but off_in is
>   adjusted
>   appropriately.<<<

Linux’s sendfile() syscall takes a similar offset parameter that may
be updated, but Python’s os.sendfile() wrapper does not return the
updated offset. Do you think we need to return the updated offsets for
copy_file_range()?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Not receiving bug tracker emails

[Martin Panter]

On 14 April 2016 at 08:51, Serhiy Storchaka  wrote:
> On 13.04.16 07:39, Terry Reedy wrote:
>>
>> On 4/4/2016 5:05 PM, Terry Reedy wrote:
>>
>> Since a few days, I am getting bug tracker emails again, in my Inbox.  I
>> just got a Rietveld review in the Inbox and I believe it went there
>> directly instead of first to Junk.  Thank you to whoever made the
>> improvements.
>
>
> AFAIK David just disabled IPv6 support.
>
> Most bug tracker emails still went in the Spam folder. I have a filter for
> Roundap emails, but there is no any mark that I can use for filtering
> Rietveld emails.

FWIW I set up the following filter in Gmail for Rietveld reviews:

Matches: http://bugs.python.org/review
Do this: Never send it to Spam

I suspect it helps, but occasionally I think stuff still goes to spam.
(Just don’t tell this secret rule to actual spammers :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Most 3.x buildbots are green again, please don't break them and watch them!

[Martin Panter]

On 14 April 2016 at 09:15, Serhiy Storchaka  wrote:
> On 13.04.16 14:40, Victor Stinner wrote:
>> By the way, I don't understand why "AMD64 OpenIndiana 3.x" is
>> considered as stable since it's failing with multiple issues since
>> many months and nobody is working on these failures. I suggest to move
>> this buildbot back to the unstable category.
>
> I think the main cause is the lack of memory in this buildbot. I tried to
> minimize memory consumption and leaks, but some leaks are left, and they
> provoke other tests failures, and additional resource leaks. Would be nice
> to add a feature for running every test in separate subprocess. This will
> isolate the effect of failed tests.

Last time I looked into the Open Indiana buildbot, I concluded that
the biggest problem was Python using fork() to spawn subprocesses. I
understand that OS does not do “memory overcommitment” like Linux
does, so every time you fork, the OS has to double the amount of
memory that is reserved. It is ironic, but running each test using the
current subprocess module (which uses fork) would probably make the
problem worse.

I suspect using posix_spawn() if possible would help a lot. But this
was rejected in  for not being
flexible enough and making maintainence too complicated.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Incomplete Internationalization in Argparse Module

[Grady Martin]

Excellent.  Issue/patch here:

http://bugs.python.org/issue26726

On 2016年04月09日 08時16分, Guido van Rossum wrote:


OK, so this should be taken to the bug tracker.

On Saturday, April 9, 2016, Grady Martin <sunnycemet...@gmail.com> wrote:


I agree.  However, an incorrect choice for an argument with a choices
parameter results in this string.

On 2016年04月08日 18時12分, Guido van Rossum wrote:



That string looks like it is aimed at the developer, not the user of
the program, so it makes sense not to translate it.

On Fri, Apr 8, 2016 at 2:07 PM, Brett Cannon <br...@python.org> wrote:




On Fri, 8 Apr 2016 at 14:05 Grady Martin <sunnycemet...@gmail.com>
wrote:



Hello, all.  I was wondering if the following string was left untouched
by
gettext for a purpose (from line 720 of argparse.py, in class
ArgumentError):

'argument %(argument_name)s: %(message)s'

There may be other untranslatable strings in the argparse module, but I
have yet to encounter them in the wild.




Probably so that anyone introspecting on the error message can count on
somewhat of a consistent format (comes into play with doctest typically).

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org





--
--Guido van Rossum (python.org/~guido)





--
--Guido (mobile)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Incomplete Internationalization in Argparse Module

[Grady Martin]

I agree.  However, an incorrect choice for an argument with a choices parameter 
results in this string.

On 2016年04月08日 18時12分, Guido van Rossum wrote:


That string looks like it is aimed at the developer, not the user of
the program, so it makes sense not to translate it.

On Fri, Apr 8, 2016 at 2:07 PM, Brett Cannon <br...@python.org> wrote:



On Fri, 8 Apr 2016 at 14:05 Grady Martin <sunnycemet...@gmail.com> wrote:


Hello, all.  I was wondering if the following string was left untouched by
gettext for a purpose (from line 720 of argparse.py, in class
ArgumentError):

'argument %(argument_name)s: %(message)s'

There may be other untranslatable strings in the argparse module, but I
have yet to encounter them in the wild.



Probably so that anyone introspecting on the error message can count on
somewhat of a consistent format (comes into play with doctest typically).

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org





--
--Guido van Rossum (python.org/~guido)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Incomplete Internationalization in Argparse Module

[Grady Martin]

Hello, all.  I was wondering if the following string was left untouched by 
gettext for a purpose (from line 720 of argparse.py, in class ArgumentError):

'argument %(argument_name)s: %(message)s'

There may be other untranslatable strings in the argparse module, but I have 
yet to encounter them in the wild.

Thank you.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com