Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Philipp A.]

Hi Guido,

As far as I can see, the general consensus seems to be to turn them back on
in general: The last person to argue against it was Paul Moore, and he
since said:

“OK, I overstated [that you’re ‘hosed’ by DeprecationWarnings appearing].
Apologies. My recollection is of a lot more end user complaints when
deprecation warnings were previously switched on than others seem to
remember, but I can't find hard facts, so I'll assume I'm misremembering.”

Besides, quite some of the problems people mention would only be fixed by
turning them on in general, not with the compromise.

So I don’t think we need a compromise, right?

Best, Philipp

Guido van Rossum  schrieb am Mi., 8. Nov. 2017 um
03:46 Uhr:

> To cut this thread short, I say we should use Nick's proposal to turn
> these warnings on for __main__ but off elsewhere. (See
> https://mail.python.org/pipermail/python-dev/2017-November/150364.html.)
>
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/flying-sheep%40web.de
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

To top off today's set of hopeful thread closures I want to put this one to
rest.

While I'm not ready to accept PEP 563 yet (I'm waiting for the first cut of
the implementation) I *am* hereby nixing the thunk idea.

When I said I was more worried about that than about the stringification
that wasn't about compatibility so much as about conceptual simplicity.
Right now the contents of __annotations__ can be one of two things: an
object created by evaluating an annotation, or a string literal. With
stringification these two possibilities remain the only two possibilities
(and they will still both occur in files that don't use the __future__
import). With thunks, there would be a third option, a 0-argument callable.

I've always disliked APIs (like Django templates, IIRC) that
"transparently" support callables by calling them and treat all other
values as-is. (What if I have a value that just *happens* to be callable?)
I don't want to add such an API.

I also don't like the idea that there's nothing you can do with a thunk
besides calling it -- you can't meaningfully introspect it (not without
building your own bytecode interpreter anyway).

Using an AST instead of a string is also undesirable -- the AST changes in
each release, and the usual strong compatibility guarantees don't apply
here. And how are you going to do anything with it? If you've got a string
and you want an AST node, it's one call away. But if you've got an AST node
and you want either a string *or* the object to which that string would
evaluate, you've got a lot of work to do. Plus the AST takes up a lot more
space than the string, and we don't have a way to put an AST in a bytecode
file. (And as Inada-san pointed out a thunk *also* takes up more space than
a string.)

Nick, please don't try to save the thunk proposal by carefully dissecting
every one of my objections. That will just prolong its demise.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Guido van Rossum]

I'd call that a feature request. :-)

On Tue, Nov 7, 2017 at 9:28 PM, Nick Coghlan  wrote:

> On 8 November 2017 at 14:58, Guido van Rossum  wrote:
> > What does "RFE" mean? I don't recall hearing that term before on the
> Python
> > bug tracker. Request For E-what? (I presume it's a RedHat internal term?)
>
> Request for Enhancement (as opposed to a bug report). It's not Red Hat
> specific, but shortening it to the initialism is a bit enterprisey :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Guido van Rossum]

It seems there must be at least two threads for each topic worth discussing
at all. Therefore I feel compelled to point to
https://mail.python.org/pipermail/python-dev/2017-November/150381.html,
where I state my own conclusion about dict order.

I know Paul Sokolovsky does not claim to speak for MicroPython, but I think
he had better shut up or he's nevertheless going to damage its reputation
beyond repair. I note that micropython.org claims "MicroPython aims to be
as compatible with normal Python as possible to allow you to transfer code
with ease from the desktop to a microcontroller or embedded system." To me
this implies that it is entirely up to the MicroPython project to decide
what they'll do about the order of dict elements -- they can keep doing
what they are doing, or choose a new dict implementation that satisfies
their space and performance needs while also preserving order, or give
developer a compile-time choice, or give users a choice at startup time, or
something else I haven't thought of yet. Any of those options is better
than continuing the flamewar that Paul is waging.

Finally: the dict type should not be endowed with other parts of the
OrderedDict API, not should other API changes to dict be considered.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Nick Coghlan]

On 8 November 2017 at 14:58, Guido van Rossum  wrote:
> What does "RFE" mean? I don't recall hearing that term before on the Python
> bug tracker. Request For E-what? (I presume it's a RedHat internal term?)

Request for Enhancement (as opposed to a bug report). It's not Red Hat
specific, but shortening it to the initialism is a bit enterprisey :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Guido van Rossum]

I'll probably get complaints because I'm not waiting for the benchmark
results to come in, but I think I've seen enough.

To me the only defensible behavior *other than the pre-3.6 behavior* is
that after deletions the order remains preserved and new insertions happen
at the end -- i.e. the same where they would go if the deleted items were
never inserted.

I find it hard to believe that there would be a speed difference that's
noticeable outside micro-benchmarks or applications making extreme use of
dicts.

PS. It seems odd that people arguing that the behavior after deletions
doesn't matter are also arguing that deletions are uncommon? Surely there
should be no speed penalty if you never delete anything from a dict, so if
you believe deletions are rare anyway, why would you care about paying a
bit extra for them?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Guido van Rossum]

What does "RFE" mean? I don't recall hearing that term before on the Python
bug tracker. Request For E-what? (I presume it's a RedHat internal term?)

On Tue, Nov 7, 2017 at 6:57 PM, Nick Coghlan  wrote:

> On 8 November 2017 at 11:46, Guido van Rossum  wrote:
> > On Tue, Nov 7, 2017 at 5:35 PM, Nick Coghlan  wrote:
> >>
> >> On 8 November 2017 at 10:03, Guido van Rossum  wrote:
> >> > OK, so let's come up with a set of heuristics that does the right
> thing
> >> > for
> >> > those cases specifically. I'd say whenever you're executing code from
> a
> >> > zipfile or some such it's not considered your own code (by default).
> >>
> >> My current preferred heuristic is just to add a new default filter to
> the
> >> list:
> >>
> >> once::DeprecationWarning:__main__
> >>
> >> Which says to warn specifically for the __main__ module, and continue
> >> ignoring everything else.
> >
> > OK, that sounds great.
> >
> >> That way ad hoc scripts and the REPL will get warnings by default,
> >> while zipapps and packages can avoid warnings by keeping their
> >> __main__.py simple, and importing a CLI helper function from another
> >> module. Entry point wrapper scripts will implicitly have the same
> >> effect for installed packages.
> >
> > That's fine.
> >
> >> If folks want to get warnings for other modules as well, then they can
> >> either pass "-Wd" to get warnings for everything, or else enable them
> >> selectively using the default main module filter as an example.
> >
> > Assuming that's how it already works, we're done here. :-)
>
> Cool :)
>
> RFE filed here for that specific change to the default filter set:
> https://bugs.python.org/issue31975
>
> Cheers,
> Nick.
>
> P.S. If anyone wants to follow up on some of the other more esoteric
> ideas we've discussed in the past few days, they can be separate RFEs.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Steven D'Aprano]

On Tue, Nov 07, 2017 at 05:37:15PM +0200, Serhiy Storchaka wrote:
> 07.11.17 16:56, Steven D'Aprano пише:
> >To clarify: if we start with an empty dict, add keys A...D, delete B,
> >then add E...H, we could expect:
[...]

> Rather
> 
> {A: 1, D: 4, C: 3}  # move the last item in place of removed
> {A: 1, D: 4, C: 3, E: 5}

Thanks for the correction.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Random832]

On Tue, Nov 7, 2017, at 09:56, Steven D'Aprano wrote:
> Don't let the perfect be the enemy of the good.
> 
> For many applications, keys are never removed from the dict, so this 
> doesn't matter. If you never delete a key, then the remaining keys will 
> never be reordered.
> 
> I think that Nick's intent was not to say that after a single deletion, 
> the ordering guarantee goes away "forever", but that a deletion may be 
> permitted to reorder the keys, after which further additions will honour 
> insertion order. At least, that's how I interpret him.

Honestly, I think the more intuitive way would be the "other way around"
- deletions don't themselves change the order, but they do cause
subsequent insertions to be allowed to insert at places other than the
end.

Does the implementation in CPython have this property?

One way I have seen this done is that the items themselves live in an
array, and each new item is always inserted in the first empty slot in
the array (empty slots form a freelist). The hash buckets contain
indices into the array.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Joao S. O. Bueno]

Sorry - trigger happy on the previous message.

On 6 November 2017 at 05:09, Guido van Rossum  wrote:
> I still find this unfriendly to users of Python scripts and small apps who
> are not the developers of those scripts. (Large apps tend to spit out so
> much logging it doesn't really matter.)
>
> Isn't there a better heuristic we can come up with so that the warnings tend
> to be on for developers but off for end users?

So, I don't know who is the "Jose Bueno" mentioned in the first message - :-)
I just conveyed a concern from the Brython developers, as I follow the
project -
and I'd rather have my terminal clean when using programs.

Making some thing when one does "python setup.py  develop" to flip the
switches so
that the deprecation warnings get turned on might be a nice idea, and
overall help people, tough.

  joao bueno
  -><-
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Joao S. O. Bueno]

On 6 November 2017 at 05:09, Guido van Rossum  wrote:
> I still find this unfriendly to users of Python scripts and small apps who
> are not the developers of those scripts. (Large apps tend to spit out so
> much logging it doesn't really matter.)
>
> Isn't there a better heuristic we can come up with so that the warnings tend
> to be on for developers but off for end users?
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Nick Coghlan]

On 8 November 2017 at 07:19, Evpok Padding  wrote:
> On 7 November 2017 at 21:47, Chris Barker  wrote:
>> if dict order is preserved in cPython , people WILL count on it!
>
> I won't, and if people do and their code break, they'll have only themselves
> to blame.
> Also, what proof do you have of that besides anecdotal evidence ?

~27 calendar years of anecdotal evidence across a multitude of CPython
API behaviours (as well as API usage in other projects).

Other implementation developers don't say "CPython's runtime behaviour
is the real Python specification" for the fun of it - they say it
because "my code works on CPython, but it does the wrong thing on your
interpreter, so I'm going to stick with CPython" is a real barrier to
end user adoption, no matter what the language specification says.
Blaming users for not writing portable code doesn't achieve anything
in that scenario - it just puts an extra road block in the way of
those users trying out the alternative interpreter implementation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Nick Coghlan]

On 8 November 2017 at 11:46, Guido van Rossum  wrote:
> On Tue, Nov 7, 2017 at 5:35 PM, Nick Coghlan  wrote:
>>
>> On 8 November 2017 at 10:03, Guido van Rossum  wrote:
>> > OK, so let's come up with a set of heuristics that does the right thing
>> > for
>> > those cases specifically. I'd say whenever you're executing code from a
>> > zipfile or some such it's not considered your own code (by default).
>>
>> My current preferred heuristic is just to add a new default filter to the
>> list:
>>
>> once::DeprecationWarning:__main__
>>
>> Which says to warn specifically for the __main__ module, and continue
>> ignoring everything else.
>
> OK, that sounds great.
>
>> That way ad hoc scripts and the REPL will get warnings by default,
>> while zipapps and packages can avoid warnings by keeping their
>> __main__.py simple, and importing a CLI helper function from another
>> module. Entry point wrapper scripts will implicitly have the same
>> effect for installed packages.
>
> That's fine.
>
>> If folks want to get warnings for other modules as well, then they can
>> either pass "-Wd" to get warnings for everything, or else enable them
>> selectively using the default main module filter as an example.
>
> Assuming that's how it already works, we're done here. :-)

Cool :)

RFE filed here for that specific change to the default filter set:
https://bugs.python.org/issue31975

Cheers,
Nick.

P.S. If anyone wants to follow up on some of the other more esoteric
ideas we've discussed in the past few days, they can be separate RFEs.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[INADA Naoki]

On Wed, Nov 8, 2017 at 5:35 AM, Paul G  wrote:
> If dictionary order is *not* guaranteed in the spec and the dictionary order 
> isn't randomized (which I think everyone agrees is a bit messed up), it would 
> probably be useful if you could enable "random order mode" in CPython, so you 
> can stress-test that your code isn't making any assumptions about dictionary 
> ordering without having to use an implementation where order isn't 
> deterministic.
>
> I could either be something like an environment variable SCRAMBLE_DICT_ORDER 
> or a flag like --scramble-dict-order. That would probably help somewhat with 
> the very real problem of "everyone's going to start counting on this ordered 
> property".

Namespace is ordered by language spec.
What does SCRAMBLE_DICT_ORDER in this code?

class A:
def __init__(self):
self.a, self.b, self.c = 1, 2, 3

a = A()
print(a.__dict__)
a.__dict__.pop('a')
print(a.__dict__)


Anyway, I'm -1 on adding such option to dict.  dict in CPython is complicated
already for performance and compatibility reason.
I don't want to add more complexity to dict for such reason.

Regards,

INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Guido van Rossum]

To cut this thread short, I say we should use Nick's proposal to turn these
warnings on for __main__ but off elsewhere. (See
https://mail.python.org/pipermail/python-dev/2017-November/150364.html.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[INADA Naoki]

> 2. Switches keyword args and class body execution namespaces over to
> odict so the test suite passes again
> 3. Measures the impact such a change would have on the benchmark suite

For now, odict use twice memory and 2x slower on iteration.
https://bugs.python.org/issue31265#msg301942

INADA Naoki  


On Wed, Nov 8, 2017 at 11:33 AM, Nick Coghlan  wrote:
> On 8 November 2017 at 11:44, Nick Coghlan  wrote:
>> 2. So far, I haven't actually come up with a perturbed iteration
>> implementation that doesn't segfault the interpreter. The dict
>> internals are nicely laid out to be iteration friendly, but they
>> really do assume that you're going to start at index zero, and then
>> iterate through to the end of the array. The bounds checking and
>> pointer validity testing becomes relatively fiddly if you try to push
>> against that and instead start iteration from a point partway through
>> the storage array.
>
> In case anyone else wants to experiment with a proof of concept:
> https://github.com/ncoghlan/cpython/commit/6a8a6fa32f0a9cd71d9078fbb2b5ea44d5c5c14d
>
> I think we've probably exhausted the utility of discussing this as a
> purely hypothetical change, and so the only way to move the discussion
> forward will be for someone to draft a patch that:
>
> 1. Perturbs iteration for regular dicts (it's OK for our purposes if
> it's still deterministic - it just shouldn't match insertion order the
> way odict does)
> 2. Switches keyword args and class body execution namespaces over to
> odict so the test suite passes again
> 3. Measures the impact such a change would have on the benchmark suite
>
> My experiment is a starting point, but it will still be a fair bit of
> work to get it from there to a viable proof of concept that can be
> assessed against the status quo.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nick Coghlan]

On 8 November 2017 at 11:44, Nick Coghlan  wrote:
> 2. So far, I haven't actually come up with a perturbed iteration
> implementation that doesn't segfault the interpreter. The dict
> internals are nicely laid out to be iteration friendly, but they
> really do assume that you're going to start at index zero, and then
> iterate through to the end of the array. The bounds checking and
> pointer validity testing becomes relatively fiddly if you try to push
> against that and instead start iteration from a point partway through
> the storage array.

In case anyone else wants to experiment with a proof of concept:
https://github.com/ncoghlan/cpython/commit/6a8a6fa32f0a9cd71d9078fbb2b5ea44d5c5c14d

I think we've probably exhausted the utility of discussing this as a
purely hypothetical change, and so the only way to move the discussion
forward will be for someone to draft a patch that:

1. Perturbs iteration for regular dicts (it's OK for our purposes if
it's still deterministic - it just shouldn't match insertion order the
way odict does)
2. Switches keyword args and class body execution namespaces over to
odict so the test suite passes again
3. Measures the impact such a change would have on the benchmark suite

My experiment is a starting point, but it will still be a fair bit of
work to get it from there to a viable proof of concept that can be
assessed against the status quo.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: PEP 479's __future__ about to become the default behavior

[Guido van Rossum]

Thanks for the reminder! I don't know if it's that simple -- did you grep
for occurrences of that flag (CO_FUTURE_GENERATOR_STOP)? And I suppose
there are tests for both sides that need to be adjusted?

On Tue, Nov 7, 2017 at 5:17 PM, Lukasz Langa  wrote:

> This is according to https://www.python.org/dev/
> peps/pep-0479/#transition-plan but looking at Objects/genobject.c that
> hasn't been implemented yet. Is this as simple as removing the `else`
> clause here?
>
> https://github.com/python/cpython/blob/master/Objects/
> genobject.c#L277-L298
>
> - Ł
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

Antoine Pitrou wrote:
> On Mon, 6 Nov 2017 23:23:25 +1000
>> - tweak the default warning filters to turn DeprecationWarning back on
>> for __main__ only
> 
> Thats sounds error-prone.  I'd rather have them on by default
> everywhere.

If DeprecationWarnings were on by default, and setuptools were modified
to silence them in entry point generated mains, and we had a simple API
to easily silence them for manually written mains, wouldn't that handle
the majority of relevant use cases nicely?

-Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 7, 2017, at 16:15, Chris Barker - NOAA Federal  
wrote:

> Actually, there is a LOT of code out there that expects reference counting. I 
> know a lot of my code does. So if cPython does abandon it some day, there 
> will be issues.

I see this all the time in code reviews:

content = open(some file).read()

I never let that go uncommented.

So while you’re right that CPython is the reference implementation, and few 
people read the language spec, it’s still encombunt on us to point out broken 
code, code with implicit assumptions, and code that is not portable between 
implementations.  Having the reference manual to point to chapter and verse is 
critical to avoid Python just devolving into an ad-hoc language ruled by its 
most popular implementation.  This is something I believe Guido realized early 
on when JPython was first invented, and it was an important distinction that 
Python held, e.g. versus Perl.  I still believe it’s an important principle to 
maintain.

Cheers,
-Barry




signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Guido van Rossum]

On Tue, Nov 7, 2017 at 5:35 PM, Nick Coghlan  wrote:

> On 8 November 2017 at 10:03, Guido van Rossum  wrote:
> > OK, so let's come up with a set of heuristics that does the right thing
> for
> > those cases specifically. I'd say whenever you're executing code from a
> > zipfile or some such it's not considered your own code (by default).
>
> My current preferred heuristic is just to add a new default filter to the
> list:
>
> once::DeprecationWarning:__main__
>
> Which says to warn specifically for the __main__ module, and continue
> ignoring everything else.
>

OK, that sounds great.


> That way ad hoc scripts and the REPL will get warnings by default,
> while zipapps and packages can avoid warnings by keeping their
> __main__.py simple, and importing a CLI helper function from another
> module. Entry point wrapper scripts will implicitly have the same
> effect for installed packages.
>

That's fine.

If folks want to get warnings for other modules as well, then they can
> either pass "-Wd" to get warnings for everything, or else enable them
> selectively using the default main module filter as an example.
>

Assuming that's how it already works, we're done here. :-)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nick Coghlan]

On 8 November 2017 at 07:15, Paul Moore  wrote:
> On 7 November 2017 at 20:35, Paul G  wrote:
>> If dictionary order is *not* guaranteed in the spec and the dictionary order 
>> isn't randomized (which I think everyone agrees is a bit messed up), it 
>> would probably be useful if you could enable "random order mode" in CPython, 
>> so you can stress-test that your code isn't making any assumptions about 
>> dictionary ordering without having to use an implementation where order 
>> isn't deterministic.
>>
>> I could either be something like an environment variable SCRAMBLE_DICT_ORDER 
>> or a flag like --scramble-dict-order. That would probably help somewhat with 
>> the very real problem of "everyone's going to start counting on this ordered 
>> property".
>
> This seems like overkill to me. By the same logic, we should add a
> "delay garbage collection" mode, that allows people to test that their
> code doesn't make unwarranted assumptions that a reference-counting
> garbage collector is in use.

Quite a few projects these days include PyPy in their CI test matrix,
and one of the things that does is test whether or not you're relying
on refcounting semantics.

We also offer ResourceWarning natively in CPython, which means if you
run under "python3 -Wd", you'll get a warning when external resources
like files are cleaned up implicitly:

$ python3 -Wd
Python 3.6.2 (default, Oct  2 2017, 16:51:32)
[GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open(".bashrc")
>>> del f
__main__:1: ResourceWarning: unclosed file <_io.TextIOWrapper
name='.bashrc' mode='r' encoding='UTF-8'>
>>>

> Most public projects (which are the only ones that really need to
> worry about this sort of detail) will probably be supporting Python
> 3.5 and likely even Python 2.7 for some time yet. So they test under
> non-order-preserving dictionary implementations anyway. And if code
> that's only targeted for Python 3.7 assumes order preserving
> dictionaries, it's likely not got a huge user base anyway, so what's
> the problem?

The concern is that if we don't explicitly perturb dict iteration
order (or offer a way to opt-in to that), then insertion ordering will
end up becoming a *de facto* compatibility requirement for Python
implementations as CPython 2.7 and other releases prior to 3.6 start
dropping out of typical test matrices. With both 2.7 and 3.5 going
end-of-life in 2020, that means 3.7 (mid 2018) and 3.8 (late 2019 or
early 2020) are our available opportunities to make that decision -
beyond that, it starts getting a lot harder to change course away from
implicit standardisation, as there'll be a lot more 3.6+-only code in
the world by then.

The way golang handled this problem is in their dict iterator: they
added an extra field to hold a randomly generated hash, and used that
hash as the starting point in their iteration sequence.

We wouldn't be able to implement per-iterator order randomisation in
CPython due to the way the PyDict_Next API works: that uses a
caller-provided Py_ssize_t entry to track the current position in the
iteration sequence.

This means the simplest change we can make is to adjust the code in
_PyDict_Next that reads the "current iteration index" from the user
supplied pointer to instead interpret that as having a constant offset
(e.g. starting with the last item in the "natural" iteration order,
and then looping back around to the first one).

"Simplest" isn't the same as "simple" though, as:

1. We can't change this globally for all dicts, as we actually *do*
need keyword argument dicts and class body execution namespaces to be
insertion ordered. That makes it either a per-instance setting, or
else a subtly different dict type.
2. So far, I haven't actually come up with a perturbed iteration
implementation that doesn't segfault the interpreter. The dict
internals are nicely laid out to be iteration friendly, but they
really do assume that you're going to start at index zero, and then
iterate through to the end of the array. The bounds checking and
pointer validity testing becomes relatively fiddly if you try to push
against that and instead start iteration from a point partway through
the storage array.

That second point also becomes a concern from a performance
perspective because this is code that runs on *each* iteration of a
loop, rather than purely as part of the loop setup.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Nick Coghlan]

On 8 November 2017 at 10:03, Guido van Rossum  wrote:
> OK, so let's come up with a set of heuristics that does the right thing for
> those cases specifically. I'd say whenever you're executing code from a
> zipfile or some such it's not considered your own code (by default).

My current preferred heuristic is just to add a new default filter to the list:

once::DeprecationWarning:__main__

Which says to warn specifically for the __main__ module, and continue
ignoring everything else.

That way ad hoc scripts and the REPL will get warnings by default,
while zipapps and packages can avoid warnings by keeping their
__main__.py simple, and importing a CLI helper function from another
module. Entry point wrapper scripts will implicitly have the same
effect for installed packages.

If folks want to get warnings for other modules as well, then they can
either pass "-Wd" to get warnings for everything, or else enable them
selectively using the default main module filter as an example.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Reminder: PEP 479's __future__ about to become the default behavior

[Lukasz Langa]

This is according to https://www.python.org/dev/peps/pep-0479/#transition-plan 
 but looking at 
Objects/genobject.c that hasn't been implemented yet. Is this as simple as 
removing the `else` clause here?

https://github.com/python/cpython/blob/master/Objects/genobject.c#L277-L298 


- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[INADA Naoki]

> By the way, I only just realized I can delete a key to demonstrate
> non-order-preservation on py 3.6. So at least I know what to tell students
> now.
>

You can't.  dict in Python 3.6 preserves insertion order even after
key deletion.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Guido van Rossum]

Let me just cut this short. typing.py will stay, and it will stay
provisional.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Chris Barker - NOAA Federal]

This seems like overkill to me. By the same logic, we should add a

"delay garbage collection" mode, that allows people to test that their

code doesn't make unwarranted assumptions that a reference-counting

garbage collector is in use.


Actually, there is a LOT of code out there that expects reference counting.
I know a lot of my code does. So if cPython does abandon it some day, there
will be issues.

Another thought:

Let’s say Python declares that dict literals are order preserving.

Then, one day, someone invents a massively more efficient non-order
preserving implementation, and we want to use it for Python 4.

So the Python 4 language spec says that dicts are not order preserving. And
this is flagged as an INCOMPATIBLE CHANGE.

Now everyone knows to go and check their code, and the 3to4 tool adds an
“o” to all dict literals.

People will complain, but it won’t be unexpected breakage.

Compare to leaving it as an implementation detail — now we will have a lot
of code in the wild that expects order-preservation (because people don’t
read the language spec) that will break with such a change without any real
warning.

I think we really do need to accept that cPython is a reference
implementation.

Because it is.

By the way, I only just realized I can delete a key to demonstrate
non-order-preservation on py 3.6. So at least I know what to tell students
now.

-CHB







But you can get pretty fine-grained control of garbage collection with
judicious use of gc.disable(). If there were a similar mechanism for
changing the ordering properties of dictionaries in code, I wouldn't
suggest it as an interpreter flag / option.

And you're right - it's not pressing - the people likely to test with
dictionaries scrambled are exactly the people likely to be supporting 2.7
and 3.5, but that won't be true forever, and it would be nice to have
*some* mechanism to test that you're not relying on this property.

@Barry Warsaw

As has been suggested elsewhere, if we decide not to make that guarantee,
then we should probably deliberately randomize iteration order.


This was my suggestion of a middle way, since deliberate randomization
seems like a step too far just to avoid people relying on implementation
details. I've seen plenty of code that assumes that `assert` statements
will always throw `AssertionError`, but that's not guaranteed, and some
people run their test suites with -O just to check that they aren't making
that assumption.

On 11/07/2017 04:15 PM, Paul Moore wrote:

On 7 November 2017 at 20:35, Paul G  wrote:

If dictionary order is *not* guaranteed in the spec and the dictionary
order isn't randomized (which I think everyone agrees is a bit messed up),
it would probably be useful if you could enable "random order mode" in
CPython, so you can stress-test that your code isn't making any assumptions
about dictionary ordering without having to use an implementation where
order isn't deterministic.


I could either be something like an environment variable
SCRAMBLE_DICT_ORDER or a flag like --scramble-dict-order. That would
probably help somewhat with the very real problem of "everyone's going to
start counting on this ordered property".




Most public projects (which are the only ones that really need to

worry about this sort of detail) will probably be supporting Python

3.5 and likely even Python 2.7 for some time yet. So they test under

non-order-preserving dictionary implementations anyway. And if code

that's only targeted for Python 3.7 assumes order preserving

dictionaries, it's likely not got a huge user base anyway, so what's

the problem?


Paul



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Guido van Rossum]

OK, so let's come up with a set of heuristics that does the right thing for
those cases specifically. I'd say whenever you're executing code from a
zipfile or some such it's not considered your own code (by default).

On Tue, Nov 7, 2017 at 2:16 PM, Nick Coghlan  wrote:

> On 8 November 2017 at 06:32, Guido van Rossum  wrote:
> > On Mon, Nov 6, 2017 at 7:23 PM, Terry Reedy  wrote:
> >>
> >> On 11/6/2017 9:47 PM, Nick Coghlan wrote:
> >> [...]
> >>>
> >>> - "only show me legacy calls in *my* code" (the "I trust my deps to
> >>> take care of themselves" use case)
> >>
> >>
> >> Perhaps this should be the new default, where 'my code' means everything
> >> under the directory containing the startup file.  If an app developer
> either
> >> fixes or suppresses warnings from app code when they first appear, then
> >> users will seldom or never see warnings.  So for users, this would then
> be
> >> close to the current default.
> >
> > Yes, this or a close a variant sounds like a decent option.
>
> Unfortunately, there are a lot of common directory layouts where a
> simple filesystem based assumption like that one will lead to warnings
> from third party code:
>
> 1. zipapp archives, where everything, including __main__.py, shares a
> common path prefix (the zip archive)
> 2. Working directories that include a ".venv" link to the default
> virtual environment for a project (this is a not uncommon way of
> telling IDEs which venv to use)
> 3. Package execution, when the package includes a "_vendor" or "_bundle"
> subtree
>
> The one thing we can be reasonably confident counts as "the
> developer's code" is "__main__", but even that isn't completely
> certain in cases where folks are publishing single file scripts for
> use by others (e.g. a DeprecationWarning from get-pip.py would be
> useful to pip developers, but almost entirely unhelpful to users of
> the script itself).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 7, 2017, at 13:34, Jakub Wilk  wrote:
> "import async" would indeed cause deprecation warning, but that's not what 
> ldap3 does. The only uses of the now-keyword "async" in their codebase are 
> like this:
> 
>  from ..strategy.async import AsyncStrategy
>  from .async import AsyncStrategy
> 
> These do not provoke deprecation warnings from Python 3.6. (They probably 
> should!)
> 
> I'm afraid that showing deprecation warnings by default wouldn't have helped 
> in this particular case.

Oh gosh, I should have tried that instead of assuming it would generate the 
same warning.  Yes, that’s definitely a bug.  I wonder if we should push back 
making async/await reserved words until Python 3.8?

https://bugs.python.org/issue31973

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Lukasz Langa]

> On Nov 7, 2017, at 2:33 PM, Nick Coghlan  wrote:
> 
> On 8 November 2017 at 04:41, Lukasz Langa  wrote:
>> 4. How do we even version this library then? Probably like this: 3.7.0.0,
>> 3.7.0.1, 3.7.1.0, and so on. But that depends on answers to the other
>> questions above.
> 
> Something you may want to consider is switching to CalVer for typing
> itself, such that we end up saying something like "Python 3.7.0
> includes typing 2017.12.1".

You don't need to sell me on CalVer, all my private packages use this 
versioning scheme (just with the shorthand 17) :-)

And yes, this is a good suggestion.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Jakub Wilk]

* Barry Warsaw , 2017-11-06, 15:56:
We also depend on ldap3 . Suddenly we 
get a SyntaxError because ldap3 has a module ldap3/strategy/async.py. 
I say "suddenly" because of course *if* DeprecationWarnings had been 
enabled by default, I'm sure someone would have noticed that those 
imports were telling the developers about the impending problem in 
Python 3.6.


https://github.com/cannatag/ldap3/issues/428


"import async" would indeed cause deprecation warning, but that's not 
what ldap3 does. The only uses of the now-keyword "async" in their 
codebase are like this:


  from ..strategy.async import AsyncStrategy
  from .async import AsyncStrategy

These do not provoke deprecation warnings from Python 3.6. (They 
probably should!)


I'm afraid that showing deprecation warnings by default wouldn't have 
helped in this particular case.


--
Jakub Wilk
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Nick Coghlan]

On 8 November 2017 at 04:41, Lukasz Langa  wrote:
> 4. How do we even version this library then? Probably like this: 3.7.0.0,
> 3.7.0.1, 3.7.1.0, and so on. But that depends on answers to the other
> questions above.

Something you may want to consider is switching to CalVer for typing
itself, such that we end up saying something like "Python 3.7.0
includes typing 2017.12.1".

My experience has been that CalVer is just *better* for
interoperability specifications, since it inherently conveys
information about the age of the specification. Saying "We target
typing 2017.12.1" in 2018 immediately lets people know they're going
to need some pretty up to date software to run a project that has that
caveat on it. By contrast, saying the same thing in 2021 means most
things released in the past 3 years should be able to handle it.

Such an approach also avoids future confusion if the final version of
Python 3.7 were to start bundling Python 3.8's version of typing.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Nick Coghlan]

On 8 November 2017 at 06:32, Guido van Rossum  wrote:
> On Mon, Nov 6, 2017 at 7:23 PM, Terry Reedy  wrote:
>>
>> On 11/6/2017 9:47 PM, Nick Coghlan wrote:
>> [...]
>>>
>>> - "only show me legacy calls in *my* code" (the "I trust my deps to
>>> take care of themselves" use case)
>>
>>
>> Perhaps this should be the new default, where 'my code' means everything
>> under the directory containing the startup file.  If an app developer either
>> fixes or suppresses warnings from app code when they first appear, then
>> users will seldom or never see warnings.  So for users, this would then be
>> close to the current default.
>
> Yes, this or a close a variant sounds like a decent option.

Unfortunately, there are a lot of common directory layouts where a
simple filesystem based assumption like that one will lead to warnings
from third party code:

1. zipapp archives, where everything, including __main__.py, shares a
common path prefix (the zip archive)
2. Working directories that include a ".venv" link to the default
virtual environment for a project (this is a not uncommon way of
telling IDEs which venv to use)
3. Package execution, when the package includes a "_vendor" or "_bundle" subtree

The one thing we can be reasonably confident counts as "the
developer's code" is "__main__", but even that isn't completely
certain in cases where folks are publishing single file scripts for
use by others (e.g. a DeprecationWarning from get-pip.py would be
useful to pip developers, but almost entirely unhelpful to users of
the script itself).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[MRAB]

On 2017-11-07 14:17, Philipp A. wrote:
Nick Coghlan > schrieb am 
Di., 7. Nov. 2017 um 14:57 Uhr:


Users of applications written in Python are not python-dev's users:
they're the users of those applications, and hence the quality of
that experience is up to the developers of those applications. […]


Thank you, that’s exactly what I’m talking about. Besides: Nobody is 
“hosed”… There will be one occurrence of every DeprecationWarning in the 
stderr of the application. Hardly the end of the world for CLI 
applications and even invisible for GUI applications.


If the devs care about the user not seeing any warnings in their CLI 
application, they’ll have a test set up for that, which will tell them 
that the newest python-dev would raise a new warning, once they turn on 
testing for that release. That’s completely fine!


Explicit is better than implicit! If I know lib X raises 
DeprecationWarnings I don’t care about, I want to explicitly silence 
them, instead of missing out on all the valuable information in other 
DeprecationWarnings.


Also, errors should never pass silently. Deprecation warnings are future 
errors.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul G]

@ Paul Moore
> This seems like overkill to me. By the same logic, we should add a
> "delay garbage collection" mode, that allows people to test that their
> code doesn't make unwarranted assumptions that a reference-counting
> garbage collector is in use.

But you can get pretty fine-grained control of garbage collection with 
judicious use of gc.disable(). If there were a similar mechanism for changing 
the ordering properties of dictionaries in code, I wouldn't suggest it as an 
interpreter flag / option.

And you're right - it's not pressing - the people likely to test with 
dictionaries scrambled are exactly the people likely to be supporting 2.7 and 
3.5, but that won't be true forever, and it would be nice to have *some* 
mechanism to test that you're not relying on this property.

@Barry Warsaw
> As has been suggested elsewhere, if we decide not to make that guarantee, 
> then we should probably deliberately randomize iteration order.

This was my suggestion of a middle way, since deliberate randomization seems 
like a step too far just to avoid people relying on implementation details. 
I've seen plenty of code that assumes that `assert` statements will always 
throw `AssertionError`, but that's not guaranteed, and some people run their 
test suites with -O just to check that they aren't making that assumption.

On 11/07/2017 04:15 PM, Paul Moore wrote:
> On 7 November 2017 at 20:35, Paul G  wrote:
>> If dictionary order is *not* guaranteed in the spec and the dictionary order 
>> isn't randomized (which I think everyone agrees is a bit messed up), it 
>> would probably be useful if you could enable "random order mode" in CPython, 
>> so you can stress-test that your code isn't making any assumptions about 
>> dictionary ordering without having to use an implementation where order 
>> isn't deterministic.
>>
>> I could either be something like an environment variable SCRAMBLE_DICT_ORDER 
>> or a flag like --scramble-dict-order. That would probably help somewhat with 
>> the very real problem of "everyone's going to start counting on this ordered 
>> property".
> 

> 
> Most public projects (which are the only ones that really need to
> worry about this sort of detail) will probably be supporting Python
> 3.5 and likely even Python 2.7 for some time yet. So they test under
> non-order-preserving dictionary implementations anyway. And if code
> that's only targeted for Python 3.7 assumes order preserving
> dictionaries, it's likely not got a huge user base anyway, so what's
> the problem?
> 
> Paul
> 



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Moore]

On 7 November 2017 at 21:13, Chris Barker  wrote:
> the "only until first deletion" part is really hard -- I hope we don't go
> that route. But I don't think insertion-order is non-obvious -- particularly
> with literals.

But I thought we *had* gone that route. Actually, there's no "route"
to go here. We're only talking about documenting the existing
semantics that cPython has, and I thought that included no longer
guaranteeing insertion order after a delete. Although I can't prove
that by experiment at the moment.

I don't know whether my confusion above is an argument for or against
documenting the behaviour :-)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Evpok Padding]

On 7 November 2017 at 21:47, Chris Barker  wrote:

> On Tue, Nov 7, 2017 at 11:50 AM, Tim Peters  wrote:
>
>> Is it worth guaranteeing that will always "work" (as intended)?  Not
>> to me, but I do have code that relies on it now -
>
>
> This is critically important -- no one looks at the language spec to
> figure out how something works -- they try it, and if it works assume it
> will continue to work.
>
> if dict order is preserved in cPython , people WILL count on it!
>

I won't, and if people do and their code break, they'll have only
themselves to blame.
Also, what proof do you have of that besides anecdotal evidence ?

E
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 7, 2017, at 12:35, Paul G  wrote:
> 
> If dictionary order is *not* guaranteed in the spec and the dictionary order 
> isn't randomized (which I think everyone agrees is a bit messed up), it would 
> probably be useful if you could enable "random order mode" in CPython, so you 
> can stress-test that your code isn't making any assumptions about dictionary 
> ordering without having to use an implementation where order isn't 
> deterministic.

As has been suggested elsewhere, if we decide not to make that guarantee, then 
we should probably deliberately randomize iteration order.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Moore]

On 7 November 2017 at 20:35, Paul G  wrote:
> If dictionary order is *not* guaranteed in the spec and the dictionary order 
> isn't randomized (which I think everyone agrees is a bit messed up), it would 
> probably be useful if you could enable "random order mode" in CPython, so you 
> can stress-test that your code isn't making any assumptions about dictionary 
> ordering without having to use an implementation where order isn't 
> deterministic.
>
> I could either be something like an environment variable SCRAMBLE_DICT_ORDER 
> or a flag like --scramble-dict-order. That would probably help somewhat with 
> the very real problem of "everyone's going to start counting on this ordered 
> property".

This seems like overkill to me. By the same logic, we should add a
"delay garbage collection" mode, that allows people to test that their
code doesn't make unwarranted assumptions that a reference-counting
garbage collector is in use.

Most public projects (which are the only ones that really need to
worry about this sort of detail) will probably be supporting Python
3.5 and likely even Python 2.7 for some time yet. So they test under
non-order-preserving dictionary implementations anyway. And if code
that's only targeted for Python 3.7 assumes order preserving
dictionaries, it's likely not got a huge user base anyway, so what's
the problem?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Chris Barker]

On Tue, Nov 7, 2017 at 7:21 AM, David Mertz  wrote:

> But like Raymond, I make most of my living TEACHING Python.
>

and I make least of my living TEACHING Python ( :-) ),and:


> I feel like the extra order guarantee would make teaching slightly harder.
>

I can't understand how this possibly makes python (or dicts) harder to
teach -- you can simply say: "dicts insertion order is preserved" or not
mention it at all -- I think most people kind of expect it to be preserved,
which is why I (used to )always bring up the lack-of-ordering of dicts
early on -- but I suspect I simply won't bother mentioning it if it's
decided as a language feature.

I'm sure he feels contrarily. It is true that with 3.6 I can no longer show
> an example where the dict display is oddly changed when printed.
>

Exactly! I have a really hard time deciding how to handle this --
explaining that ordering is not guaranteed, but not being able to
demonstrate it! And frankly, my students are all going to forget what I
"explained" soon enough, and replace it with their experience -- which will
be that dicts retain their order.

But then, unordered sets also wind up sorting small integers on printing,
> even though that's not a guarantee.
>

but it's not hard to make an easy example with order not preserved -- jsut
start with a non order example:

In [6]: s = {3,7,4}

In [7]: s

Out[7]: {3, 4, 7}

or use other types...

And  "set" is a mathematical concept that has no oder, whereas the
"dictionary" metaphor DOES have order...

> Ordering by insertion order (possibly "only until first deletion") is
> simply not obvious to beginners.
>

the "only until first deletion" part is really hard -- I hope we don't go
that route. But I don't think insertion-order is non-obvious --
particularly with literals.


> If we had, hypothetically, a dict that "always alphabetized keys" that
> would be more intuitive to them, for example. Insertion order feels obvious
> to us experts, but it really is an extra cognitive burden to learners
> beyond understanding "key/Val association".
>

again, I don't think so -- I kind of agree if dicts did not preserve order
in practice -- demonstrating that right out of the gate does help make the
"key/Val association" clear -- but if you can't demonstrate it, I think
we're looking at more confusion...

Maybe I'll ask my students this evening -- this is the first class I'm
teaching with py3.6 

We've lived without order for so long that it seems that some of us now
> think data scrambling is a virtue.  But it isn't.  Scrambled data is the
> opposite of human friendly.
>
>
exactly!

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Tim Delaney]

On 8 November 2017 at 03:55, Barry Warsaw  wrote:

> On Nov 7, 2017, at 05:44, Paul Moore  wrote:
>
> > If you're a user and your application developer didn't do (1) or a
> > library developer developing one of the libraries your application
> > developer chose to use didn't do (2), you're hosed. If you're a user
> > who works in an environment where moving to a new version of the
> > application is administratively complex, you're hosed.
>
> “hosed” feels like too strong of a word here.  DeprecationWarnings usually
> don’t break anything.  Sure, they’re annoying but they can usually be
> easily ignored.
>
> Yes, there are some situations where DWs do actively break things (as I’ve
> mentioned, some Debuntu build/test environments).  But those are also
> relatively easier to silence, or at least the folks running those
> environments, or writing the code for those environments, are usually more
> advanced developers for whom setting an environment variable or flag isn’t
> that big of a deal.
>

One other case would be if you've got an application with no stderr (e.g. a
GUI application) - with enough deprecation warnings the stderr buffer could
become full and block, preventing the application from progressing. I've
just had a similar issue where a process was running as a service and used
subprocess.check_output() - stderr was written to the parent's stderr,
which didn't exist and caused the program to hang.

However, I'm definitely +1 on enabling DeprecationWarning by default, but
with mechanisms or recommendations for the application developer to silence
them selectively for the current release.

Tim Delaney
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Chris Barker]

On Tue, Nov 7, 2017 at 11:50 AM, Tim Peters  wrote:

> Is it worth guaranteeing that will always "work" (as intended)?  Not
> to me, but I do have code that relies on it now -


This is critically important -- no one looks at the language spec to figure
out how something works -- they try it, and if it works assume it will
continue to work.

if dict order is preserved in cPython , people WILL count on it!

And similarly, having order preserved only until a delete is going to cause
bugs in people's code that are less than careful :-( -- and that's most of
us.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul G]

If dictionary order is *not* guaranteed in the spec and the dictionary order 
isn't randomized (which I think everyone agrees is a bit messed up), it would 
probably be useful if you could enable "random order mode" in CPython, so you 
can stress-test that your code isn't making any assumptions about dictionary 
ordering without having to use an implementation where order isn't 
deterministic.

I could either be something like an environment variable SCRAMBLE_DICT_ORDER or 
a flag like --scramble-dict-order. That would probably help somewhat with the 
very real problem of "everyone's going to start counting on this ordered 
property".

On 11/07/2017 12:58 PM, Barry Warsaw wrote:
> On Nov 7, 2017, at 09:39, Paul Sokolovsky  wrote:
> 
>> So, the problem is that there's no "Python language spec”.
> 
> There is a language specification: 
> https://docs.python.org/3/reference/index.html
> 
> But there are still corners that are undocumented, or topics that are 
> deliberately left as implementation details.
> 
> Cheers,
> -Barry
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
> 



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Guido van Rossum]

On Mon, Nov 6, 2017 at 7:23 PM, Terry Reedy  wrote:

> On 11/6/2017 9:47 PM, Nick Coghlan wrote:
> [...]
>
>> - "only show me legacy calls in *my* code" (the "I trust my deps to
>> take care of themselves" use case)
>>
>
> Perhaps this should be the new default, where 'my code' means everything
> under the directory containing the startup file.  If an app developer
> either fixes or suppresses warnings from app code when they first appear,
> then users will seldom or never see warnings.  So for users, this would
> then be close to the current default.
>

Yes, this or a close a variant sounds like a decent option.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nathaniel Smith]

On Nov 7, 2017 12:02 PM, "Barry Warsaw"  wrote:

On Nov 7, 2017, at 09:39, Paul Sokolovsky  wrote:

> So, the problem is that there's no "Python language spec”.

There is a language specification: https://docs.python.org/3/refe
rence/index.html

But there are still corners that are undocumented, or topics that are
deliberately left as implementation details.


Also, specs don't mean that much unless there are multiple implementations
in widespread use. In JS the spec matters because it describes the common
subset of the language you can expect to see across browsers, and lets the
browser vendors coordinate on future changes. Since users actually target
and test against multiple implementations, this is useful. In python,
CPython's dominance means that most libraries are written against CPython's
behavior instead of the spec, and alternative implementations generally
don't care about the spec, they care about whether they can run the code
their users want to run. So PyPy has found that for their purposes, the
python spec includes all kinds of obscure internal implementation details
like CPython's static type/heap type distinction, the exact tricks CPython
uses to optimize local variable access, the CPython C API, etc. The Pyston
devs found that for their purposes, refcounting actually was a mandatory
part of the python language. Jython, MicroPython, etc make a different set
of compatibility tradeoffs again.

I'm not saying the spec is useless, but it's not magic either. It only
matters to the extent that it solves some problem for people.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Tim Peters]

[Peter Ludemann]
> Does it matter whether the dict order after pop/delete is explicitly
> specified, or just specified that it's deterministic?

Any behavior whatsoever becomes essential after it becomes known ;-)

For example, dicts as currently ordered easily support LRU (least
recently used) purging like so:

On access:

result = d.pop(key)
d[key] = result

This moves `key` from wherever it was to the _most_ recently used position.

To purge the `numtopurge` least recently used keys (since traversing
the dict is always from least-recently to most-recently added):

topurge = tuple(itertools.islice(d, numtopurge))
for key in topurge:
del d[key]

Is it worth guaranteeing that will always "work" (as intended)?  Not
to me, but I do have code that relies on it now - and we can count on
someone else saying it's utterly crucial ;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Barry Warsaw]

Antoine Pitrou wrote:

> Well... It really depends what kind of problem you're solving.  I
> certainly delete or pop items from dicts quite often.
> 
> Let's not claim that deleting items from a dict is a rare or advanced
> feature.  It is not.

+1.  It's a pretty common pattern for handling optional keyword
arguments, e.g. in subclass methods.

class Foo(Bar):
def foo(self, *args **kws):
mine = kws.pop('mine', None)
super().foo(self, *args, **kws)
do_something_myself(mine)

Now the question is, what guarantees does the language make about the
ordering of kws that Foo.foo() is passing to Bar.foo()?

-Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Barry Warsaw]

On Nov 7, 2017, at 10:41, Lukasz Langa  wrote:

> 3. Does that mean that Debian is going to rip it out and make people install 
> a `python-typing` .deb? Sadly, probably yes. We need to figure out what that 
> means for us.

Maybe.  Full disclosure, I’ve recently scaled back my contributions to Debian, 
so I won’t be doing this work, but if I was, I’d probably handle it very 
similarly to other replaceable external dependencies (e.g. pip).

There is a small loophole in policy to allow for the building and use of wheels 
for just this *limited* purpose.  So roughly I would propose packaging the 
external python-typing package as a separately installable deb, but also to 
build this into a wheel that we can pull in at python3.7 interpreter package 
build time.

It’s fairly easy to do, and all the infrastructure is already there.  What 
would be useful is for upstream CPython to make it easy to import an externally 
built and installed wheel, from some location outside of its own installed tree 
(/usr/share/python-wheels).

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Peter Ludemann via Python-Dev]

Does it matter whether the dict order after pop/delete is explicitly
specified, or just specified that it's deterministic?

On 7 November 2017 at 11:28, Antoine Pitrou  wrote:

> On Wed, 8 Nov 2017 06:19:46 +1100
> Chris Angelico  wrote:
> >
> > I've used a good few dictionary objects in my time, but most of them
> > have literally never had any items deleted from them.
>
> Well... It really depends what kind of problem you're solving.  I
> certainly delete or pop items from dicts quite often.
>
> Let's not claim that deleting items from a dict is a rare or advanced
> feature.  It is not.
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> pludemann%40google.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Antoine Pitrou]

On Wed, 8 Nov 2017 06:19:46 +1100
Chris Angelico  wrote:
> 
> I've used a good few dictionary objects in my time, but most of them
> have literally never had any items deleted from them.

Well... It really depends what kind of problem you're solving.  I
certainly delete or pop items from dicts quite often.

Let's not claim that deleting items from a dict is a rare or advanced
feature.  It is not.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Chris Angelico]

On Wed, Nov 8, 2017 at 1:32 AM, Antoine Pitrou  wrote:
> On Wed, 8 Nov 2017 00:01:04 +1000
> Nick Coghlan  wrote:
>
>> On 7 November 2017 at 23:48, Stefan Krah  wrote:
>> >
>> >
>> > This is just a reminder that the current dict is not an "OrderedDict":
>> >
>>  from collections import OrderedDict
>>  OrderedDict(a=0, b=1) == OrderedDict(b=1, a=0)
>> > False
>>  dict(a=0, b=1) == dict(b=1, a=0)
>> > True
>> >
>> > The recent proposal was primarily about guaranteeing the insertion order of
>> > dict literals.
>> >
>> > If further guarantees are proposed, perhaps it would be a good idea to
>> > open a new thread and state what exactly is being proposed.
>>
>> "Insertion ordered until the first key removal" is the only guarantee
>> that's being proposed.
>
> Is it?  It seems to me that many arguments being made are only relevant
> under the hypothesis that insertion is ordered even after the first key
> removal.  For example the user-friendliness argument, for I don't
> think it's very user-friendly to have a guarantee that disappears
> forever on the first __del__.

I've used a good few dictionary objects in my time, but most of them
have literally never had any items deleted from them. Consider a
simple anagram finder:

anagrams = defaultdict(list)
for word in words:
anagrams[''.join(sorted(word))].append(word)
for words in anagrams.values():
if len(words) > 5:
print(words)

New items get added to the dictionary, but nothing is ever removed. I
can assume, with CPython's current semantics, that the final iteration
will be in order of first seen; whatever order the incoming word list
was in, the output will be in too. This IS a useful guarantee, even
with the caveats.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Lukasz Langa]

> On Nov 7, 2017, at 9:39 AM, Brett Cannon  wrote:
> 
> On Tue, 7 Nov 2017 at 03:34 Paul Moore  > wrote:
> 
> Because type annotations are a development-time feature, they should
> *not* require a dependency in the final deployment (apart from Python
> itself). However, because they are a language syntax feature they are
> of necessity written in the application source. And type specification
> of anything more complex than basic types (for example, List[int])
> requires classes defined in the typing module. Therefore, typing must
> be in the stdlib so that use of type annotations by the developer
> doesn't impose a runtime dependency on the end user.
> 
> That's not necessarily true if Lukasz's PEP lands and annotations become 
> strings. The dependency would only need to be installed if someone chose to 
> introspect the annotations and then "instantiate" them into actual objects. 
> And that only comes up if someone does it from outside by a 3rd-party, who 
> would then need to install the type annotation dependencies themselves.

PEP 563 is the first step there but it's not enough. For this idea to work, 
`typing.TYPE_CHECKING` would need to be moved to the Python runtime, so that 
you can do

if TYPE_CHECKING:
from typing import *

More importantly, you would put type aliases, type variables and new-types in 
this `if` block, too.

But even if all this is true, there's still two remaining features that require 
runtime availability:

1. `cast()`; and
2. creating generic classes by subclassing `Generic[T]` or any of its 
subclasses. And soon enough, Protocols.

I hope I'm not forgetting any other cases. For `cast()` I have an idea how to 
move it to a variable annotation. For generic classes, there's no magic, you 
need `typing` at runtime. Fortunately, that latter case is (I hope?) relatively 
unlikely.

All the above is summarized here:
https://github.com/python/typing/issues/496 


- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Lukasz Langa]

> On Nov 6, 2017, at 8:01 PM, Nick Coghlan  wrote:
> 
> As I indicated in my comment there, I'm now wondering if there might
> be an opportunity here whereby we could use the *dataclasses* module
> to define a stable non-provisional syntactically compatible subset of
> the typing module, and require folks to start explicitly depending on
> the typing module (as a regular PyPI dependency) if they want anything
> more sophisticated than that (Unions, Generics, TypeVars, etc).

I have an issue open about essentially this idea:
https://github.com/python/typing/issues/496


The consensus is that this is too expensive to do this in time for Python 3.7. 
On the other hand, not doing anything is terrible in terms of usability: now 
users will be forced to use both `typing` (a frozen non-provisional version in 
the stdlib) and `typing_extensions` (new features that we couldn't add directly 
to `typing`).


This is the reason why Guido asked about moving `typing` out of the standard 
library. Currently it seems that the best thing that we can do is to ship an 
upgradeable version of `typing` with Python 3.7.0. There are many details to be 
figured out here:

1. Does that mean `typing` is going to be provisional forever? The answer is 
surprisingly "yes" since it makes sense to keep updating the bundled `typing` 
in 3.7.1, 3.7.2, etc. We need to update PEP 411 to talk about this new case.

2. Does that mean `typing` can now become backwards incompatible randomly? The 
answer should be "no" and already is "no". If you look at typing.py, it already 
ships in Python 3.6 with some crazy dances that makes it work even on Python 
3.3.

3. Does that mean that Debian is going to rip it out and make people install a 
`python-typing` .deb? Sadly, probably yes. We need to figure out what that 
means for us.

4. How do we even version this library then? Probably like this: 3.7.0.0, 
3.7.0.1, 3.7.1.0, and so on. But that depends on answers to the other questions 
above.

5. What will happen to typing_extensions and mypy_extensions? They can probably 
be folded into typing 3.7.0.0. This is a huge win for usability.

6. Where will we track issues for it? Which repo will be the source of truth? 
Fortunately, we already have https://github.com/python/typing/ 



There's more things to figure out for sure. And the clock is ticking. But this 
is a great case study to figure out since more packages in the standard library 
could use a release model like this, and maybe in the future this will finally 
lead to unbundling of the standard library from the runtime. Yeah, it's a long 
shot and the dependency graph between libraries in the standard library is 
rather extensive. But this is the first step.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 7, 2017, at 09:39, Paul Sokolovsky  wrote:

> So, the problem is that there's no "Python language spec”.

There is a language specification: 
https://docs.python.org/3/reference/index.html

But there are still corners that are undocumented, or topics that are 
deliberately left as implementation details.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Mark Lawrence]

On 07/11/17 04:05, David Mertz wrote:
I strongly opposed adding an ordered guarantee to regular dicts. If the 
implementation happens to keep that, great. Maybe OrderedDict can be 
rewritten to use the dict implementation. But the evidence that all 
implementations will always be fine with this restraint feels poor, and 
we have a perfectly good explicit OrderedDict for those who want that.



If there is an ordered guarantee for regular dicts but not for dict 
literals, which is the subject of this thread, then haven't we got a 
recipe for the kind of confusion that will lead to the number of 
questions from newbies going off of the Richter scale?


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Sokolovsky]

Hello,

On Tue, 7 Nov 2017 14:40:07 +0900
INADA Naoki  wrote:

> I agree with Raymond.  dict ordered by default makes better developer
> experience.
> 
> So, my concern is how "language spec" is important for minor (sorry
> about my bad vocabulary) implementation?
> What's difference between "MicroPython is 100% compatible with
> language spec" and
> "MicroPython is almost compatible with Python language spec, but has
> some restriction"?

So, the problem is that there's no "Python language spec". And over
time, that becomes a problem for alternative implementations, especially
not mainstream ("we have infinite amount of memory to burn") ones. What
we have is just CPython documentation, which mixes Python language spec
and CPython implementation details. And is being changed (including
"language spec" part) on the fiat of CPython developers, apparently
without any guarantees of platform stability and backward
compatibility.

Over time, this really becomes a visible drawback, comparing to the
close competitors. For example, year goes by year, but in JavaScript,

[] + []

is still:

''

That's stability!

[]

-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Brett Cannon]

On Tue, 7 Nov 2017 at 03:34 Paul Moore  wrote:

> On 7 November 2017 at 10:18, Steve Holden  wrote:
> > On Tue, Nov 7, 2017 at 12:35 AM, Donald Stufft  wrote:
> > [..]
> >
> >>
> >> Maybe we just need to fully flesh out the idea of a "Python Core" (What
> >> exists now as “Python”) and a “Python Platform” (Python Core + A select
> set
> >> of preinstalled libraries). Then typing can just be part of the Python
> >> Platform, and gets installed as part of your typical installation, but
> is
> >> otherwise an independent piece of code.
> >>
> > Given that (type and other) annotations have been promoted as an optional
> > feature of the language it seems unfair and perhaps unwise to add a
> > dependency specifically to support
> > them
> >  to the stdlib and therefore the Python core.
> > Since type annotations are, as Paul pointed out, development-time
> features,
> > it would appear to behoove those wishing to use them to separate them in
> > such a way that the software can be installed without annotations, and
> > therefore without the need for the typing module. Assuming they would
> like
> > to see the widest possible distribution, of course. For selected
> audiences I
> > am sure typing will be de rigeur,
>
> From my point of view, I take the same premise and come to the
> opposite conclusion.
>
> Because type annotations are a development-time feature, they should
> *not* require a dependency in the final deployment (apart from Python
> itself). However, because they are a language syntax feature they are
> of necessity written in the application source. And type specification
> of anything more complex than basic types (for example, List[int])
> requires classes defined in the typing module. Therefore, typing must
> be in the stdlib so that use of type annotations by the developer
> doesn't impose a runtime dependency on the end user.
>

That's not *necessarily* true if Lukasz's PEP lands and annotations become
strings. The dependency would only need to be installed if someone chose to
introspect the annotations and then "instantiate" them into actual objects.
And that only comes up if someone does it from outside by a 3rd-party, who
would then need to install the type annotation dependencies themselves.

I fully admit that could get messy if introspection from 3rd-parties
happens at all regularly and trying to manage that kind of situation. In
this instance I would argue that any code that is to facilitate creating an
object from an annotation exist outside of the stdlib if 'typing' gets
removed to prevent this sort of situation without the user of such code
being fully aware of what they are up against.

-Brett


>
> If there were a way of including type annotations that had no runtime
> effect on the final deployed program, things would be different. But
> the decision to make annotations part of the language syntax precludes
> that. In particular, "it would appear to behoove those wishing to use
> them to separate them" - there's no way of doing that *precisely*
> because they are a language syntax feature.
>
> Paul
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Sokolovsky]

Hello,

On Tue, 7 Nov 2017 17:33:03 +1100
Steven D'Aprano  wrote:

> On Mon, Nov 06, 2017 at 06:35:48PM +0200, Paul Sokolovsky wrote:
> 
> > For MicroPython, it would lead to quite an overhead to make
> > dictionary items be in insertion order. As I mentioned, MicroPython
> > optimizes for very low bookkeeping memory overhead, so lookups are
> > effectively O(n), but orderedness will increase constant factor
> > significantly, perhaps 5x.  
> 
> Paul, it would be good if you could respond to Raymond's earlier 
> comments where he wrote:
> 
> I've just looked at the MicroPython dictionary implementation and 
> think they won't have a problem implementing O(1) compact dicts
> with ordering.
> 
> The likely reason for the confusion is that they are already have
> an option for an "ordered array" dict variant that does a brute-force 
> linear search.  However, their normal hashed lookup is very
> similar to ours and is easily amenable to being compact and ordered.
> 
> See:
> https://github.com/micropython/micropython/blob/77a48e8cd493c0b0e0ca2d2ad58a110a23c6a232/py/map.c#L139
> 
> Raymond has also volunteered to assist with this.

I tried to do that, let me summarize previous point and give IMHO re:
contributing an alternative:

MicroPython's dict implementation is optimized for the least
bookkeeping overhead, not performance on overlarge datasets. For the
heap sizes we target (64KB on average), that's a good choice (put it
differently, MicroPython's motto is "system's memory (all bunch
kilobytes of it) belongs to user, not to MicroPython").

Adding insertion order would either:

1. Lead to significant (several times) slowdown, or

2. To noticeable memory overhead. Note that MicroPython uses the
absolute minimum for a dictionary entry - 2 words (key and value).
Adding even one extra word (e.g. some indirection pointer) means
increasing overhead by 50%, cutting useful user memory size by third.


With all that in mind, MicroPython is a very configurable project (215
config options as of this moment). We can have a config option for dict
implementation too.


But, the points above still hold - MicroPython targets low-memory
systems and doesn't really target plenty-of-memory systems (there was
never an aim to compete with CPython, the aim was to bring Python
(the language) where CPython could never go). Put it another way,
the alternative dict implementation is not expected to be used by
default.

If, with all the above in mind, someone, especially a CPython developer,
wants to contribute an alternative dict implementation, it would be
gladly accepted. (Note that if CPython followed the same policy, i.e.
allowed compile-time selection of old vs new dict algorithm, we
wouldn't have this thread.)

(Disclaimer: all the above is just my IMHO as a long-time contributor,
I'm not a MicroPython BDFL).

And I really appreciate all the attention to MicroPython - that's the
biggest case on my memory on python-dev.


[]

-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 7, 2017, at 05:44, Paul Moore  wrote:

> If you're a user and your application developer didn't do (1) or a
> library developer developing one of the libraries your application
> developer chose to use didn't do (2), you're hosed. If you're a user
> who works in an environment where moving to a new version of the
> application is administratively complex, you're hosed.

“hosed” feels like too strong of a word here.  DeprecationWarnings usually 
don’t break anything.  Sure, they’re annoying but they can usually be easily 
ignored.

Yes, there are some situations where DWs do actively break things (as I’ve 
mentioned, some Debuntu build/test environments).  But those are also 
relatively easier to silence, or at least the folks running those environments, 
or writing the code for those environments, are usually more advanced 
developers for whom setting an environment variable or flag isn’t that big of a 
deal.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 7, 2017, at 01:51, Petr Viktorin  wrote:
> 
> When I explained this in 3.5, dicts rearranging themselves seemed quite weird 
> to the newcomers.
> This year, I'm not looking forward to saying that dicts behave "intuitively", 
> but you shouldn't rely on that, because they're theoretically allowed to 
> rearrange themselves.
> The concept of "implementation detail" and language spec vs. multiple 
> interpreter implementations isn't easy to explain to someone in a "basic 
> coding literacy" course.

Perhaps, but IME, it’s not hard to teach someone that in a code review.  Today, 
if I see someone submit a change that includes an implicit assumption about 
ordering, I’ll call that out.  I can say “you can’t rely on dicts being 
ordered, so if that’s what you want, use an OrderedDict or sort your test 
data”.  That’s usually a localized observation, meaning, I can look at the diff 
and see that they are assuming dict iteration ordering.

What happens when built-in dict’s implementation behavior becomes a language 
guarantee?  Now the review is much more difficult because I probably won’t be 
able to tell just from a diff whether the ordering guarantees are preserved.  
Do they delete a key somewhere?  Who knows?   I’m not even sure that would be 
statically determinable since I’d have to trace the use of that dictionary at 
run time to see if some “del d[key]” is deleting the key in the dict under 
review or not.  I can probably only tell that at run time.

So how to I accurately review that code?  Is the order presumption valid or 
invalid?

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nathaniel Smith]

On Nov 7, 2017 06:24, "Nick Coghlan"  wrote:

On 7 November 2017 at 19:30, Paul Moore  wrote:
> On 7 November 2017 at 04:09, Nick Coghlan  wrote:
>> Given the status quo, how do educators learn that the examples they're
>> teaching to their students are using deprecated APIs?
>
> By reading the documentation on what they are teaching, and by testing
> their examples with new versions with deprecation warnings turned on?
> Better than having warnings appear the first time they run a course
> with a new version of Python, surely?
>
> I understand the "but no-one actually does this" argument. And I
> understand that breakage as a result is worse than a few warnings. But
> enabling deprecation warnings by default feels to me like favouring
> the developer over the end user. I remember before the current
> behaviour was enabled and it was *immensely* frustrating to try to use
> 3rd party code and get a load of warnings. The only options were:
>
> 1. Report the bug - usually not much help, as I want to run the
> program *now*, not when a new release is made.
> 2. Fix the code (and ideally submit a PR upstream) - I want to *use*
> the program, not debug it.
> 3. Find the right setting/environment variable, and tweak how I call
> the program to apply it - which doesn't fix the root cause, it's just
> a workaround.

Yes, this is why I've come around to the view that we need to come up
with a viable definition of "third party code" and leave deprecation
warnings triggered by that code disabled by default.

My suggestion for that definition is to have the *default* meaning of
"third party code" be "everything that isn't __main__".


That way, if you get a deprecation warning at the REPL, it's
necessarily because of something *you* did, not because of something a
library you called did. Ditto for single file scripts.


IPython actually made this change a few years ago; since 2015 I think it
has shown DeprecationWarnings by default if they're triggered by __main__.

It's helpful but I haven't noticed it eliminating this problem. One
limitation in particular is that it requires that the warnings are
correctly attributed to the code that triggered them, which means that
whoever is issuing the warning has to set the stacklevel= correctly, and
most people don't. (The default of stacklevel=1 is always wrong for
DeprecationWarning.) Also, IIRC it's actually impossible to set the
stacklevel= correctly when you're deprecating a whole module and issue the
warning at import time, because you need to know how many stack frames the
import system uses.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 6, 2017, at 22:33, Steven D'Aprano  wrote:

> I think it would be reasonable to say that builtin dicts only maintain
> insertion order for insertions, lookups, and changing the value. Any
> mutation which deletes keys may arbitrarily re-order the dict.
> 
> If the user wants a stronger guarantee, then they should use
> OrderedDict.

In fact, that *is* leaking CPython’s implementation into the language 
specification.  If by chance CPython’s implementation preserved order even 
after key deletion, either now or in the future, would that be defined as the 
ordering guarantees for built-in dict in the Python Language Reference?

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Barry Warsaw]

On Nov 7, 2017, at 07:12, Antoine Pitrou  wrote:

> The problem is this is taking things to a level of precision that makes
> the guarantee tedious to remember and reason about.
> 
> The only thing that's friendly to (non-expert) users is either "dicts
> are always ordered [by insertion order], point bar" or "dicts are not
> ordered, point bar".  Anything in-between, with reservations depending
> on which operations are invoked and when, is not really helpful to the
> average (non-expert) user.
> 
> Which is why I think the user-friendliness argument does not apply if
> order ceases to be guaranteed after __del__ is called.

That’s a very important point.  If it’s difficult to explain, teach, and retain 
the different ordering guarantees between built-in dict and OrderedDict, it 
might in fact be better to not guarantee any ordering for built-in dict *in the 
language specification*.  Otherwise we might need a section as big as chapter 5 
in the Python Language Reference just to dict ordering semantics. ;)

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[INADA Naoki]

>> > If further guarantees are proposed, perhaps it would be a good idea to
>> > open a new thread and state what exactly is being proposed.
>>
>> "Insertion ordered until the first key removal" is the only guarantee
>> that's being proposed.
>
> Is it?  It seems to me that many arguments being made are only relevant
> under the hypothesis that insertion is ordered even after the first key
> removal.  For example the user-friendliness argument, for I don't
> think it's very user-friendly to have a guarantee that disappears
> forever on the first __del__.
>

I agree with Antoine.  It's "hard to explain" than "preserving insertion order".

Dict performance is important because it's used for namespace.
But delete-heavy workload is not happen for namespace.

It may make workloads like LRU caching slightly.
But I don't think performance gain is large enough.  Many overhead
comes from API layer wrapping LRU cache. (e.g. functools.lru_cache)

So I expect performance difference can be found only on some
micro benchmarks.

Additionally, class namespace should keep insertion order.  It's language
spec from 3.6.  So we should have two mode for such optimization.
It makes dict more complicated.

So I'm +0.5 on making dict order as language spec, and -1 on "preserves
insertion order until deletion" idea.

But my expect may be wrong.  Serhiy is working on it so I'm waiting it
to benchmark.

Regards,

INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Serhiy Storchaka]

07.11.17 16:56, Steven D'Aprano пише:

To clarify: if we start with an empty dict, add keys A...D, delete B,
then add E...H, we could expect:

{A: 1}
{A: 1, B: 2}
{A: 1, B: 2, C: 3}
{A: 1, B: 2, C: 3, D: 4}
{D: 4, A: 1, C: 3}  # some arbitrary reordering
{D: 4, A: 1, C: 3, E: 5}
{D: 4, A: 1, C: 3, E: 5, F: 6}
{D: 4, A: 1, C: 3, E: 5, F: 6, G: 7}
{D: 4, A: 1, C: 3, E: 5, F: 6, G: 7, H: 8}


Rather

{A: 1, D: 4, C: 3}  # move the last item in place of removed
{A: 1, D: 4, C: 3, E: 5}
{A: 1, D: 4, C: 3, E: 5, F: 6}
{A: 1, D: 4, C: 3, E: 5, F: 6, G: 7}
{A: 1, D: 4, C: 3, E: 5, F: 6, G: 7, H: 8}

or

{A: 1, C: 3, D: 4}
{A: 1, E: 5, C: 3, D: 4}  # place the new item in place of removed
{A: 1, E: 5, C: 3, D: 4, F: 6}
{A: 1, E: 5, C: 3, D: 4, F: 6, G: 7}
{A: 1, E: 5, C: 3, D: 4, F: 6, G: 7, H: 8}

or

{A: 1, C: 3, D: 4}
{A: 1, C: 3, D: 4, E: 5}  # add new items at end until fill the array
{A: 1, F: 6, C: 3, D: 4, E: 5}  # and fill holes after that
{A: 1, F: 6, C: 3, D: 4, E: 5, G: 7}  # reallocate the array
{A: 1, F: 6, C: 3, D: 4, E: 5, G: 7, H: 8}

These scenarios are more probably.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[David Mertz]

On Nov 6, 2017 9:11 PM, "Raymond Hettinger" 
wrote:

> On Nov 6, 2017, at 8:05 PM, David Mertz  wrote:
> I strongly opposed adding an ordered guarantee to regular dicts. If the
implementation happens to keep that, great. Maybe OrderedDict can be
rewritten to use the dict implementation. But the evidence that all
implementations will always be fine with this restraint feels poor, and we
have a perfectly good explicit OrderedDict for those who want that.

I think this post is dismissive of the value that users would get from
having reliable ordering by default.


Dismissive seems like an overly strong word. I recognize I disagree with
Raymond on best official semantics. Someone else points out that if someday
an "even more efficient unordered dict" is discovered, user-facing "dict"
doesn't strictly have to be the same data structure as "internal dict". The
fact they are is admittedly an implementation detail also.

I've had all those same uses about round-tripping serialization that
Raymond mentions. I know the standard work arounds (which are not
difficult, but DO require a little extra code if we don't have order).

But like Raymond, I make most of my living TEACHING Python. I feel like the
extra order guarantee would make teaching slightly harder. I'm sure he
feels contrarily. It is true that with 3.6 I can no longer show an example
where the dict display is oddly changed when printed. But then, unordered
sets also wind up sorting small integers on printing, even though that's
not a guarantee.

Ordering by insertion order (possibly "only until first deletion") is
simply not obvious to beginners. If we had, hypothetically, a dict that
"always alphabetized keys" that would be more intuitive to them, for
example. Insertion order feels obvious to us experts, but it really is an
extra cognitive burden to learners beyond understanding "key/Val
association".

Having worked with Python 3.6 for a while, it is repeatedly delightful to
encounter the effects of ordering.  When debugging, it is a pleasure to be
able to easily see what has changed in a dictionary.  When creating XML, it
is joy to see the attribs show in the same order you added them.  When
reading a configuration, modifying it, and writing it back out, it is a
godsend to have it written out in about the same order you originally typed
it in.  The same applies to reading and writing JSON.  When adding a VIA
header in a HTTP proxy, it is nice to not permute the order of the other
headers. When generating url query strings for REST APIs, it is nice have
the parameter order match documented examples.

We've lived without order for so long that it seems that some of us now
think data scrambling is a virtue.  But it isn't.  Scrambled data is the
opposite of human friendly.


Raymond


P.S. Especially during debugging, it is often inconvenient, difficult, or
impossible to bring in an OrderedDict after the fact or to inject one into
third-party code that is returning regular dicts.  Just because we have
OrderedDict in collections doesn't mean that we always get to take
advantage of it.  Plain dicts get served to us whether we want them or not.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Thomas Nyberg]

On 11/07/2017 09:00 AM, Wolfgang wrote:
> Hi,
> 
> I have to admit sometimes I don't understand why small things produce so much
> mail traffic. :-)
> 
> If I use a mapping like dict most of the time I don't care if it is ordered.
> If I need an ordering I use OrderedDict. In a library if I need a dict to be
> ordered most of the time there is a parameter allowing me to do this or to
> specify the used class as dict.
> If this is not the case it can be fixed. In rare cases a workaround is needed.
> 
> As of now I have dict and OrderedDict, it is clear and explicit.
> No need to change.
> 
>   [...]
>
> Regards,
> 
> Wolfgang

I feel a bit out of place on this list since I'm a lurker and not a core
developer, but I just wanted to add my agreement with this as well. So
maybe take my opinion with a grain of salt...

I agree with Wolfgang. I just don't understand why this change is
needed. We have dict and we have OrderedDict. Why does dict need to
provide the extra ordering constraints? I've read many posts in this
discussion and find none of them convincing. Python already guarantees
things like ordering of keyword arguments. I've seen some people point
out confusion of newcomers (e.g. they are surprised when order is not
surprised), but that just seems to me natural confusion that comes about
when learning. I would argue that a better solution to that problem is
exactly the go solution: i.e. purposely perturbing the ordering in a way
that shows up immediately so that users realize the problems in their
thinking earlier. The dict provides a mapping from key to value. I
personally think that that is mentally much simpler object than a
mapping from key to value with certain ordering guarantees. If I want to
extra guarantees I import OrderedDict and read what the guarantees are.
This seems totally fine to me. I don't really see any advantages to this
change but a lack of implementation flexibility and a more complicated
core object in Python.

Cheers,
Thomas
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Evpok Padding]

Hello,

I agree with Wolfgang here. From what I gathered of the discussion, the
argument started from « It would be nice if dict litterals returned ordered
dicts instead of an unordered ones », which is mostly a good thing, as it
allows e.g. `OrderedDict({'spam': 'ham', 'sausages': 'eggs'})` instead of
having to rely on lists of couples to create an OrderedDict. It is not of
utmost utility, but it would be nice to have and not dissimilar to what we
already have with kwargs being ordered dicts, also a matter of slightly
better usability. Possibly, in order to avoid implying that all dicts
guarantee ordering at all time, a syntax such as `o{}` might be used,
mostly to help newcomers. So far, so good.

Where it started to go all haywire is when it became conflated that with
the fact that CPython and Pypy dicts are actually ordered (up to a point at
least) and it suddenly became « Let's guarantee the ordering of all dicts »
which apparently is an issue for at least one implementation of Python, and
still have to be implemented in several others (said implementation would
be trivial, or so it is said, but it still has to be written, along with
appropriate tests, regression checks…). So far, the arguments I have seen
for that are

  1. It is useful in context where one has to guarantee the ordering of
some mapping (such as in json)
  2. It is useful in contexts where ordering is facultative, but nice to
have (debugging has been mentionned)
  3. It is already this way in CPython, so people are going to use that
anyway

I take issue with all of those arguments.

  1. If the ordered should be guaranteed, why would it be so hard to use
OrderedDict ?
- Just write `d = OrderedDict(((key, val) for key, value in …))`
instead of `{key: value for key, value in …}`. It is not that hard and at
least it is explicit that the order is important. And if it is really so
hard, we could have dict comprehensions be ordered too in addition to
litterals, it still doesn't mean that dicts should all be ordered
- It is even easier if you fill your dict value-per-value, just
initialise it as `d = OrderedDict` instead of `d = {}` and voilà !
  2. I would like to see some examples of cases where this is really much
more convenient than any other soution, but even then I suspect that these
cases are not sufficently relevant to wed all Python backends to ordered
dicts forever.
  3. This is just a pure fallacy. The language has a documented API that
says that if order of insertion is important, you should explicitely use an
OrderedDict. If people stray away from it and use implementation details
such as the ordering of dict in CPython, they are on their own and
shouldn't expect it to be portable to any other version. Again, it's not as
if OrderedDict did not exist or was much more inconvenient to use than dict.

Also, since the proposed implementation does not keep ordering on deletion,
those relying implicitely on the ordering of dicts without reading the docs
might get bitten by it later in much more subtle ways.
Note that I don't sugest mandatory shuffling of dicts to advertise their
non-guaranteed ordering, either. Just that reading the docs (or having your
instructor tell you that dict does not guarantee the order) is the
reponsibility of the end user.

To sum it up

  - Ordered dict litterals are a nice idea, but probably not that
important. If it happens, it would be nice if it could be extended to dict
comprehensions, though.
  - Guaranteeing the ordering of all `dicts` does not make a lot of sense
  - Changing the API to guarantee the order of dicts **is** an API change,
which still means work

Am I missing something ?

Cheers,

E

On 7 November 2017 at 10:51, Petr Viktorin  wrote:

> On 11/07/2017 09:00 AM, Wolfgang wrote:
> [...]
>
>> Also it is fine to teach people that dict (Mapping) is not ordered
>> but CPython has an implementation detail and it is ordered.
>> But if you want the guarantee use OrderedDict.
>>
>
> I don't think that is fine.
> When I explained this in 3.5, dicts rearranging themselves seemed quite
> weird to the newcomers.
> This year, I'm not looking forward to saying that dicts behave
> "intuitively", but you shouldn't rely on that, because they're
> theoretically allowed to rearrange themselves.
> The concept of "implementation detail" and language spec vs. multiple
> interpreter implementations isn't easy to explain to someone in a "basic
> coding literacy" course.
>
> Today I can still show an example on Python 3.5. But most people I teach
> today won't run their code on 3.5, or on MicroPython or Brython, and quite
> soon they'll forget that there's no dict ordering guarantee.
>
> Also: I happen to read python-dev and the language docs. I suspect not all
> teachers do, and when they see that dict order randomization was "fixed",
> they might just remove the explanation from the lesson and teach something
> practical instead.
>
> ___
> 

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Antoine Pitrou]

On Tue, 7 Nov 2017 09:44:07 -0500
Yury Selivanov  wrote:
> 
> One common pattern that I see frequently is this:
> 
>  def foo(**kwargs):
>   kwargs.pop('somekey', None)
>   bar(**kwargs)

I see it frequently too, but that's in code meant to be
Python 2-compatible (and therefore cannot count on any ordering
guarantee, even de facto). On Python 3 you can write:

   def foo(somekey=None, **kwargs):
   # do something with somekey?
   bar(**kwargs)

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Antoine Pitrou]

On Wed, 8 Nov 2017 01:56:42 +1100
Steven D'Aprano  wrote:
> 
> I think that Nick's intent was not to say that after a single deletion, 
> the ordering guarantee goes away "forever", but that a deletion may be 
> permitted to reorder the keys, after which further additions will honour 
> insertion order. At least, that's how I interpret him.

The problem is this is taking things to a level of precision that makes
the guarantee tedious to remember and reason about.

The only thing that's friendly to (non-expert) users is either "dicts
are always ordered [by insertion order], point bar" or "dicts are not
ordered, point bar".  Anything in-between, with reservations depending
on which operations are invoked and when, is not really helpful to the
average (non-expert) user.

Which is why I think the user-friendliness argument does not apply if
order ceases to be guaranteed after __del__ is called.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Steven D'Aprano]

On Tue, Nov 07, 2017 at 03:32:29PM +0100, Antoine Pitrou wrote:

[...]
> > "Insertion ordered until the first key removal" is the only guarantee
> > that's being proposed.
> 
> Is it?  It seems to me that many arguments being made are only relevant
> under the hypothesis that insertion is ordered even after the first key
> removal.  For example the user-friendliness argument, for I don't
> think it's very user-friendly to have a guarantee that disappears
> forever on the first __del__.

Don't let the perfect be the enemy of the good.

For many applications, keys are never removed from the dict, so this 
doesn't matter. If you never delete a key, then the remaining keys will 
never be reordered.

I think that Nick's intent was not to say that after a single deletion, 
the ordering guarantee goes away "forever", but that a deletion may be 
permitted to reorder the keys, after which further additions will honour 
insertion order. At least, that's how I interpret him.

To clarify: if we start with an empty dict, add keys A...D, delete B, 
then add E...H, we could expect:

{A: 1}
{A: 1, B: 2}
{A: 1, B: 2, C: 3}
{A: 1, B: 2, C: 3, D: 4}
{D: 4, A: 1, C: 3}  # some arbitrary reordering
{D: 4, A: 1, C: 3, E: 5}
{D: 4, A: 1, C: 3, E: 5, F: 6}
{D: 4, A: 1, C: 3, E: 5, F: 6, G: 7}
{D: 4, A: 1, C: 3, E: 5, F: 6, G: 7, H: 8}


Nick, am I correct that this was your intent?



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Yury Selivanov]

On Tue, Nov 7, 2017 at 9:32 AM, Antoine Pitrou  wrote:
> On Wed, 8 Nov 2017 00:01:04 +1000
> Nick Coghlan  wrote:
>
>> On 7 November 2017 at 23:48, Stefan Krah  wrote:
>> >
>> >
>> > This is just a reminder that the current dict is not an "OrderedDict":
>> >
>>  from collections import OrderedDict
>>  OrderedDict(a=0, b=1) == OrderedDict(b=1, a=0)
>> > False
>>  dict(a=0, b=1) == dict(b=1, a=0)
>> > True
>> >
>> > The recent proposal was primarily about guaranteeing the insertion order of
>> > dict literals.
>> >
>> > If further guarantees are proposed, perhaps it would be a good idea to
>> > open a new thread and state what exactly is being proposed.
>>
>> "Insertion ordered until the first key removal" is the only guarantee
>> that's being proposed.
>
> Is it?  It seems to me that many arguments being made are only relevant
> under the hypothesis that insertion is ordered even after the first key
> removal.  For example the user-friendliness argument, for I don't
> think it's very user-friendly to have a guarantee that disappears
> forever on the first __del__.

One common pattern that I see frequently is this:

 def foo(**kwargs):
  kwargs.pop('somekey', None)
  bar(**kwargs)

With ordering breaking on first pop/delete we essentially have no
guarantee about kwargs order, or at least it's very easy to break.  It
would make writing wrappers like this extremely tedious -- we are
essentially forcing people to use OrderedDict to just pop an item from
kwargs.  Not to mention that this isn't cheap in terms of performance.

Is there a *real* motivation for saying that pop/delete can break the order?

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Toshio Kuratomi]

On Nov 7, 2017 5:47 AM, "Paul Moore"  wrote:

On 7 November 2017 at 13:35, Philipp A.  wrote:
> Sorry, I still don’t understand how any of this is a problem.
>
> If you’re an application developer, google “python disable
> DeprecationWarning” and paste the code you found, so your users don’t see
> the warnings.
> If you’re a library developer, and a library you depend on raises
> DeprecationWarnings without it being your fault, file an issue/bug there.
>
> For super-increased convenience in case 2., we could also add a
convenience
> API that blocks deprecation warnings raised from certain module or its
> submodules.
> Best, Philipp

If you're a user and your application developer didn't do (1) or a
library developer developing one of the libraries your application
developer chose to use didn't do (2), you're hosed. If you're a user
who works in an environment where moving to a new version of the
application is administratively complex, you're hosed.

As I say, the proposal prioritises developer convenience over end user
experience.


I don't agree with this characterisation.  Even if we assume a user isn't
going to fix a DeprecationWarning they still benefit: (1) if they're a
sysadmin it will warn them that they need to be careful when upgrading a
dependency. (2) if the developer never hears about the DeprecationWarning
then it is ultimately the user who suffers when the tool they depend on
breaks without warning so seeing and reporting the DeprecationWarning helps
the end user. (3) if DeprecationWarnings are allowed to linger through
multiple releases, it may tell the user about the quality of the software
they're using.

More information is helpful to end users.  Developers are actually the ones
that it inconveniences as we'll be the ones grumbling when an end user who
hasn't evaluated the deprecation cycles of upstream projects as we have
demand immediate changes for deprecations that are still years away from
causing problems.  But unlike end users, we do have the ability to solve
that by turning those deprecations off in our code if we've done our due
diligence (or even if we haven't done our due diligence).

-Toshio
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Antoine Pitrou]

On Wed, 8 Nov 2017 00:01:04 +1000
Nick Coghlan  wrote:

> On 7 November 2017 at 23:48, Stefan Krah  wrote:
> >
> >
> > This is just a reminder that the current dict is not an "OrderedDict":
> >  
>  from collections import OrderedDict
>  OrderedDict(a=0, b=1) == OrderedDict(b=1, a=0)  
> > False  
>  dict(a=0, b=1) == dict(b=1, a=0)  
> > True
> >
> > The recent proposal was primarily about guaranteeing the insertion order of
> > dict literals.
> >
> > If further guarantees are proposed, perhaps it would be a good idea to
> > open a new thread and state what exactly is being proposed.  
> 
> "Insertion ordered until the first key removal" is the only guarantee
> that's being proposed.

Is it?  It seems to me that many arguments being made are only relevant
under the hypothesis that insertion is ordered even after the first key
removal.  For example the user-friendliness argument, for I don't
think it's very user-friendly to have a guarantee that disappears
forever on the first __del__.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Paul Moore]

On 7 November 2017 at 14:21, Ethan Furman  wrote:
> On 11/07/2017 05:44 AM, Paul Moore wrote:
>
>> If you're a user and your application developer didn't do (1) or a
>> library developer developing one of the libraries your application
>> developer chose to use didn't do (2), you're hosed. If you're a user
>> who works in an environment where moving to a new version of the
>> application is administratively complex, you're hosed.
>
> Suffering from DeprecationWarnings is not "being hosed".  Having your
> script/application/framework suddenly stop working because nobody noticed
> something was being deprecated is "being hosed".

OK, I overstated. Apologies. My recollection is of a lot more end user
complaints when deprecation warnings were previously switched on than
others seem to remember, but I can't find hard facts, so I'll assume
I'm misremembering.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Ethan Furman]

On 11/07/2017 05:44 AM, Paul Moore wrote:


If you're a user and your application developer didn't do (1) or a
library developer developing one of the libraries your application
developer chose to use didn't do (2), you're hosed. If you're a user
who works in an environment where moving to a new version of the
application is administratively complex, you're hosed.


Suffering from DeprecationWarnings is not "being hosed".  Having your script/application/framework suddenly stop working 
because nobody noticed something was being deprecated is "being hosed".


+1 to turn them back on.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Philipp A.]

Nick Coghlan  schrieb am Di., 7. Nov. 2017 um 14:57 Uhr:

> Users of applications written in Python are not python-dev's users:
> they're the users of those applications, and hence the quality of that
> experience is up to the developers of those applications. […]
>

Thank you, that’s exactly what I’m talking about. Besides: Nobody is
“hosed”… There will be one occurrence of every DeprecationWarning in the
stderr of the application. Hardly the end of the world for CLI applications
and even invisible for GUI applications.

If the devs care about the user not seeing any warnings in their CLI
application, they’ll have a test set up for that, which will tell them that
the newest python-dev would raise a new warning, once they turn on testing
for that release. That’s completely fine!

Explicit is better than implicit! If I know lib X raises
DeprecationWarnings I don’t care about, I want to explicitly silence them,
instead of missing out on all the valuable information in other
DeprecationWarnings.

Best, Philipp
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Stefan Krah]

On Wed, Nov 08, 2017 at 12:01:04AM +1000, Nick Coghlan wrote:
> > The recent proposal was primarily about guaranteeing the insertion order of
> > dict literals.
> >
> > If further guarantees are proposed, perhaps it would be a good idea to
> > open a new thread and state what exactly is being proposed.
> 
> "Insertion ordered until the first key removal" is the only guarantee
> that's being proposed.
> 
> OrderedDict just comes into the discussion because reaching for its
> stronger guarantees is currently the only way to obtain that guarantee
> in a formally implementation-independent and future-proof way.

Ok good, I was primarily worried about collections.UnorderedDict coming up
and users thinking that OrderedDict could be replaced entirely by dict().



Stefan Krah




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Nick Coghlan]

On 7 November 2017 at 23:48, Stefan Krah  wrote:
>
>
> This is just a reminder that the current dict is not an "OrderedDict":
>
 from collections import OrderedDict
 OrderedDict(a=0, b=1) == OrderedDict(b=1, a=0)
> False
 dict(a=0, b=1) == dict(b=1, a=0)
> True
>
> The recent proposal was primarily about guaranteeing the insertion order of
> dict literals.
>
> If further guarantees are proposed, perhaps it would be a good idea to
> open a new thread and state what exactly is being proposed.

"Insertion ordered until the first key removal" is the only guarantee
that's being proposed.

OrderedDict just comes into the discussion because reaching for its
stronger guarantees is currently the only way to obtain that guarantee
in a formally implementation-independent and future-proof way.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nick Coghlan]

On 7 November 2017 at 23:44, Paul Moore  wrote:
> As I say, the proposal prioritises developer convenience over end user
> experience.

Users of applications written in Python are not python-dev's users:
they're the users of those applications, and hence the quality of that
experience is up to the developers of those applications. This is no
different from the user experience of Instagram being Facebook's
problem, the user experience of RHEL being Red Hat's problem, the user
experience of YouTube being Google's problem, etc.

*python-dev's* users are developers, data analysts, educators, and so
forth that are actually writing Python code, and at the moment we're
making it hard for them to be suitably forewarned of upcoming breaking
changes - they have to know the secret knock that says "I'd like to be
warned about future breaking changes, please". Sure, a lot of people
do learn what that knock is, and they often even remember to ask for
it, but the entire reason this thread started was because *I* forgot
that I needed to run "python3 -Wd" in order to check for async/await
deprecation warnings in 3.6, and incorrectly assumed that their
absence meant we'd forgotten to include them.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] The current dict is not an "OrderedDict"

[Stefan Krah]

This is just a reminder that the current dict is not an "OrderedDict":

>>> from collections import OrderedDict
>>> OrderedDict(a=0, b=1) == OrderedDict(b=1, a=0)
False
>>> dict(a=0, b=1) == dict(b=1, a=0)
True


The recent proposal was primarily about guaranteeing the insertion order of
dict literals.

If further guarantees are proposed, perhaps it would be a good idea to
open a new thread and state what exactly is being proposed.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Paul Moore]

On 7 November 2017 at 13:35, Philipp A.  wrote:
> Sorry, I still don’t understand how any of this is a problem.
>
> If you’re an application developer, google “python disable
> DeprecationWarning” and paste the code you found, so your users don’t see
> the warnings.
> If you’re a library developer, and a library you depend on raises
> DeprecationWarnings without it being your fault, file an issue/bug there.
>
> For super-increased convenience in case 2., we could also add a convenience
> API that blocks deprecation warnings raised from certain module or its
> submodules.
> Best, Philipp

If you're a user and your application developer didn't do (1) or a
library developer developing one of the libraries your application
developer chose to use didn't do (2), you're hosed. If you're a user
who works in an environment where moving to a new version of the
application is administratively complex, you're hosed.

As I say, the proposal prioritises developer convenience over end user
experience.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Philipp A.]

Sorry, I still don’t understand how any of this is a problem.

   1. If you’re an application developer, google “python disable
   DeprecationWarning” and paste the code you found, so your users don’t see
   the warnings.
   2. If you’re a library developer, and a library you depend on raises
   DeprecationWarnings without it being your fault, file an issue/bug there.

For super-increased convenience in case 2., we could also add a convenience
API that blocks deprecation warnings raised from certain module or its
submodules.
Best, Philipp

Nick Coghlan  schrieb am Di., 7. Nov. 2017 um 13:25 Uhr:

> On 7 November 2017 at 19:30, Paul Moore  wrote:
> > On 7 November 2017 at 04:09, Nick Coghlan  wrote:
> >> Given the status quo, how do educators learn that the examples they're
> >> teaching to their students are using deprecated APIs?
> >
> > By reading the documentation on what they are teaching, and by testing
> > their examples with new versions with deprecation warnings turned on?
> > Better than having warnings appear the first time they run a course
> > with a new version of Python, surely?
> >
> > I understand the "but no-one actually does this" argument. And I
> > understand that breakage as a result is worse than a few warnings. But
> > enabling deprecation warnings by default feels to me like favouring
> > the developer over the end user. I remember before the current
> > behaviour was enabled and it was *immensely* frustrating to try to use
> > 3rd party code and get a load of warnings. The only options were:
> >
> > 1. Report the bug - usually not much help, as I want to run the
> > program *now*, not when a new release is made.
> > 2. Fix the code (and ideally submit a PR upstream) - I want to *use*
> > the program, not debug it.
> > 3. Find the right setting/environment variable, and tweak how I call
> > the program to apply it - which doesn't fix the root cause, it's just
> > a workaround.
>
> Yes, this is why I've come around to the view that we need to come up
> with a viable definition of "third party code" and leave deprecation
> warnings triggered by that code disabled by default.
>
> My suggestion for that definition is to have the *default* meaning of
> "third party code" be "everything that isn't __main__".
>
> That way, if you get a deprecation warning at the REPL, it's
> necessarily because of something *you* did, not because of something a
> library you called did. Ditto for single file scripts.
>
> We'd then offer some straightforward interfaces for people to say
> "Please also report legacy calls from 'module' as warnings".
>
> You'd still get less-than-helpful warnings if you were running a
> single file script that someone *else* wrote (rather than one you
> wrote yourself), but that's an inherent flaw in that distribution
> model: as soon as you ask people to supply their own Python runtime,
> you're putting them in the position of acting as an application
> integrator (finding a combination of Python language runtime and your
> script that actually work together), rather than as a regular software
> user.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/flying-sheep%40web.de
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Jan Claeys]

On Tue, 2017-11-07 at 16:39 +1000, Nick Coghlan wrote:
> And this is the key point for me: "choosing not to choose" is
> effectively the same as standardising the feature, as enough Python
> code will come to rely on CPython's behaviour that most alternative
> implementations will feel obliged to start behaving the same way
> CPython does (with MicroPython being the potential exception due to
> memory usage constraints always winning over algorithmic efficiency
> concerns in that context).

Maybe an UnorderedDict could be added which Python implementations
_can_ implement as an optimized (less memory use, faster, ...) version
without ordering guarantees if they have a need for it.  In other
implementations it could just be a synonym for a regular dict.

That way it would be explicit that the programmer doesn't care about
the ordered behaviour.  It would also avoid current mistakes some
(especially those new to the language and occasional users) make,
because of assumptions from default dict behaviour.

(Maybe a commandline switch or other mechanisms to explicitly use that
UnorderedDict as the default could also be useful.  It would be a no-op 
in implementations which don't have differing implementations.)



-- 
Jan Claeys

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nick Coghlan]

On 7 November 2017 at 19:30, Paul Moore  wrote:
> On 7 November 2017 at 04:09, Nick Coghlan  wrote:
>> Given the status quo, how do educators learn that the examples they're
>> teaching to their students are using deprecated APIs?
>
> By reading the documentation on what they are teaching, and by testing
> their examples with new versions with deprecation warnings turned on?
> Better than having warnings appear the first time they run a course
> with a new version of Python, surely?
>
> I understand the "but no-one actually does this" argument. And I
> understand that breakage as a result is worse than a few warnings. But
> enabling deprecation warnings by default feels to me like favouring
> the developer over the end user. I remember before the current
> behaviour was enabled and it was *immensely* frustrating to try to use
> 3rd party code and get a load of warnings. The only options were:
>
> 1. Report the bug - usually not much help, as I want to run the
> program *now*, not when a new release is made.
> 2. Fix the code (and ideally submit a PR upstream) - I want to *use*
> the program, not debug it.
> 3. Find the right setting/environment variable, and tweak how I call
> the program to apply it - which doesn't fix the root cause, it's just
> a workaround.

Yes, this is why I've come around to the view that we need to come up
with a viable definition of "third party code" and leave deprecation
warnings triggered by that code disabled by default.

My suggestion for that definition is to have the *default* meaning of
"third party code" be "everything that isn't __main__".

That way, if you get a deprecation warning at the REPL, it's
necessarily because of something *you* did, not because of something a
library you called did. Ditto for single file scripts.

We'd then offer some straightforward interfaces for people to say
"Please also report legacy calls from 'module' as warnings".

You'd still get less-than-helpful warnings if you were running a
single file script that someone *else* wrote (rather than one you
wrote yourself), but that's an inherent flaw in that distribution
model: as soon as you ask people to supply their own Python runtime,
you're putting them in the position of acting as an application
integrator (finding a combination of Python language runtime and your
script that actually work together), rather than as a regular software
user.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Paul Moore]

On 7 November 2017 at 10:18, Steve Holden  wrote:
> On Tue, Nov 7, 2017 at 12:35 AM, Donald Stufft  wrote:
> [..]
>
>>
>> Maybe we just need to fully flesh out the idea of a "Python Core" (What
>> exists now as “Python”) and a “Python Platform” (Python Core + A select set
>> of preinstalled libraries). Then typing can just be part of the Python
>> Platform, and gets installed as part of your typical installation, but is
>> otherwise an independent piece of code.
>>
> Given that (type and other) annotations have been promoted as an optional
> feature of the language it seems unfair and perhaps unwise to add a
> dependency specifically to support
> them
>  to the stdlib and therefore the Python core.
> Since type annotations are, as Paul pointed out, development-time features,
> it would appear to behoove those wishing to use them to separate them in
> such a way that the software can be installed without annotations, and
> therefore without the need for the typing module. Assuming they would like
> to see the widest possible distribution, of course. For selected audiences I
> am sure typing will be de rigeur,

From my point of view, I take the same premise and come to the
opposite conclusion.

Because type annotations are a development-time feature, they should
*not* require a dependency in the final deployment (apart from Python
itself). However, because they are a language syntax feature they are
of necessity written in the application source. And type specification
of anything more complex than basic types (for example, List[int])
requires classes defined in the typing module. Therefore, typing must
be in the stdlib so that use of type annotations by the developer
doesn't impose a runtime dependency on the end user.

If there were a way of including type annotations that had no runtime
effect on the final deployed program, things would be different. But
the decision to make annotations part of the language syntax precludes
that. In particular, "it would appear to behoove those wishing to use
them to separate them" - there's no way of doing that *precisely*
because they are a language syntax feature.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Steve Holden]

On Tue, Nov 7, 2017 at 12:35 AM, Donald Stufft  wrote:
[..]


> Maybe we just need to fully flesh out the idea of a "Python Core" (What
> exists now as “Python”) and a “Python Platform” (Python Core + A select set
> of preinstalled libraries). Then typing can just be part of the Python
> Platform, and gets installed as part of your typical installation, but is
> otherwise an independent piece of code.
>
> Given that (type and other) annotations have been promoted as an optional
feature of the language it seems unfair and perhaps unwise to add a
dependency specifically to support
​them
 to the stdlib and therefore the Python core.
​ Since type annotations are, as Paul pointed out, development-time
features, it​ would appear to behoove those wishing to use them to separate
them in such a way that the software can be installed without annotations,
and therefore without the need for the typing module. Assuming they would
like to see the widest possible distribution, of course. For selected
audiences I am sure typing will be *de rigeur*,


​In this scenario, surely the most "typical" installation would be a
virtualenv. ​Do all virtualenvs rely on the same Platform? Who decides
which additional libraries are required for the Platform? Doesn't this just
add another "type of installation" distinction to confuse the unwary? How
do I maintain the Platform separately from the Core? Does my sysadmin
maintain either or both? I'd like to see a little more clarity about the
benefits such a schism would offer.

regards
 Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Moore]

On 7 November 2017 at 06:39, Nick Coghlan  wrote:
> On 7 November 2017 at 09:23, Chris Barker  wrote:
>> in short -- we don't have a choice (unless we add an explicit randomization
>> as some suggested -- but that just seems perverse...)
>
> And this is the key point for me: "choosing not to choose" is
> effectively the same as standardising the feature, as enough Python
> code will come to rely on CPython's behaviour that most alternative
> implementations will feel obliged to start behaving the same way
> CPython does (with MicroPython being the potential exception due to
> memory usage constraints always winning over algorithmic efficiency
> concerns in that context).

Personally, I think that having an ordered implementation and then
deliberately breaking that ordering is pretty silly. Not least because
we chose not to do that for 3.6. So we're left with the simple
question of whether we make the behaviour required in the
documentation (which is basically where this thread started). I see 3
options. First, we maintain the status quo, treat it as a CPython/PyPy
implementation detail, and accept that this means that people will
expect it - resulting in pressure on alternative implementations to
conform, but without a language spec to support them. Second, we
document the requirement in the language spec, requiring alternative
implementations to either implement it or document it as a way in
which they don't confirm to the language spec (which is unattractive
for them, as "implements the official Python language spec" is a
selling point). Or third, we could document it as an optional
behaviour, that language implementations are expected to implement if
possible, and if they can't they should document the variance. That is
to some extent the best of both worlds, in that it allows
implementations to claim conformance with the spec while still not
implementing the feature if it causes them problems to do so - they
just have to document that they don't implement this optional but
recommended behaviour. The downside of this option is that there's no
precedent for it in the Python spec (it's basically C's
"implementation defined behaviour", which is something Python hasn't
needed this far).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Petr Viktorin]

On 11/07/2017 09:00 AM, Wolfgang wrote:
[...]

Also it is fine to teach people that dict (Mapping) is not ordered
but CPython has an implementation detail and it is ordered.
But if you want the guarantee use OrderedDict.


I don't think that is fine.
When I explained this in 3.5, dicts rearranging themselves seemed quite 
weird to the newcomers.
This year, I'm not looking forward to saying that dicts behave 
"intuitively", but you shouldn't rely on that, because they're 
theoretically allowed to rearrange themselves.
The concept of "implementation detail" and language spec vs. multiple 
interpreter implementations isn't easy to explain to someone in a "basic 
coding literacy" course.


Today I can still show an example on Python 3.5. But most people I teach 
today won't run their code on 3.5, or on MicroPython or Brython, and 
quite soon they'll forget that there's no dict ordering guarantee.


Also: I happen to read python-dev and the language docs. I suspect not 
all teachers do, and when they see that dict order randomization was 
"fixed", they might just remove the explanation from the lesson and 
teach something practical instead.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Antoine Pitrou]

On Tue, 7 Nov 2017 09:30:19 +
Paul Moore  wrote:
> 
> I understand the "but no-one actually does this" argument. And I
> understand that breakage as a result is worse than a few warnings. But
> enabling deprecation warnings by default feels to me like favouring
> the developer over the end user.

I understand this characterization.

> I'd prefer it if rather than simply switching warnings on by default,
> we worked on making it easier for the people in a position to actually
> *fix* the issue (coders writing programs, educators developing
> training materials, etc) to see the warnings. For example, encourage
> the various testing frameworks (unittest, pytest, nose, tox, ...) to
> enable warnings by default,

pytest does nowadays.  That doesn't mean warnings get swiftly fixed,
though.  There are many reasons why (see my initial reply to Nick's
proposal).

> ensure that all new deprecations are
> documented in the "Porting to Python 3.x" notes, etc.

In my experience, Python deprecations are in the minority.
Most often you have to deal with deprecations in third-party libraries
rather than Python core/stdlib, because we (Python) are more reluctant
to change and deprecate APIs than the average library maintainer is.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Paul Moore]

On 7 November 2017 at 05:20, Ethan Smith  wrote:
> I'm not so keen on this because I think some things in typing (such as
> NamedTuple) probably deserve to be in the collections module. And some of
> the ABCs could probably also be merged with collections.abc but doing this
> correctly and not all at once would be quite difficult.
> I do think the typing concepts should be better integrated into the standard
> library. However, a fair amount of the clases you list (such as NamedTuple)
> are in of themselves dependent on parts of typing.

This is a good point. To what extent is it true that the stdlib
*already* uses the typing module internally, it's just that the usage
is hidden by the fact that the examples of this are in the typing
module - not because they "should" be there but simply because it
isolates the use of typing to that one module?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Paul Moore]

On 7 November 2017 at 00:35, Donald Stufft  wrote:
> Maybe we just need to fully flesh out the idea of a "Python Core" (What
> exists now as “Python”) and a “Python Platform” (Python Core + A select set
> of preinstalled libraries). Then typing can just be part of the Python
> Platform, and gets installed as part of your typical installation, but is
> otherwise an independent piece of code.

I quite like this idea, but at a minimum it would mean that the
Windows installers for Python should include "core" and "platform"
builds, with "platform" being the version that new users are directed
to. And I'm not sure python-dev would be comfortable with agreeing and
distributing what would essentially be a curated subset of PyPI
packages. The most recent discussion on "a recommended set of
packages" didn't get much further than 3 or 4 candidates, which hardly
constitutes a "platform".

Conversely, something like Anaconda could be considered as the "Python
Platform" (assuming they bundle typing...), but I'm not really
comfortable with python-dev aligning themselves that strongly with a
single distributor. (We're talking about a recommendation along the
lines of "people wanting to gain, for example, corporate approval for
"Python" should request approval for the Anaconda distribution rather
than the python.org distribution").

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Paul Moore]

On 7 November 2017 at 04:09, Nick Coghlan  wrote:
> Given the status quo, how do educators learn that the examples they're
> teaching to their students are using deprecated APIs?

By reading the documentation on what they are teaching, and by testing
their examples with new versions with deprecation warnings turned on?
Better than having warnings appear the first time they run a course
with a new version of Python, surely?

I understand the "but no-one actually does this" argument. And I
understand that breakage as a result is worse than a few warnings. But
enabling deprecation warnings by default feels to me like favouring
the developer over the end user. I remember before the current
behaviour was enabled and it was *immensely* frustrating to try to use
3rd party code and get a load of warnings. The only options were:

1. Report the bug - usually not much help, as I want to run the
program *now*, not when a new release is made.
2. Fix the code (and ideally submit a PR upstream) - I want to *use*
the program, not debug it.
3. Find the right setting/environment variable, and tweak how I call
the program to apply it - which doesn't fix the root cause, it's just
a workaround.

I appreciate that this is open source, and using free programs comes
with an obligation to contribute back or deal with issues like this,
but even so, it's a pretty bad user experience.

I'd prefer it if rather than simply switching warnings on by default,
we worked on making it easier for the people in a position to actually
*fix* the issue (coders writing programs, educators developing
training materials, etc) to see the warnings. For example, encourage
the various testing frameworks (unittest, pytest, nose, tox, ...) to
enable warnings by default, promote "test with warnings enabled" in
things like the packaging guide, ensure that all new deprecations are
documented in the "Porting to Python 3.x" notes, etc.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Wolfgang]

Hi,

I have to admit sometimes I don't understand why small things produce so much
mail traffic. :-)

If I use a mapping like dict most of the time I don't care if it is ordered.
If I need an ordering I use OrderedDict. In a library if I need a dict to be
ordered most of the time there is a parameter allowing me to do this or to
specify the used class as dict.
If this is not the case it can be fixed. In rare cases a workaround is needed.

As of now I have dict and OrderedDict, it is clear and explicit.
No need to change.

Yes it is useful for debugging to have things ordered.
Yes in other places the implicit ordering is also fine.
For pypy and CPython good and take it, be happy.

Also it is fine to teach people that dict (Mapping) is not ordered
but CPython has an implementation detail and it is ordered.
But if you want the guarantee use OrderedDict.

To be really practical even if this is guaranteed in 3.9
I cannot rely on it because of Python 2.7, 3.5, 3.6, ... compatibility.
If this versions are out of order in 10 years, even then if I want to
produce a small library running on another implementation I have to
care, because of the list of differences to the CPython implementation
or because the project is not yet up to date with the implementation (Jython).
To be save then I will still use OrderedDict guaranteeing me this
what I want.

Finally even when dict literals will be guaranteed to be ordered
it is good to teach and use OrderedDict because it is explicit.

For implementations (algorithm) I cannot foresee the
future so I cannot tell if it will be a burden or not.

Finally someone have to decide it.
As long as OrderedDict is available for me to specify it explicit
it will be fine. ;-)

Regards,

Wolfgang

On 04.11.2017 18:30, Stefan Krah wrote:
> 
> Hello,
> 
> would it be possible to guarantee that dict literals are ordered in v3.7?
> 
> 
> The issue is well-known and the workarounds are tedious, example:
> 
>https://mail.python.org/pipermail/python-ideas/2015-December/037423.html
> 
> 
> If the feature is guaranteed now, people can rely on it around v3.9.
> 
> 
> 
> Stefan Krah
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/tds333%40mailbox.org
> 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com