Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Serhiy Storchaka]

06.11.17 09:09, Guido van Rossum пише:
I still find this unfriendly to users of Python scripts and small apps 
who are not the developers of those scripts. (Large apps tend to spit 
out so much logging it doesn't really matter.)


Isn't there a better heuristic we can come up with so that the warnings 
tend to be on for developers but off for end users?


There was a proposition to make deprecation warnings visible by default 
in debug build and interactive interpreter.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Serhiy Storchaka]

06.11.17 05:01, INADA Naoki пише:

FYI, Raymond's original compact dict (moving last item to slot used
for deleted item) will break OrderedDict.  So it's not easy to implement
than it looks.

OrderedDict uses linked list to keep which slot is used for the key.
Moving last item will break it.
It means odict.__delitem__ can't use PyDict_DelItem anymore and
OrderedDict should touch internal structure of dict.


All this is implementation details. We did have little time for 
coordinated changing dict and OrderedDict implementations before 
releasing 3.6, and left existing coupling. This also prevents from 
in-place compactization of dict items.


We could add a private flag to dict that denotes whether this dict is 
ordered or no. The ordinal dict could be more compact, while OrderedDict 
left ordered. And I like your issue31265.


The current dict implementation still is young. It takes several 
optimizations in 3.7 (thank to you Inada) and AFAIK there are still not 
merged patches. I would wait until it become more stable before making 
change in language specification.



I think current OrderedDict implementation is fragile loose coupling.
While two object has different file (dictobject.c and odictobject.c),
OrderedDict depends on dict's internal behavior heavily.
It prevents optimizing dict. See comment here.

https://github.com/python/cpython/blob/a5293b4ff2c1b5446947b4986f98ecf5d52432d4/Objects/dictobject.c#L1082

I don't have strong opinion about what should we do about dict and OrderedDict.
But I feel PyPy's approach (using same implementation and just override
__eq__ and add move_to_end() method) is most simple.


I think that the coupling with dict and OrderedDict should be more 
robust, but left private. Dict implementation should be aware of 
OrderedDict and provide some private methods for using in OrderedDict, 
but not restrict the optimization of unordered dicts.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 6 November 2017 at 16:36, Lukasz Langa  wrote:
>
> On 5 Nov, 2017, at 9:55 PM, Nick Coghlan  wrote:
>
> Python's name resolution rules are already ridiculously complicated,
> and PEP 563 is proposing to make them *even worse*, purely for the
> sake of an optional feature primarily of interest to large enterprise
> users.
>
>
> Solving forward references in type annotations is one of the two explicit
> goals of the PEP. That alone changes how name resolution works. It sounds
> like you're -1 on that idea alone?

That's just lazy evaluation, and no different from the way name
resolution works for globals and builtins in functions, and for
globals, builtins, and closure references in classes.

So while I do expect that's going to be confusing, it's also entirely
independent of how the lazy evaluation is implemented, and I think the
basic goal of deferring annotation evaluation is a good one.

[snip]

> Since your example that illustrates the problem shows that those things fail
> for regular attribute lookup, too, that can be simply fixed in the PEP. I
> did that here:
> https://github.com/ambv/static-annotations/blob/master/pep-0563.rst#backwards-compatibility

Unfortunately, it isn't quite that trivial to fix. To see the
remaining problem, take your nested class example from the PEP and
move it inside a function.

Today, that makes no difference, since "C" will transparently switch
from being accessed via LOAD_NAME to instead being accessed with
LOAD_CLASSDEREF.

By contrast, without the ability to access the outer "C" class
definition via a closure reference, you'll no longer have a generally
applicable way to evaluate *any* of the annotations that reference it,
since you won't have access to C from the module namespace any more.

I've been persuaded that a nested *function* isn't the right answer
for type annotations (since it doesn't let you tinker with locals()
prior to execution, which in turn means you can't readily allow access
to any class attributes at execution time), but that still leaves the
more exec-friendly logic of class body compilation available.

The difference between this and class body creation is that instead of
passing the compiled code object to MAKE_FUNCTION, and then passing
the resulting function to __build_class__, we'd instead introduce a
new MAKE_THUNK opcode that implemented __call__ differently from the
way regular functions implement it.

That way, the compiler changes would be limited to:

- compile annotations like a small nested class body (but returning
the expression result, rather than None)
- emit MAKE_THUNK instead of the expression's opcodes
- emit STORE_ANNOTATION as usual

>From a name resolution perspective, the new things folks would need to
learn are:

- annotations are now lazily evaluated (just like functions)
- but name resolution works the same way it does in class bodies
(unlike lambda expressions)

To allow typing.get_type_hints() to provide access to attributes
defined in the class, you'd need one final piece of the puzzle:

- rather than accepting regular function arguments (since thunks won't
have parameter lists) thunk.__call__ would instead accept an optional
pre-populated locals() namespace to use

With those changes, blindly calling annotations would usually just
work - the one case that couldn't be handled that way would be
annotations that implicitly accessed class level attrbutes, which
would require passing in "vars(class)" when calling the thunk.

Cheers,
Nick.

P.S. Back when I made the implicit scope change for list
comprehensions, I tried all sorts of potential tweaks to the
compiler's name resolution logic before finally giving up and deciding
that using a real nested function was the only way to make sure I
avoided introducing any weird new edge cases. Lexically nested
closures are generally great, but they make it *really* hard to
emulate Python's name resolution logic without direct assistance from
the compiler at the point where the name reference appears.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Guido van Rossum]

I still find this unfriendly to users of Python scripts and small apps who
are not the developers of those scripts. (Large apps tend to spit out so
much logging it doesn't really matter.)

Isn't there a better heuristic we can come up with so that the warnings
tend to be on for developers but off for end users?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Serhiy Storchaka]

06.11.17 04:05, Nick Coghlan пише:

On the 12-weeks-to-3.7-feature-freeze thread, Jose Bueno & I both
mistakenly though the async/await deprecation warnings were missing
from 3.6.

They weren't missing, we'd just both forgotten those warnings were off
by default (7 years after the change to the default settings in 2.7 &
3.2).


Following issues on GitHub related to new Python releases I have found 
that many projects try to fix deprecation warning, but there are 
projects that are surprised by ending of deprecation periods and 
removing features.



So my proposal is simple (and not really new): let's revert back to
the way things were in 2.6 and earlier, with DeprecationWarning being
visible by default, and app devs having to silence it explicitly
during application startup (before they start importing third party
modules) if they don't want their users seeing it when running on the
latest Python version (e.g. this would be suitable for open source
apps that get integrated into Linux distros and use the system Python
there).

This will also restore the previously clear semantic and behavioural
different between PendingDeprecationWarning (hidden by default) and
DeprecationWarning (visible by default).


There was a proposition to make DeprecationWarning visible by default in 
debug builds and in interactive interpreter.


What if first implement this idea in 3.7 and make DeprecationWarning 
visible by default in production scripts only in 3.8? This will make 
less breakage.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nick Coghlan]

On 6 November 2017 at 16:26, Stephen J. Turnbull
 wrote:
> Nick Coghlan writes:
>
>  > Hence the proposed documentation change: the responsibility for
>  > silencing these warnings (for both their own code and for their
>  > dependencies) should rest with *application* developers,
>
> How do you propose to handle users with legacy apps that they can't or
> their organization won't or they don't wanna upgrade?  As I understand
> it, their only option would be something global, which they may not
> want to do.

Put "PYTHONWARNINGS=ignore::DeprecationWarning" before whatever
command is giving them the warnings.

Even on Windows, you can put that in a batch file with the actual
command you want to run and silence the warnings that way.

This is the same philosophy we applied in PEP 493 to provide a
smoother transition to HTTPS verification (via selective application
of PYTHONHTTPSVERIFY=0)

>  > We've been running the current experiment for 7 years, and the main
>  > observable outcome
>
> Well, yeah.  You can't observe something that doesn't happen, period.
>
> Bottom line: this is NOT a simple proposal, because it inherently
> deals in counterfactual reasoning.

No, it doesn't, as we've tried both approaches now: warning by default
for the ~20 years leading up to Python 2.7, and not warning by default
for the ~7 years since.

Prior to 2.7, the complaints we received mainly related to app
developers wanting to pass responsibility for *their* UX problems to
us, and ditto for poorly maintained institutional infrastructure.

So we've tried both ways now, and the status quo has led to *us*
failing to provide adequate advance notice of breaking changes to
*our* direct users. That's a far more important consideration for
CPython's default behaviour than the secondary impact on users of
applications that happen to be written in Python that are paying
sufficient attention to stderr to be scared by DeprecationWarnings,
but not paying sufficient attention to learn what those
DeprecationWarnings actually mean.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On 5 Nov, 2017, at 9:55 PM, Nick Coghlan  wrote:
> 
> Python's name resolution rules are already ridiculously complicated,
> and PEP 563 is proposing to make them *even worse*, purely for the
> sake of an optional feature primarily of interest to large enterprise
> users.

Solving forward references in type annotations is one of the two explicit goals 
of the PEP. That alone changes how name resolution works. It sounds like you're 
-1 on that idea alone?


> That's enough to leave nested classes as the main problematic case,
> since they can't see each other's attributes by default, and the
> proposed injected locals semantics in PEP 563 don't get this right
> either (they only account for MRO-based name resolution, not lexical
> nesting, even though the PEP claims the latter is expected to work)


You're right! I went too far here. I meant to be as compatible as possible with 
how regular attribute access works for nested classes. Originally I didn't want 
to support any locals at all for simplicity. I was convinced this is important 
(documented in "Rejected ideas") but overdid the example.

Since your example that illustrates the problem shows that those things fail 
for regular attribute lookup, too, that can be simply fixed in the PEP. I did 
that here:
https://github.com/ambv/static-annotations/blob/master/pep-0563.rst#backwards-compatibility
 


Note that I am still including the "def method(self) -> D.field2:" example as 
valid as including a class' own name in the locals() chain map provided by 
get_type_hints() is going to be trivial. In fact, I think I'll add this to 
get_type_hints() independently of this PEP since users of string literal 
forward references probably don't expect it to fail for nested classes.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Stephen J. Turnbull]

Nick Coghlan writes:

 > Hence the proposed documentation change: the responsibility for
 > silencing these warnings (for both their own code and for their
 > dependencies) should rest with *application* developers,

How do you propose to handle users with legacy apps that they can't or
their organization won't or they don't wanna upgrade?  As I understand
it, their only option would be something global, which they may not
want to do.

 > We've been running the current experiment for 7 years, and the main
 > observable outcome

Well, yeah.  You can't observe something that doesn't happen, period.

Bottom line: this is NOT a simple proposal, because it inherently
deals in counterfactual reasoning.

Steve


-- 
Associate Professor  Division of Policy and Planning Science
http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information
Email: turnb...@sk.tsukuba.ac.jp   University of Tsukuba
Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Reminder: 12 weeks to 3.7 feature code cutoff

[Nick Coghlan]

On 6 November 2017 at 16:00, Stephen J. Turnbull
 wrote:
> -committers and some individuals dropped from address list.
>
> Nick Coghlan writes:
>
>  > Gah, seven years on from Python 2.7's release, I still get caught by
>  > that. I'm tempted to propose we reverse that decision and go back to
>  > enabling them by default :P
>  >
>  > If app devs don't want their users seeing deprecation warnings, they
>  > can silence them globally during app startup, and end users can do the
>  > same in PYTHONSTARTUP for their interactive sessions.
>
> This point was debated then, and there were good reasons why a lot of
> users can't/won't do this.  The two I remember are (1) a lot of
> non-technical users use apps that aren't getting upgraded, and so will
> always emit those warnings, which often scare or confuse them, and

If folks get scared away from running unmaintained software, that's a
good thing, not a bad thing.

> (2)
> doing it in PYTHONSTARTUP is indeed global, and the kind of people who
> use interactive sessions typically *want* to see those warnings, but
> only some of them and only sometimes.

Then that's a good motivation to learn how to manage which deprecation
warnings they actually see.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Reminder: 12 weeks to 3.7 feature code cutoff

[Stephen J. Turnbull]

-committers and some individuals dropped from address list.

Nick Coghlan writes:

 > Gah, seven years on from Python 2.7's release, I still get caught by
 > that. I'm tempted to propose we reverse that decision and go back to
 > enabling them by default :P
 > 
 > If app devs don't want their users seeing deprecation warnings, they
 > can silence them globally during app startup, and end users can do the
 > same in PYTHONSTARTUP for their interactive sessions.

This point was debated then, and there were good reasons why a lot of
users can't/won't do this.  The two I remember are (1) a lot of
non-technical users use apps that aren't getting upgraded, and so will
always emit those warnings, which often scare or confuse them, and (2)
doing it in PYTHONSTARTUP is indeed global, and the kind of people who
use interactive sessions typically *want* to see those warnings, but
only some of them and only sometimes.

FWIW YMMV

Steve



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 6 November 2017 at 14:40, Lukasz Langa  wrote:
> On 4 Nov, 2017, at 6:32 PM, Nick Coghlan  wrote:
>> The only workaround I can see for that breakage is that instead of
>> using strings, we could instead define a new "thunk" type that
>> consists of two things:
>
>> 1. A code object to be run with eval()
>> 2. A dictionary mapping from variable names to closure cells (or None
>> for not yet resolved references to globals and builtins)
>
> This is intriguing.
>
> 1. Would that only be used for type annotations? Any other interesting
> things we could do with them?

Yes, they'd have the potential to replace strings for at least some
data analysis use cases, where passing in lambdas is too awkward
syntactically, since you have to spell out all the parameters.

The pandas.DataFrame.query operation is a reasonable example of that
kind of thing: 
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.query.html
(Not an exact example, since Pandas uses a python-like expression
language, rather than specifically Python)

Right now, folks tend to use strings for this kind of use case, which
has the same performance problem that pre-f-string string formatting
does: it defers the expression parsing and compilation step until
runtime, rather than being able to do it once and then cache the
result in __pycache__.

> 2. It feels to me like that would make annotations *heavier* at runtime
> instead of leaner, since now we're forcing the relevant closures to stay in
> memory.

Cells are pretty cheap (they're just a couple of pointers), and if
they're references to module or class attributes, the object
referenced by the cell would have remained alive regardless.

Even for nonlocal variable references (which a solely string-based
approach would disallow), the referenced objects will already be
getting kept alive anyway by way of the typing machinery.

> 3. This form of lazy evaluation seems pretty implicit to me for the reader.
> Peter Ludemann's example of magic logging.debug() is a case in point here.

One of the biggest advantages though is that just like functions, all
of the necessary logic for doing the delayed evaluation can be
captured in a __call__ method, rather than via elaborate instructions
on how to appropriately invoke eval() based on knowledge of where the
annotation came from.

This is especially important if typing gets taken out of the standard
library: you'll need a replacement for typing.get_type_hints() in PEP
563, and a thunk.__call__() method would be a good spelling for that.

> All in all, unless somebody else is ready to step up and write the PEP on
> this subject (and its implementation) right now, I think this idea will miss
> Python 3.7.

As long as we don't argue for that being an adequate excuse to rush
into "We're using plain strings with ill-defined name resolution
semantics because we couldn't be bothered coming up with a proper
thunk-based design to evaluate", I'd be fine with that. None of this
is urgent, and it's mainly of interest to large organisations that
will see a direct economic benefit from implementing it, so the entire
idea can easily be delayed to 3.8 if they're not prepared to fund a
proper evaluation of the available design options over the next 3
months.

Python's name resolution rules are already ridiculously complicated,
and PEP 563 is proposing to make them *even worse*, purely for the
sake of an optional feature primarily of interest to large enterprise
users. If delayed evaluation of type annotations is deemed important
enough to burden every future Pythonista with learning a second set of
name resolution semantics purely for type annotations, then it's
important enough to postpone implementing it until someone invests the
time in coming up with a competing thunk-based alternative that is
able to rely entirely on the *existing* name resolution semantics.

Exploring that potential thunk-based approach a bit further:

1. We'd continue to eagerly compile annotations (as we do today), but
treat them like a nested class body with a single expression. Unlike
an implicit lambda, this compilation mode will allow the resulting
code object to be used with the two-argument form of the exec builtin
2. That code object would be the main item stored on the thunk object
3. If __classcell__ is defined in the current namespace and names from
the current namespace are referenced, then that can be captured on the
thunk, giving its __call__ method access to any class attributes
needed for name resolution
4. Closure references would be captured automatically, but class
bodies already allow locals to override nonlocals (for compatibility
with pre-populated namespaces returned from __prepare__)
5. A thunk's __globals__ reference would be implicitly captured the
same way it is for a regular function

That's enough to leave nested classes as the main problematic case,
since they can't see each other's attributes by default, and the

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nick Coghlan]

On 6 November 2017 at 14:14, Barry Warsaw  wrote:
> On Nov 5, 2017, at 18:05, Nick Coghlan  wrote:
>
>> So my proposal is simple (and not really new): let's revert back to
>> the way things were in 2.6 and earlier, with DeprecationWarning being
>> visible by default
>
> +1
>
>> As part of this though, I'd suggest amending the documentation for
>> DeprecationWarning [1] to specifically cover how to turn it off
>> programmatically (`warnings.simplefilter("ignore",
>> DeprecationWarning)`), at the command line (`python -W
>> ignore::DeprecationWarning ...`), and via the environment
>> (`PYTHONWARNINGS=ignore::DeprecationWarning`).
>
> +1
>
> I’d also consider adding convenient shortcuts for each of these.  I think 
> DeprecationWarning is special enough to warrant it.  Possibly:
>
> warnings.silence_deprecations()
> python -X silence-deprecations
> PYTHONSILENCEDEPRECATIONS=x

It could be interesting to combine this with Tim's suggestion of
putting an upper version limit on the silencing, so the above may look
like:

warnings.ignore_deprecations((3, 7))
python -X ignore-deprecations=3.7
PYTHONIGNOREDEPRECATIONS=3.7

(Using "ignore" to match the existing action name so the intent is a
bit more self-explanatory)

The ignore_deprecations function would then look like:

def ignore_deprecations(max_version):
"""Ignore DeprecationWarning on Python versions up to &
including the given one

*max_version* is an iterable suitable for ordered comparison
against sys.version_info
"""
if sys.version_info <= max_version:
warnings.simplefilter('ignore', DeprecationWarning)

So the conventional usage would be that if you were regularly updating
your application, by the time Python 3.8 actually existed, the check
would have been bumped to say 3.8. But if you stopped updating (or the
publisher stopped releasing updates), you'd eventually start getting
deprecation warnings again as the underlying Python version changed.

I could definitely see that working well for the community Linux
distro use case, where there isn't necessarily anyone closely
monitoring old packages to ensure they're actively tracking upstream
releases (and attempting to institute more ruthless pruning processes
can lead to potentially undesirable community dynamics)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On 4 Nov, 2017, at 6:32 PM, Nick Coghlan  wrote:
> 
> The PEP's current attitude towards this is "Yes, it will break, but
> that's OK, because it doesn't matter for the type annotation use case,
> since static analysers will still understand it". Adopting such a
> cavalier approach towards backwards compatibility with behaviour that
> has been supported since Python 3.0 *isn't OK*, since it would mean we
> were taking the step from "type annotations are the primary use case"
> to "Other use cases for function annotations are no longer supported".

Well, this is what the PEP literally says in "Deprecation policy":

> In Python 4.0 this will become the default behavior. Use of annotations 
> incompatible with this PEP is no longer supported.

The rationale here is that type annotations as defined by PEP 484 and others is 
the only notable use case. Note that "type annotations" includes things like 
data classes, auto_attribs in attrs, the dependency injection frameworks 
mentioned before, etc. Those are compatible with PEP 484. So, despite the open 
nature of annotations since Python 3.0, no alternative use case emerged that 
requires eager evaluation and access to local state. PEP 563 is addressing the 
pragmatic issue of improving usability of type annotations, instead of worrying 
about some unknown theoretically possible use case.

While function annotations were open to arbitrary use, typing was increasingly 
hinted (pun not intended) as *the* use case for them:

1. From Day 1, type checking is listed as the number one intended use case in 
PEP 3107 (and most others listed there are essentially type annotations by any 
other name).
2. PEP 484 says "We do hope that type hints will eventually become the sole use 
for annotations", and that "In order for maximal compatibility with offline 
type checking it may eventually be a good idea to change interfaces that rely 
on annotations to switch to a different mechanism, for example a decorator."
3. Variable annotations in PEP 526 were designed with type annotations as the 
sole stated purpose.

PEP 563 simply brings this multi-PEP dance to its logical conclusion, stating 
in "Rationale and Goals" that "uses for annotations incompatible with the 
aforementioned PEPs should be considered deprecated." The timeline for full 
deprecation is Python 4.0.


> The only workaround I can see for that breakage is that instead of
> using strings, we could instead define a new "thunk" type that
> consists of two things:
> 
> 1. A code object to be run with eval()
> 2. A dictionary mapping from variable names to closure cells (or None
> for not yet resolved references to globals and builtins)

This is intriguing.

1. Would that only be used for type annotations? Any other interesting things 
we could do with them?
2. It feels to me like that would make annotations *heavier* at runtime instead 
of leaner, since now we're forcing the relevant closures to stay in memory.
3. This form of lazy evaluation seems pretty implicit to me for the reader. 
Peter Ludemann's example of magic logging.debug() is a case in point here.

All in all, unless somebody else is ready to step up and write the PEP on this 
subject (and its implementation) right now, I think this idea will miss Python 
3.7.


> Now, even without the introduction of the IndirectAttributeCell
> concept, this is amenable to a pretty simple workaround:
> 
> A = Optional[int]
> class C:
>field: A = 1
>def method(self, arg: A) -> None: ...
>C.A = A
>del A

This is a poor workaround, worse in fact than using a string literal as a 
forward reference. This is more verbose and error-prone. Decorators address the 
same construct and their wild popularity suggests that this notation is 
inferior.


> But I genuinely can't see how breaking annotation evaluation at class
> scope can be seen as a deal-breaker for the implicit lambda based
> approach without breaking annotation evaluation for nested functions
> also being seen as a deal-breaker for the string based approach.

The main reason to use type annotations is readability, just like decorators. 
While there's nothing stopping the programmer to write:

class C: ...
def method(self, arg1): ...
method.__annotations__ = {'arg1': str, 'return': int}
C.__annotations__ = {'attribute1': ...}

...this notation doesn't fit the bill. Since nested classes and types embedded 
as class attributes are popular among type hinting users, supporting this case 
is a no-brainer. On the other hand, if you have a factory function that 
generates some class or function, then you either:

1. Use annotations in the generated class/function for type checking; OR
2. Add annotations in the generated class/function for them to be preserved in 
__annotations__ for some future runtime use.

In the former case, you are unlikely to use local state. But even if you were, 
that doesn't matter since the static type checker 

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On 4 Nov, 2017, at 11:43 AM, Peter Ludemann via Python-Dev 
>  wrote:
> 
> If type annotations are treated like implicit lambdas, then that's a first 
> step to something similar to Lisp's "special forms". A full generalization of 
> that would allow, for example, logging.debug to not evaluate its args unless 
> debugging is turned on (I use a logging.debug wrapper that allows lambdas as 
> args, and evaluates them if debugging is turned on).

Interestingly enough, at Facebook we found out that using f-strings is *faster* 
at runtime than the lazy form of logging.log("format with %s and %d", arg1, 
arg2), including for cases when the log message is not emitted.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 5, 2017, at 18:05, Nick Coghlan  wrote:

> So my proposal is simple (and not really new): let's revert back to
> the way things were in 2.6 and earlier, with DeprecationWarning being
> visible by default

+1

> As part of this though, I'd suggest amending the documentation for
> DeprecationWarning [1] to specifically cover how to turn it off
> programmatically (`warnings.simplefilter("ignore",
> DeprecationWarning)`), at the command line (`python -W
> ignore::DeprecationWarning ...`), and via the environment
> (`PYTHONWARNINGS=ignore::DeprecationWarning`).

+1

I’d also consider adding convenient shortcuts for each of these.  I think 
DeprecationWarning is special enough to warrant it.  Possibly:

warnings.silence_deprecations()
python -X silence-deprecations
PYTHONSILENCEDEPRECATIONS=x

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Tim Delaney]

On 6 November 2017 at 13:05, Nick Coghlan  wrote:

> As part of this though, I'd suggest amending the documentation for
> DeprecationWarning [1] to specifically cover how to turn it off
> programmatically (`warnings.simplefilter("ignore",
> DeprecationWarning)`), at the command line (`python -W
> ignore::DeprecationWarning ...`), and via the environment
> (`PYTHONWARNINGS=ignore::DeprecationWarning`).
>

I'm wondering if it would be sensible to recommend only disabling the
warnings if running with a known version of Python e.g.

if sys.version_info < (3, 8):
with warnings.simplefilter('ignore', DeprecationWarning):
import module

The idea here is to prompt the developer to refactor to not use the
deprecated functionality early enough that users aren't impacted.

Tim Delaney
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Lukasz Langa]

> On 4 Nov, 2017, at 3:39 AM, Paul Moore  wrote:
> 
> Lukasz Langa said:
>> So, the difference is in perceived usability. It's psychological.
> 
> Please, let's not start the "not in the stdlib isn't an issue" debate
> again. If I concede it's a psychological issue, will you concede that
> the fact that it's psychological doesn't mean that it's not a real,
> difficult to solve, problem for some people? I'm also willing to
> concede that it's a *minority* problem, if that helps. But can we stop
> dismissing it as a non-existent problem?

Paul, if you read the words I wrote in my e-mail verbatim, you will note that I 
am not saying it's not real or it's not important. Quite the opposite. Can you 
elaborate what made you think that my assertion that the issue is psychological 
made you think I'm being dismissive? To me it looks like you're aggressively 
agreeing with me, so I'd like to understand what caused your reaction.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nick Coghlan]

On 6 November 2017 at 12:29, Oleg Broytman  wrote:
> On Sun, Nov 05, 2017 at 09:20:12PM -0500, Yury Selivanov 
>  wrote:
>> Big +1 from me.  The whole point of DeprecationWarnings is to be
>> visible
>
>The whole point of DeprecationWarnings is to be visible to
> developers while in reality they will be visible to users -- and what
> the users would do with the warnings?

Hence the proposed documentation change: the responsibility for
silencing these warnings (for both their own code and for their
dependencies) should rest with *application* developers, with our
responsibility as providers of the underlying platform then being to
make it completely obvious how to actually do that (it's currently
really unclear, with the relevant info being scattered across the list
of builtin warnings, different parts of the warnings module and
CPython command line usage documentation, with no explicit examples of
exactly what you need to write anywhere).

To put that another way:

- if we ever write "import foo" ourselves, then we're a Python
developer, and it's our responsibility to work out how to manage
DeprecationWarning when it gets raised by either our own code, or the
libraries and frameworks that we use
- if we ship Python code in a "supply your own runtime" model, such
that we have actual non-developer users and operators (rather than
solely fellow Python developers) to worry about, then it's still our
responsibility to decide whether or not we want to let deprecation
warnings appear on stderr (based on what we think is most appropriate
for our particular user base)
- if we want to categorically ensure our users don't get unexpected
deprecation warnings on stderr, then we should be bundling a Python
runtime as well (e.g. via PyInstaller or a Linux container image, or
by operating a network service), rather than asking users and
operators to handle the runtime+application integration step

We've been running the current experiment for 7 years, and the main
observable outcome has been folks getting surprised by breaking
changes in CPython releases, especially folks that primarily use
Python interactively (e.g. for data analysis), or as a scripting
engine (e.g. for systems administration).

That means the status quo is defeating the main purpose of
DeprecationWarnings (providing hard-to-miss advance notice of upcoming
breaking changes in the language definition and standard library), for
the sake of letting app developers duck responsibility for managing
what their own software writes to stderr.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Lukasz Langa]

> On 5 Nov, 2017, at 6:29 PM, Oleg Broytman  wrote:
> 
> On Sun, Nov 05, 2017 at 09:20:12PM -0500, Yury Selivanov 
>  wrote:
>> Big +1 from me.  The whole point of DeprecationWarnings is to be
>> visible
> 
> The whole point of DeprecationWarnings is to be visible to
> developers while in reality they will be visible to users -- and what
> the users would do with the warnings?

Complain to the authors and make them remove the issue.

https://github.com/requests/requests/issues/3954 

https://github.com/scikit-learn-contrib/sklearn-pandas/issues/76 

https://github.com/pandas-dev/pandas/issues/5824 

https://github.com/pypa/setuptools/issues/472 



+1 to re-enable this from me, too. At Facebook we are running unit tests and 
development builds with warnings. Just be aware that for some applications that 
will spew a lot. Python 3.6's warnings on invalid escapes is a big source of 
this.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nick Coghlan]

On 6 November 2017 at 09:43, Raymond Hettinger
 wrote:
> On Nov 4, 2017, at 7:04 PM, Nick Coghlan  wrote:
>> I don't think that situation should change the decision, but I do
>> think it would be helpful if folks that understand CPython's dict
>> implementation could take a look at MicroPython's dict implementation
>> and see if it might be possible for them to avoid having to make that
>> trade-off and instead be able to use a naturally insertion ordered
>> hashmap implementation.
>
> I've just looked at the MicroPython dictionary implementation and think they 
> won't have a problem implementing O(1) compact dicts with ordering.
>
> The likely reason for the confusion is that they are already have an option 
> for an "ordered array" dict variant that does a brute-force linear search.  
> However, their normal hashed lookup is very similar to ours and is easily 
> amenable to being compact and ordered.
>
> See:  
> https://github.com/micropython/micropython/blob/77a48e8cd493c0b0e0ca2d2ad58a110a23c6a232/py/map.c#L139
>
> Pretty much any implementation hashed lookup of keys and values is amenable 
> to being compact and ordered.  Whatever existing logic that looks up an entry 
> becomes a lookup into a table of indices which in turn references a 
> sequential array of keys and values.  This logic is independent of hashing 
> scheme or density, and it has no effect on the number of probes or collision 
> rate.
>
> The cost is an extra level of indirection and an extra array of indices 
> (typically very small). The benefit is faster iteration over the smaller 
> dense key/value array, overall memory savings resulting in improved cache 
> utilization, and the side-effect of remembering insertion order.
>
> Summary:  I think MicroPython will be just fine and if needed I will help 
> create the patch that implements compact-and-ordered behavior.

Nice! That's what I thought based on reading some of the design
discussions about CPython's dict implementation, but I didn't know the
details of either dict implementation well enough to be confident that
the CPython changes would map cleanly to MicroPython's variant.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nick Coghlan]

On 6 November 2017 at 06:50, Peter Ludemann via Python-Dev
 wrote:
> Isn't ordered dict also useful for **kwargs?

3.6 already made this semantic change for **kwargs and class execution
namespaces - it just left the door open to having implementations meet
those requirements by way of substituting in collections.OrderedDict
for those use cases, rather than using a regular builtin dictionary.

So we're also already imposing the requirement on conformant Python
implementations to have a reasonably performant insertion-ordered
mapping implementation available.

The open questions around insertion ordering thus relate to what user
expectations should be for:

- dictionary displays
- explicit invocations of the builtin dict constructor
- mutating methods on builtin dicts
- serialisation and deserialisation operations that use regular dicts
(e.g. the JSON module, csv.DictReader/Writer)

It's that last category which is particularly problematic now, as in
*C*Python 3.6, the dicts used for these operations actually *are*
insertion ordered, so round trips from disk, through a CPython builtin
dict, and then back to disk *will* be order preserving, whereas in
previous versions there was no guarantee that that would be the case
(and hash randomisation meant the key reordering wasn't even
deterministic).

While such operations were already order preserving in PyPy (since
their switch to an insertion-ordered builtin dict implementation
served as prior art for the subsequent switch in CPython), we know
from experience that it's incredibly easy for even things that are
nominally implementation details of CPython to become expected
behaviour for Python users (e.g. there's still plenty of code out
there that relies on the use of an eager refcounting memory management
strategy to handle external resources properly).

That means our choices for 3.7 boil down to:

* make this a language level guarantee that Python devs can reasonably rely on
* deliberately perturb dict iteration in CPython the same way the
default Go runtime does [1]

When we did the "insertion ordered hash map" availability review, the
main implementations we were checking on behalf of were Jython & VOC
(JVM implementations), Batavia (JavaScript implementation), and
MicroPython (C implementation). Adding IronPython (C# implementation)
to the mix gives:

* for the JVM, the insertion ordered LinkedHashMap [2] has been
available since J2SE 1.4 was released in 2002
* for JavaScript, ECMA 6 defines the Map type [3] as an insertion
ordered key/value store
* for the .NET CLR, System.Collections.Specialized.OrderedDictionary
[4] seems to be the best builtin option, but the Java implementations
also map reasonably well to C# semantics
* for MicroPython, it's not yet clear whether or not the hash map
design used in CPython could also be adapted to their design
constraints (i.e. giving both insertion ordering and O(1) lookup
without requiring excessive amounts of memory), but it should at least
be feasible to make the semantics configurable based on a compile time
option (since CPython has a working C implementation of the desired
semantics already)

Since the round-trip behaviour that comes from guaranteed order
preservation is genuinely useful, and we're comfortable with folks
switching to more specialised containers when they need different
performance characteristics from what the builtins provide, elevating
insertion order preservation to a language level requirements makes
sense.

Cheers,
Nick.

[1] https://blog.golang.org/go-maps-in-action (search for "Iteration Order")
[2] https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashMap.html
[3] 
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map
[4] 
https://docs.microsoft.com/en-us/dotnet/api/system.collections.specialized.ordereddictionary?view=netframework-4.7.1

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[INADA Naoki]

On Mon, Nov 6, 2017 at 4:54 AM, Serhiy Storchaka  wrote:
...
>
> I didn't try to implement this. But the current implementation requires
> periodical reallocating if add and remove items. The following loop
> reallocates the dict every len(d) iterations, while the size of the dict is
> not changed, and the half of its storage is empty.
>
> while True:
> v = d.pop(k)
> ...
> d[k] = v
>

FYI, Raymond's original compact dict (moving last item to slot used
for deleted item) will break OrderedDict.  So it's not easy to implement
than it looks.

OrderedDict uses linked list to keep which slot is used for the key.
Moving last item will break it.
It means odict.__delitem__ can't use PyDict_DelItem anymore and
OrderedDict should touch internal structure of dict.

I think current OrderedDict implementation is fragile loose coupling.
While two object has different file (dictobject.c and odictobject.c),
OrderedDict depends on dict's internal behavior heavily.
It prevents optimizing dict. See comment here.

https://github.com/python/cpython/blob/a5293b4ff2c1b5446947b4986f98ecf5d52432d4/Objects/dictobject.c#L1082

I don't have strong opinion about what should we do about dict and OrderedDict.
But I feel PyPy's approach (using same implementation and just override
__eq__ and add move_to_end() method) is most simple.

Regards,

INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Raymond Hettinger]

> On Nov 5, 2017, at 4:31 PM, Nathaniel Smith  wrote:
> 
> CPython does in practice provide ordering guarantees for dicts, and this 
> solves a whole bunch of pain points: it makes json roundtripping work better, 
> it gives ordered kwargs, it makes it possible for metaclasses to see the 
> order class items were defined, etc. And we got all these goodies for 
> better-than-free: the new dict is faster and uses less memory. So it seems 
> very unlikely that CPython is going to revert this change in the foreseeable 
> future, and that means people will write code that depends on this, and that 
> means in practice reverting it will become impossible due to backcompat and 
> it will be important for other interpreters to implement, regardless of what 
> the language definition says.
> 
> That said, there are real benefits to putting this in the spec. Given that 
> we're not going to get rid of it, we might as well reward the minority of 
> programmers who are conscientious about following the spec by letting them 
> use it too.

Thanks. Your note resonated with me -- the crux of your argument seems to be 
that the proposal results in a net reduction in complexity for both users and 
implementers.

That makes sense. Even having read all the PEPs, read all the patches, and 
having participated in the discussions, I tend to forget where ordering is 
guaranteed and where it isn't.

This discussion reminds me of when Timsort was introduced many years ago.  Sort 
stability wasn't guaranteed at first, but it was so darned convenient (and a 
pain to work around when not present) that it became guaranteed in the 
following release.   The current proposal is different in many ways, but does 
share the virtue of being a nice-to-have for users.

> MicroPython deviates from the language spec in lots of ways. Hopefully this 
> won't need to be another one, but it won't be the end of the world if it is.

I've looked at the MicroPython source and think this won't be a problem.  It 
will be even easier for them than it was for us (the code is simpler because it 
doesn't have special cases for key-sharing, unicode optimizations, and whatnot).


Raymond 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Oleg Broytman]

On Sun, Nov 05, 2017 at 09:20:12PM -0500, Yury Selivanov 
 wrote:
> Big +1 from me.  The whole point of DeprecationWarnings is to be
> visible

   The whole point of DeprecationWarnings is to be visible to
developers while in reality they will be visible to users -- and what
the users would do with the warnings?

> Yury

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Yury Selivanov]

Big +1 from me.  The whole point of DeprecationWarnings is to be
visible so that they are resolved faster.  The current behaviour
allows them to go unnoticed for the majority of Python users.

Yury

On Sun, Nov 5, 2017 at 9:05 PM, Nick Coghlan  wrote:
> On the 12-weeks-to-3.7-feature-freeze thread, Jose Bueno & I both
> mistakenly though the async/await deprecation warnings were missing
> from 3.6.
>
> They weren't missing, we'd just both forgotten those warnings were off
> by default (7 years after the change to the default settings in 2.7 &
> 3.2).
>
> So my proposal is simple (and not really new): let's revert back to
> the way things were in 2.6 and earlier, with DeprecationWarning being
> visible by default, and app devs having to silence it explicitly
> during application startup (before they start importing third party
> modules) if they don't want their users seeing it when running on the
> latest Python version (e.g. this would be suitable for open source
> apps that get integrated into Linux distros and use the system Python
> there).
>
> This will also restore the previously clear semantic and behavioural
> different between PendingDeprecationWarning (hidden by default) and
> DeprecationWarning (visible by default).
>
> As part of this though, I'd suggest amending the documentation for
> DeprecationWarning [1] to specifically cover how to turn it off
> programmatically (`warnings.simplefilter("ignore",
> DeprecationWarning)`), at the command line (`python -W
> ignore::DeprecationWarning ...`), and via the environment
> (`PYTHONWARNINGS=ignore::DeprecationWarning`).
>
> (Structurally, I'd probably put that at the end of the warnings
> listing as a short introduction to warnings management, with links out
> to the relevant sections of the documentation, and just use
> DeprecationWarning as the specific example)
>
> Cheers,
> Nick.
>
> [1] https://docs.python.org/3/library/exceptions.html#DeprecationWarning
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Nick Coghlan]

On 4 November 2017 at 02:46, Eric V. Smith  wrote:
> On 11/3/2017 12:15 PM, Victor Stinner wrote:
>>
>> Hi,
>>
>> 2017-11-03 15:36 GMT+01:00 Guido van Rossum :
>>>
>>> Maybe we should remove typing from the stdlib?
>>> https://github.com/python/typing/issues/495
>
>
>> The typing module is not used yet in the stdlib, so there is no
>> technically reason to keep typing part of the stdlib. IMHO it's
>> perfectly fine to keep typing and annotations out of the stdlib, since
>> the venv & pip tooling is now rock solid ;-)
>
>
> I'm planning on using it for PEP 557:
> https://www.python.org/dev/peps/pep-0557/#class-variables
>
> The way the code currently checks for this should still work if typing is
> not in the stdlib, although of course it's assuming that the name "typing"
> really is the "official" typing library.
>
> # If typing has not been imported, then it's impossible for
> #  any annotation to be a ClassVar. So, only look for ClassVar
> #  if typing has been imported.
> typing = sys.modules.get('typing')
> if typing is not None:
> # This test uses a typing internal class, but it's the best
> #  way to test if this is a ClassVar.
> if type(a_type) is typing._ClassVar:
> # This field is a ClassVar. Ignore it.
> continue
>
> See also https://github.com/ericvsmith/dataclasses/issues/14

That particular dependency could also be avoided by defining an
"is_class_var(annotation)" generic function and a "ClassVar" helper
object in the dataclasses module. For example:

class _ClassVar:
def __init__(self, annotation):
self.annotation = annotation

class _MakeClassVar:
def __getitem__(self, key):
return _ClassVar(key)

ClassVar = _MakeClassVar()

@functools.singledispatch
def is_class_var(annotation):
return isinstance(annotation, _ClassVar)

It would put the burden on static analysers and the typing module to
understand that `dataclasses.ClassVar` meant the same thing
conceptually as `typing.ClassVar`, but I think that's OK.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Nick Coghlan]

On the 12-weeks-to-3.7-feature-freeze thread, Jose Bueno & I both
mistakenly though the async/await deprecation warnings were missing
from 3.6.

They weren't missing, we'd just both forgotten those warnings were off
by default (7 years after the change to the default settings in 2.7 &
3.2).

So my proposal is simple (and not really new): let's revert back to
the way things were in 2.6 and earlier, with DeprecationWarning being
visible by default, and app devs having to silence it explicitly
during application startup (before they start importing third party
modules) if they don't want their users seeing it when running on the
latest Python version (e.g. this would be suitable for open source
apps that get integrated into Linux distros and use the system Python
there).

This will also restore the previously clear semantic and behavioural
different between PendingDeprecationWarning (hidden by default) and
DeprecationWarning (visible by default).

As part of this though, I'd suggest amending the documentation for
DeprecationWarning [1] to specifically cover how to turn it off
programmatically (`warnings.simplefilter("ignore",
DeprecationWarning)`), at the command line (`python -W
ignore::DeprecationWarning ...`), and via the environment
(`PYTHONWARNINGS=ignore::DeprecationWarning`).

(Structurally, I'd probably put that at the end of the warnings
listing as a short introduction to warnings management, with links out
to the relevant sections of the documentation, and just use
DeprecationWarning as the specific example)

Cheers,
Nick.

[1] https://docs.python.org/3/library/exceptions.html#DeprecationWarning

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Reminder: 12 weeks to 3.7 feature code cutoff

[Nick Coghlan]

On 6 November 2017 at 02:02, Yury Selivanov  wrote:
> On Fri, Nov 3, 2017 at 11:30 PM, Nick Coghlan  wrote:
>> The current lack of DeprecationWarnings in 3.6 is a fairly major
>> oversight/bug, though:
>
> There's no oversight.  We had PendingDeprecationWarning for
> async/await names in 3.5, and DeprecationWarning in 3.6.  You just
> need to enable warnings to see them:

Gah, seven years on from Python 2.7's release, I still get caught by
that. I'm tempted to propose we reverse that decision and go back to
enabling them by default :P

If app devs don't want their users seeing deprecation warnings, they
can silence them globally during app startup, and end users can do the
same in PYTHONSTARTUP for their interactive sessions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Nathaniel Smith]

On Nov 5, 2017 2:41 PM, "Paul Ganssle"  wrote:

I think the question of whether any specific implementation of dict could
be made faster for a given architecture or even that the trade-offs made by
CPython are generally the right ones is kinda beside the point. It's
certainly feasible that an implementation that does not preserve ordering
could be better for some implementation of Python, and the question is
really how much is gained by changing the language semantics in such a way
as to cut off that possibility.


The language definition is not nothing, but I think it's easy to
overestimate its importance. CPython does in practice provide ordering
guarantees for dicts, and this solves a whole bunch of pain points: it
makes json roundtripping work better, it gives ordered kwargs, it makes it
possible for metaclasses to see the order class items were defined, etc.
And we got all these goodies for better-than-free: the new dict is faster
and uses less memory. So it seems very unlikely that CPython is going to
revert this change in the foreseeable future, and that means people will
write code that depends on this, and that means in practice reverting it
will become impossible due to backcompat and it will be important for other
interpreters to implement, regardless of what the language definition says.

That said, there are real benefits to putting this in the spec. Given that
we're not going to get rid of it, we might as well reward the minority of
programmers who are conscientious about following the spec by letting them
use it too. And there were multiple PEPs that went away when this was
merged; no one wants to resurrect them just for hypothetical future
implementations that may never exist. And putting it in the spec will mean
that we can stop having this argument over and over with the same points
rehashed for those who missed the last one. (This isn't aimed at you or
anything; it's not your fault you don't know all these arguments off the
top of your head, because how would you? But it is a reality of mailing
list dynamics that rehashing this kind of thing sucks up energy without
producing much.)

MicroPython deviates from the language spec in lots of ways. Hopefully this
won't need to be another one, but it won't be the end of the world if it is.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Raymond Hettinger]

> On Nov 4, 2017, at 7:04 PM, Nick Coghlan  wrote:
> 
> When I asked Damien George about this for MicroPython, he indicated
> that they'd have to choose between guaranteed order and O(1) lookups
> given their current dict implementation. That surprised me a bit
> (since PyPy and CPython both *saved* memory by switching to their
> guaranteed order implementations, hence the name "compact dict
> representation"), but my (admittedly vague) understand is that the
> presence of a space/speed trade-off in their case has something to do
> with MicroPython deliberately running with a much higher chance of
> hash collisions in general (since the data sets it deals with are
> naturally smaller).
> 
> So if we make the change, MicroPython will likely go along with it,
> but it may mean that dict lookups there become O(N), and folks will be
> relying on "N" being consistently small due to memory constraints (but
> some typically O(N) algorithms will still become O(N^2) when run on
> MicroPython).
> 
> I don't think that situation should change the decision, but I do
> think it would be helpful if folks that understand CPython's dict
> implementation could take a look at MicroPython's dict implementation
> and see if it might be possible for them to avoid having to make that
> trade-off and instead be able to use a naturally insertion ordered
> hashmap implementation.

I've just looked at the MicroPython dictionary implementation and think they 
won't have a problem implementing O(1) compact dicts with ordering.

The likely reason for the confusion is that they are already have an option for 
an "ordered array" dict variant that does a brute-force linear search.  
However, their normal hashed lookup is very similar to ours and is easily 
amenable to being compact and ordered.

See:  
https://github.com/micropython/micropython/blob/77a48e8cd493c0b0e0ca2d2ad58a110a23c6a232/py/map.c#L139

Pretty much any implementation hashed lookup of keys and values is amenable to 
being compact and ordered.  Whatever existing logic that looks up an entry 
becomes a lookup into a table of indices which in turn references a sequential 
array of keys and values.  This logic is independent of hashing scheme or 
density, and it has no effect on the number of probes or collision rate.

The cost is an extra level of indirection and an extra array of indices 
(typically very small). The benefit is faster iteration over the smaller dense 
key/value array, overall memory savings resulting in improved cache 
utilization, and the side-effect of remembering insertion order.

Summary:  I think MicroPython will be just fine and if needed I will help 
create the patch that implements compact-and-ordered behavior.


Raymond




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steve Dower]

Since there is voting going on, -1 on changing the guarantees of `dict`. If 
ordering is important, OrderedDict is more explicit and more obvious to the 
over reading the code, even if the underlying implementation is identical.

Good luck teaching people about why Python went from OrderedDict to 
UnorderedDict within a version. I remember learning about this same thing in 
C++ and just thinking “wut?”. And they just added a new type and said to stop 
using the old one – not changing something that people already understand.

Better to underspecify the default dict and offer explicitly named extensions 
(DefaultDict, OrderedDict, etc.) for those who want more guarantees.

Cheers,
Steve

Top-posted from my Windows phone

From: Peter Ludemann via Python-Dev
Sent: Sunday, November 5, 2017 12:53
To: Sven R. Kunze
Cc: Python-Dev
Subject: Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

Isn't ordered dict also useful for **kwargs? 

If it turns out that there's a dict implementation that's faster by not 
preserving order, collections.UnorderedDict could be added.
There could also be specialized implementations that pre-size the dict (cf: C++ 
unordered_map::reserve), etc., etc.
But these are all future things, which might not be necessary.

On 5 November 2017 at 12:44, Sven R. Kunze  wrote:
+1 from me too.

On 04.11.2017 21:55, Jim Baker wrote:
+1, as Guido correctly recalls, this language guarantee will work well with 
Jython when we get to the point of implementing 3.7+. 

On Sat, Nov 4, 2017 at 12:35 PM, Guido van Rossum  wrote:
This sounds reasonable -- I think when we introduced this in 3.6 we were 
worried that other implementations (e.g. Jython) would have a problem with 
this, but AFAIK they've reported back that they can do this just fine. So let's 
just document this as a language guarantee.

On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah  wrote:

Hello,

would it be possible to guarantee that dict literals are ordered in v3.7?


The issue is well-known and the workarounds are tedious, example:

   https://mail.python.org/pipermail/python-ideas/2015-December/037423.html


If the feature is guaranteed now, people can rely on it around v3.9.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/pludemann%40google.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Raymond Hettinger]

> On Nov 3, 2017, at 9:15 AM, Victor Stinner  wrote:
> 
> 2017-11-03 15:36 GMT+01:00 Guido van Rossum :
>> Maybe we should remove typing from the stdlib?
>> https://github.com/python/typing/issues/495
> 
> I'm strongly in favor on such move.
> 
> My experience with asyncio in the stdlib is that users expect changes
> faster than the very slow release process of the stdlib (a release
> every 18 months in average).
> 
> I saw many PEPs and discussion on the typing design (meta-classes vs
> regular classes), as if the typing is not stable enough to be part of
> the stdlib.
> 
> The typing module is not used yet in the stdlib, so there is no
> technically reason to keep typing part of the stdlib. IMHO it's
> perfectly fine to keep typing and annotations out of the stdlib, since
> the venv & pip tooling is now rock solid ;-)

I concur with Victor on every point.  In particular, many of the good reasons 
that typeshed is external to the standard library will also apply to typing.py. 
 

It would also be nice to not have typing.py vary with each version of CPython's 
release cycle.  Not only would typing benefit from more frequent updates, it 
would be nice to have updates that aren't tied to a specific version of CPython 
-- that would help folks who have to maintain code that works across multiple 
CPython versions (i.e. the same benefit that we get by always installing the 
most up-to-date versions of requests, typeshed, jinja2, etc).

Already, we've have updates to typing.py in the point releases of Python 
because those updates were considered so useful and important.


Raymond 





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Tim Delaney]

On 6 November 2017 at 06:09, Serhiy Storchaka  wrote:

> 05.11.17 20:39, Stefan Krah пише:
>
>> On Sun, Nov 05, 2017 at 01:14:54PM -0500, Paul G wrote:
>>
>>> 2. Someone invents a new arbitrary-ordered container that would improve
>>> on the memory and/or CPU performance of the current dict implementation
>>>
>>
>> I would think this is very unlikely, given that the previous dict
>> implementation
>> has always been very fast. The new one is very fast, too.
>>
>
> The modification of the current implementation that don't preserve the
> initial order after deletion would be more compact and faster.


I would personally be happy with this as the guarantee (it covers dict
literals and handles PEP 468), but it might be more confusing. "dicts are
in arbitrary order" and "dicts maintain insertion order" are fairly simple
to explain, "dicts maintain insertion order up to the point that a key is
deleted" is less so.

Tim Delaney
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Tim Delaney]

On 6 November 2017 at 07:50, Peter Ludemann via Python-Dev <
python-dev@python.org> wrote:

> Isn't ordered dict also useful for **kwargs?
>

**kwargs is already specified as insertion ordered as of Python  3.6.

https://www.python.org/dev/peps/pep-0468/

Tim Delaney
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Ganssle]

> If it turns out that there's a dict implementation that's faster by not
> preserving order, collections.UnorderedDict could be added.
> There could also be specialized implementations that pre-size the dict (cf:
> C++ unordered_map::reserve), etc., etc.
> But these are all future things, which might not be necessary.

I think that the problem with this is that for the most part, people will use 
`dict`, and for most uses of `dict`, order doesn't matter (and it never has 
before). Given that "arbitrary order" includes *any* fixed ordering (insertion 
ordered, reverse insertion ordered, etc), the common case should keep the 
existing "no order guarantee" specification. This gives interpreter authors 
maximum freedom with a fundamental, widely used data type.

> Isn't ordered dict also useful for **kwargs?

If this is useful (and it seems like it would be), I think again a syntax 
modification that allows users to indicate that they want a particular 
implementation of **kwargs would be better than modifying dict semantics. It 
could possibly be handled with a type-hinting like syntax:

def f(*args, **kwargs : OrderedKwargs):

Or a riff on the existing syntax:

def f(*args, ***kwargs):
def f(*args, ^^kwargs):
def f(*args, .**kwargs):

In this case, the only guarantee you'd need (which relatively minor compared to 
a change in the dict semantics) would be that keyword argument order passed to 
a function would be preserved as the order that it is passed into the `kwargs` 
constructor. The old **kwargs syntax would give you a `dict` as normal, and the 
new ^^kwargs would give you an OrderedDict or some other dict subclass with 
guaranteed order.

On 11/05/2017 03:50 PM, Peter Ludemann via Python-Dev wrote:

> 
> 
> On 5 November 2017 at 12:44, Sven R. Kunze  wrote:
> 
>> +1 from me too.
>>
>> On 04.11.2017 21:55, Jim Baker wrote:
>>
>> +1, as Guido correctly recalls, this language guarantee will work well
>> with Jython when we get to the point of implementing 3.7+.
>>
>> On Sat, Nov 4, 2017 at 12:35 PM, Guido van Rossum 
>> wrote:
>>
>>> This sounds reasonable -- I think when we introduced this in 3.6 we were
>>> worried that other implementations (e.g. Jython) would have a problem with
>>> this, but AFAIK they've reported back that they can do this just fine. So
>>> let's just document this as a language guarantee.
>>>
>>> On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah  wrote:
>>>

 Hello,

 would it be possible to guarantee that dict literals are ordered in v3.7?


 The issue is well-known and the workarounds are tedious, example:

https://mail.python.org/pipermail/python-ideas/2015-Decembe
 r/037423.html


 If the feature is guaranteed now, people can rely on it around v3.9.



 Stefan Krah



 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: https://mail.python.org/mailma
 n/options/python-dev/guido%40python.org

>>>
>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido )
>>>
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jbaker%
>>> 40zyasoft.com
>>>
>>>
>>
>>
>> ___
>> Python-Dev mailing 
>> listPython-Dev@python.orghttps://mail.python.org/mailman/listinfo/python-dev
>>
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
>>
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> pludemann%40google.com
>>
>>
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
> 



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Peter Ludemann via Python-Dev]

Isn't ordered dict also useful for **kwargs?

If it turns out that there's a dict implementation that's faster by not
preserving order, collections.UnorderedDict could be added.
There could also be specialized implementations that pre-size the dict (cf:
C++ unordered_map::reserve), etc., etc.
But these are all future things, which might not be necessary.

On 5 November 2017 at 12:44, Sven R. Kunze  wrote:

> +1 from me too.
>
> On 04.11.2017 21:55, Jim Baker wrote:
>
> +1, as Guido correctly recalls, this language guarantee will work well
> with Jython when we get to the point of implementing 3.7+.
>
> On Sat, Nov 4, 2017 at 12:35 PM, Guido van Rossum 
> wrote:
>
>> This sounds reasonable -- I think when we introduced this in 3.6 we were
>> worried that other implementations (e.g. Jython) would have a problem with
>> this, but AFAIK they've reported back that they can do this just fine. So
>> let's just document this as a language guarantee.
>>
>> On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah  wrote:
>>
>>>
>>> Hello,
>>>
>>> would it be possible to guarantee that dict literals are ordered in v3.7?
>>>
>>>
>>> The issue is well-known and the workarounds are tedious, example:
>>>
>>>https://mail.python.org/pipermail/python-ideas/2015-Decembe
>>> r/037423.html
>>>
>>>
>>> If the feature is guaranteed now, people can rely on it around v3.9.
>>>
>>>
>>>
>>> Stefan Krah
>>>
>>>
>>>
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: https://mail.python.org/mailma
>>> n/options/python-dev/guido%40python.org
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido )
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jbaker%
>> 40zyasoft.com
>>
>>
>
>
> ___
> Python-Dev mailing 
> listPython-Dev@python.orghttps://mail.python.org/mailman/listinfo/python-dev
>
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> pludemann%40google.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Sven R. Kunze]

+1 from me too.


On 04.11.2017 21:55, Jim Baker wrote:
+1, as Guido correctly recalls, this language guarantee will work well 
with Jython when we get to the point of implementing 3.7+.


On Sat, Nov 4, 2017 at 12:35 PM, Guido van Rossum > wrote:


This sounds reasonable -- I think when we introduced this in 3.6
we were worried that other implementations (e.g. Jython) would
have a problem with this, but AFAIK they've reported back that
they can do this just fine. So let's just document this as a
language guarantee.

On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah > wrote:


Hello,

would it be possible to guarantee that dict literals are
ordered in v3.7?


The issue is well-known and the workarounds are tedious, example:

https://mail.python.org/pipermail/python-ideas/2015-December/037423.html




If the feature is guaranteed now, people can rely on it around
v3.9.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org





-- 
--Guido van Rossum (python.org/~guido )


___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:
https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul Ganssle]

I think the question of whether any specific implementation of dict could be 
made faster for a given architecture or even that the trade-offs made by 
CPython are generally the right ones is kinda beside the point. It's certainly 
feasible that an implementation that does not preserve ordering could be better 
for some implementation of Python, and the question is really how much is 
gained by changing the language semantics in such a way as to cut off that 
possibility.

On 11/05/2017 02:54 PM, Serhiy Storchaka wrote:
> 05.11.17 21:30, Stefan Krah пише:
>> On Sun, Nov 05, 2017 at 09:09:37PM +0200, Serhiy Storchaka wrote:
>>> 05.11.17 20:39, Stefan Krah пише:
 On Sun, Nov 05, 2017 at 01:14:54PM -0500, Paul G wrote:
> 2. Someone invents a new arbitrary-ordered container that would improve 
> on the memory and/or CPU performance of the current dict implementation

 I would think this is very unlikely, given that the previous dict 
 implementation
 has always been very fast. The new one is very fast, too.
>>>
>>> The modification of the current implementation that don't preserve
>>> the initial order after deletion would be more compact and faster.
>>
>> How much faster?
> 
> I didn't try to implement this. But the current implementation requires 
> periodical reallocating if add and remove items. The following loop 
> reallocates the dict every len(d) iterations, while the size of the dict is 
> not changed, and the half of its storage is empty.
> 
> while True:
>     v = d.pop(k)
>     ...
>     d[k] = v
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Antoine Pitrou]

On Sun, 5 Nov 2017 21:06:12 +0100
Stefan Krah  wrote:

> On Sun, Nov 05, 2017 at 09:35:38PM +0200, Serhiy Storchaka wrote:
> > 05.11.17 21:20, Stefan Krah пише:  
> > >On Sun, Nov 05, 2017 at 09:01:40PM +0200, Serhiy Storchaka wrote:  
> > >>Do you suggest to make dictionary displays producing OrderedDict
> > >>instead of dict?  
> > >
> > >No, this is essentially a language spec doc issue that would guarantee
> > >the ordering properties of the current dict implementation.  
> > 
> > Wouldn't be enough to guarantee just the ordering of dicts before
> > first deletion? Or before first resizing (the maximal size of
> > dictionary displays is known at compile time, so they can be
> > presized)?  
> 
> Yes, for my use case that would be sufficient and that's what
> I had in mind initially.
> 
> 
> A luxury syntax addition like {a = 10, b = {c = "foo"}} that is read
> as an OrderedDict (where the keys a, b, c are implicitly strings) would
> of course also be sufficient for my use case.

Or you are ok with:

  OD = OrderedDict
  # ...
  OD(a=10, b=OD(c="foo"))

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Stefan Krah]

On Sun, Nov 05, 2017 at 09:35:38PM +0200, Serhiy Storchaka wrote:
> 05.11.17 21:20, Stefan Krah пише:
> >On Sun, Nov 05, 2017 at 09:01:40PM +0200, Serhiy Storchaka wrote:
> >>Do you suggest to make dictionary displays producing OrderedDict
> >>instead of dict?
> >
> >No, this is essentially a language spec doc issue that would guarantee
> >the ordering properties of the current dict implementation.
> 
> Wouldn't be enough to guarantee just the ordering of dicts before
> first deletion? Or before first resizing (the maximal size of
> dictionary displays is known at compile time, so they can be
> presized)?

Yes, for my use case that would be sufficient and that's what
I had in mind initially.


A luxury syntax addition like {a = 10, b = {c = "foo"}} that is read
as an OrderedDict (where the keys a, b, c are implicitly strings) would
of course also be sufficient for my use case.


But I suspect many users who have other use cases find it tantalizing
not to be able to use the properties of the current regular dict.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Serhiy Storchaka]

05.11.17 21:30, Stefan Krah пише:

On Sun, Nov 05, 2017 at 09:09:37PM +0200, Serhiy Storchaka wrote:

05.11.17 20:39, Stefan Krah пише:

On Sun, Nov 05, 2017 at 01:14:54PM -0500, Paul G wrote:

2. Someone invents a new arbitrary-ordered container that would improve on the 
memory and/or CPU performance of the current dict implementation


I would think this is very unlikely, given that the previous dict implementation
has always been very fast. The new one is very fast, too.


The modification of the current implementation that don't preserve
the initial order after deletion would be more compact and faster.


How much faster?


I didn't try to implement this. But the current implementation requires 
periodical reallocating if add and remove items. The following loop 
reallocates the dict every len(d) iterations, while the size of the dict 
is not changed, and the half of its storage is empty.


while True:
v = d.pop(k)
...
d[k] = v

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Serhiy Storchaka]

05.11.17 21:20, Stefan Krah пише:

On Sun, Nov 05, 2017 at 09:01:40PM +0200, Serhiy Storchaka wrote:

Do you suggest to make dictionary displays producing OrderedDict
instead of dict?


No, this is essentially a language spec doc issue that would guarantee
the ordering properties of the current dict implementation.


Wouldn't be enough to guarantee just the ordering of dicts before first 
deletion? Or before first resizing (the maximal size of dictionary 
displays is known at compile time, so they can be presized)?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Stefan Krah]

On Sun, Nov 05, 2017 at 09:09:37PM +0200, Serhiy Storchaka wrote:
> 05.11.17 20:39, Stefan Krah пише:
> >On Sun, Nov 05, 2017 at 01:14:54PM -0500, Paul G wrote:
> >>2. Someone invents a new arbitrary-ordered container that would improve on 
> >>the memory and/or CPU performance of the current dict implementation
> >
> >I would think this is very unlikely, given that the previous dict 
> >implementation
> >has always been very fast. The new one is very fast, too.
> 
> The modification of the current implementation that don't preserve
> the initial order after deletion would be more compact and faster.

How much faster?


Stefan Krah


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Stefan Krah]

On Sun, Nov 05, 2017 at 09:01:40PM +0200, Serhiy Storchaka wrote:
> Do you suggest to make dictionary displays producing OrderedDict
> instead of dict?

No, this is essentially a language spec doc issue that would guarantee
the ordering properties of the current dict implementation.


Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Serhiy Storchaka]

05.11.17 20:39, Stefan Krah пише:

On Sun, Nov 05, 2017 at 01:14:54PM -0500, Paul G wrote:

2. Someone invents a new arbitrary-ordered container that would improve on the 
memory and/or CPU performance of the current dict implementation


I would think this is very unlikely, given that the previous dict implementation
has always been very fast. The new one is very fast, too.


The modification of the current implementation that don't preserve the 
initial order after deletion would be more compact and faster.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Serhiy Storchaka]

04.11.17 19:30, Stefan Krah пише:

would it be possible to guarantee that dict literals are ordered in v3.7?


The issue is well-known and the workarounds are tedious, example:

https://mail.python.org/pipermail/python-ideas/2015-December/037423.html


If the feature is guaranteed now, people can rely on it around v3.9.


Do you suggest to make dictionary displays producing OrderedDict instead 
of dict?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul G]

> Scientific applications want something like
> 
>{'a': 10, 'b': "foo", 'c': {'this': b'123'}}
> 
> as an ordered initializer for unboxed or typed (or both) data. In general,
> if dicts are ordered, they can be used for example as initializers for
> (nested) C structs.

I can understand why you'd want an ordered container, I just don't see why it 
must be a dict. Why can't it be:

OrderedDict(a=10, b='foo', c=OrderedDict(this=b'123'))

Is it just that you don't want to type OrderedDict that many times? If it's so 
important to provide ordered dictionary literals, I would think it's a 
no-brainer to give them their own literal syntax (rather than re-defining dicts 
to have guaranteed order). e.g.:

o{'a': 10, 'b': 'foo', 'c': o{'this': b'123'}

Then there is no backwards incompatibility problem and users can express 
whether order does or does not matter to them when initializing a container.

On 11/05/2017 01:39 PM, Stefan Krah wrote:
> On Sun, Nov 05, 2017 at 01:14:54PM -0500, Paul G wrote:
>> I'm not entirely sure I understand the full set of reasoning for this - I 
>> couldn't really tell what the problem with OrderedDict is from the link 
>> Stefan provided. It seems to me like a kind of huge change for the language 
>> to move from arbitrary-ordered to guaranteed-ordered dict. The problem I see 
>> is that this introduces a huge backwards compatibility burden on all 
>> implementations of Python.
> 

> 
> 
> 
>> 2. Someone invents a new arbitrary-ordered container that would improve on 
>> the memory and/or CPU performance of the current dict implementation
> 
> I would think this is very unlikely, given that the previous dict 
> implementation
> has always been very fast. The new one is very fast, too.
> 
> 
> 
> Stefan Krah
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
> 



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Stefan Krah]

On Sun, Nov 05, 2017 at 01:14:54PM -0500, Paul G wrote:
> I'm not entirely sure I understand the full set of reasoning for this - I 
> couldn't really tell what the problem with OrderedDict is from the link 
> Stefan provided. It seems to me like a kind of huge change for the language 
> to move from arbitrary-ordered to guaranteed-ordered dict. The problem I see 
> is that this introduces a huge backwards compatibility burden on all 
> implementations of Python.

Scientific applications want something like

   {'a': 10, 'b': "foo", 'c': {'this': b'123'}}

as an ordered initializer for unboxed or typed (or both) data. In general,
if dicts are ordered, they can be used for example as initializers for
(nested) C structs.



> 2. Someone invents a new arbitrary-ordered container that would improve on 
> the memory and/or CPU performance of the current dict implementation

I would think this is very unlikely, given that the previous dict implementation
has always been very fast. The new one is very fast, too.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Paul G]

I'm not entirely sure I understand the full set of reasoning for this - I 
couldn't really tell what the problem with OrderedDict is from the link Stefan 
provided. It seems to me like a kind of huge change for the language to move 
from arbitrary-ordered to guaranteed-ordered dict. The problem I see is that 
this introduces a huge backwards compatibility burden on all implementations of 
Python.

It's possible that no implementations of Python (either future CPython versions 
or current/future alternative interpreters) will find any reason to use 
anything but an insertion-order sorted dictionary, but given that we've done 
just fine with arbitrary-order semantics for the entire lifetime of the 
language /and/ there is a container (OrderedDict) which has guaranteed order 
semantics, it doesn't seem worth it to me.

I think I would mostly be concerned with (in terms of likeliness to occur):

1. An edge case we haven't thought of where arbitrary order dictionaries would 
allow some crticial optimization for a given platform (perhaps in someone 
writing a transpiler to another language where the convenient equivalent 
container has arbitrary order, e.g. if Brython wants to implement dicts in 
terms of objects - 
https://stackoverflow.com/questions/5525795/does-javascript-guarantee-object-property-order
 )

2. Someone invents a new arbitrary-ordered container that would improve on the 
memory and/or CPU performance of the current dict implementation

3. Some sort of bug or vulnerability is discovered that makes insertion-ordered 
dictionaries an unwise choice (similar to the hash collision vulnerability that 
necessitated hash randomization - 
https://stackoverflow.com/questions/14956313#14959001 ).

Perhaps these concerns are overblown, but if indeed guaranteed-order Mapping 
literals are critical in some or many cases, maybe it would be preferable to 
introduce new syntax for OrderedDict literals.

Best,
Paul

On 11/05/2017 12:50 PM, Guido van Rossum wrote:
> Yup. At least such code will not break in 3.5.
> 
> In general if you write code using a newer version you should expect
> arbitrary breakage when trying to run it under older versions. There is no
> compatibility guarantee in that direction for anything anyways.
> 
> I don't see this as a reason to not put this into the language spec at 3.7.
> 
> On Sun, Nov 5, 2017 at 9:37 AM, Barry Warsaw  wrote:
> 
>> On Nov 4, 2017, at 11:35, Guido van Rossum  wrote:
>>>
>>> This sounds reasonable -- I think when we introduced this in 3.6 we were
>> worried that other implementations (e.g. Jython) would have a problem with
>> this, but AFAIK they've reported back that they can do this just fine. So
>> let's just document this as a language guarantee.
>>
>> The other concern was backward compatibility issues.  For example, if 3.7
>> makes this guarantee official, and folks write code with this assumption
>> that has to work with older versions of Python, then that code could be
>> buggy in previous versions and work in 3.7.  This will probably be most
>> evident in test suites, and such failures are often mysterious to debug (as
>> we’ve seen in the past).
>>
>> That doesn’t mean we shouldn’t do it, but it does mean we have to be
>> careful and explicit to educate users about how to write safe
>> multi-Python-version code.
>>
>> Cheers,
>> -Barry
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> guido%40python.org
>>
>>
> 
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
> 



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Guido van Rossum]

Yup. At least such code will not break in 3.5.

In general if you write code using a newer version you should expect
arbitrary breakage when trying to run it under older versions. There is no
compatibility guarantee in that direction for anything anyways.

I don't see this as a reason to not put this into the language spec at 3.7.

On Sun, Nov 5, 2017 at 9:37 AM, Barry Warsaw  wrote:

> On Nov 4, 2017, at 11:35, Guido van Rossum  wrote:
> >
> > This sounds reasonable -- I think when we introduced this in 3.6 we were
> worried that other implementations (e.g. Jython) would have a problem with
> this, but AFAIK they've reported back that they can do this just fine. So
> let's just document this as a language guarantee.
>
> The other concern was backward compatibility issues.  For example, if 3.7
> makes this guarantee official, and folks write code with this assumption
> that has to work with older versions of Python, then that code could be
> buggy in previous versions and work in 3.7.  This will probably be most
> evident in test suites, and such failures are often mysterious to debug (as
> we’ve seen in the past).
>
> That doesn’t mean we shouldn’t do it, but it does mean we have to be
> careful and explicit to educate users about how to write safe
> multi-Python-version code.
>
> Cheers,
> -Barry
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Brett Cannon]

On Fri, 3 Nov 2017 at 17:59 Lukasz Langa  wrote:

>
> > On 3 Nov, 2017, at 4:01 PM, Victor Stinner 
> wrote:
> >
> > The question is if you would only need  or  > pip install typing>.
> >
> > If typing is removed from the stdlib, you can still use it in your
> > application. It's "just" another dependency no?
>
>
> The ideal situation is that something is built-in and just works,
> examples: dicts, lists, sorted().
>
> So, if you have to import it to use it, it's still great but less
> seamless, current example: regular expressions. Let's say Guido suggests we
> should import sorted, dict, and list before use. Not a big deal, right? I
> mean, how many applications do you know that don't use any other imports?
>
> Finally, if you have to find a third-party package, add it to
> requirements.txt and manage the dependency forward, that's even less
> seamless. The standard library has a pretty conservative approach to
> backwards compatibility. On the other hand, third-party libraries often
> don't. Sure, there's noble exceptions but the general feel is that you need
> to be more careful with dependencies from PyPI. If somebody suggested that
> regular expressions or dictionaries should be moved to PyPI, in my book
> that would suggest strange things will start happening in the future.
>
> So, the difference is in perceived usability. It's psychological.
>

I think another psychological side-effect of removing 'typing' is how
strongly people will view the guidance that annotations are generally
expected to be type hints. With 'typing' in the stdlib, it made that
assumption a bit strong, especially if the stdlib ever started to use type
hints itself. But by saying "we expect annotations will be used for type
hints, but we actually don't do that in the stdlib nor provide a mechanism
for actually specifying type hints", that weakens the message. It starts to
feel like a "do as I say, not as I do" bit of guidance (which to me is
different than the worry that the usage of type hints will decrease).

But that might be fine; I personally don't know. But my suspicion is we
will see an uptick of alternative annotation uses that aren't type hints if
we take 'typing' out, and that's something we should be aware of and okay
with.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 4, 2017, at 11:35, Guido van Rossum  wrote:
> 
> This sounds reasonable -- I think when we introduced this in 3.6 we were 
> worried that other implementations (e.g. Jython) would have a problem with 
> this, but AFAIK they've reported back that they can do this just fine. So 
> let's just document this as a language guarantee.

The other concern was backward compatibility issues.  For example, if 3.7 makes 
this guarantee official, and folks write code with this assumption that has to 
work with older versions of Python, then that code could be buggy in previous 
versions and work in 3.7.  This will probably be most evident in test suites, 
and such failures are often mysterious to debug (as we’ve seen in the past).

That doesn’t mean we shouldn’t do it, but it does mean we have to be careful 
and explicit to educate users about how to write safe multi-Python-version code.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Reminder: 12 weeks to 3.7 feature code cutoff

[Yury Selivanov]

On Fri, Nov 3, 2017 at 11:30 PM, Nick Coghlan  wrote:
> On 4 November 2017 at 09:52, Jelle Zijlstra  wrote:
>>
>> 2017-11-03 16:44 GMT-07:00 Joao S. O. Bueno :
>>>
>>> This just popped up in Brython's issue tracker discussion:
>>>
>>> """
>>> Pierre Quentel 
>>>
>>> 04:57 (16 hours ago)
>>> to brython-dev/br., Subscribed
>>>
>>> I think it's better to rename all occurences of async now, although
>>> it's strange that :
>>>
>>> there is currently no deprecation warning in CPython with code that
>>> uses it as a variable name, PEP492 said that "async and await names
>>> will be softly deprecated in CPython 3.5 and 3.6"
>>> there is no mention of async and await becoming keywords in What's new
>>> in Python 3.7
>>>
>>> Maybe the idea was finally given up, but I can't find a reference.
>>>
>>> """
>>>
>>> So, what is the status of promoting async and await to full keyword for
>>> 3.7?
>>>
>> This was implemented, and it's in NEWS:
>> https://github.com/python/cpython/pull/1669.
>
> That's a big enough change that it should be in What's New as well (at
> least in the porting section, and probably more prominent than that).
>
> The current lack of DeprecationWarnings in 3.6 is a fairly major
> oversight/bug, though:

There's no oversight.  We had PendingDeprecationWarning for
async/await names in 3.5, and DeprecationWarning in 3.6.  You just
need to enable warnings to see them:

~ » python3 -Wall
Python 3.6.2 (default, Aug  2 2017, 22:29:27)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> async = 1
:1: DeprecationWarning: 'async' and 'await' will become
reserved keywords in Python 3.7


> So if we're going to go ahead with making them real keywords in 3.7
> (as specified in PEP 492), then the missing DeprecationWarning problem
> in 3.6 needs to be fixed.

They are already keywords in 3.7, I've committed that change a month ago.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com