Re: [Python-Dev] PEP 487: Simpler customization of class creation

[Nikita Nemkin]

On Mon, Jun 20, 2016 at 9:48 PM, Guido van Rossum <gu...@python.org> wrote:
> On Thu, Jun 16, 2016 at 3:24 PM, Nikita Nemkin <nik...@nemkin.ru> wrote:
>>
>> I didin't know that PyPy has actually implemented packed ordered dicts!
>>
>> https://morepypy.blogspot.ru/2015/01/faster-more-memory-efficient-and-more.html
>> https://mail.python.org/pipermail/python-dev/2012-December/123028.html
>>
>> This old idea by Raymond Hettinger is vastly superior to
>> __definition_order__ duct tape (now that PyPy has validated it).
>> It also gives kwarg order for free, which is important in many
>> metaprogramming scenarios.
>> Not to mention memory usage reduction and dict operations speedup...
>
>
> That idea is only vastly superior if we want to force all other Python
> implementations to also have an order-preserving dict with the same
> semantics and API.

Right. Ordered by default is a very serious implementation constraint.
It's only superior in a sense that it completely subsumes/obsoletes
PEP 520.

> I'd like to hear more about your metaprogramming scenarios -- often such
> things end up being code the author is ashamed of. Perhaps they should stay
> in the shadows? Or could we do something to make it so you won't have to be
> ashamed of it?

What I meant is embedding declarative domain-specific languages
in Python. Examples of such languages include SQL table
definitions, binary data definitions (in-memory C structs or
wire protocol), GUI definitions (look up enaml for an interesting
example), etc. etc. DSLs are a well defined field and the point
of embedding into Python is to implement in Python and to
empower DSL with Python constructs for generation and logic.

Basic blocks for a declarative language are lists and "objects" -
groups of ordered, named fields.

Representing lists is easy and elegant, commas make a tuple
and [] makes a list.

It's when trying to represent "objects" the issues arise.
Literal dicts are "ugly" (for DSL purposes) and unordered.
Lists of 2-tuples are even uglier. Py3 gave us __prepare__ for
ordered class bodies, and this became a first valid option.
For example, SQL table:

class MyTable(SqlTable):
field1 = Type1(options...)
field2 = Type2()

Unfortunately, class declarations don't look good when nested,
and nesting is a common thing.

class MainWindow:
caption = "Window"
class HSplit:
label1 = Label(...)
text1 = Text(...)

You get the idea.
Another option for expressing "objects" are function calls with kwargs:

packet = Struct(type=uint8,
length=uint32,
body=Array(uint8, 'type'))

Looks reasonably clean, but more often than not requires kwargs
to be ordered. THIS is the scenario I was talking about.

Function attributes also have a role, but being
attached to function definitions, their scope is somewhat limited.

Of course, all of the above is largely theoretical, for two basic
reasons:
1) Python syntax/runtime is too rigid for a declarative DSL.
   (Specifically, _embedded_ DSL. The syntax alone can be re-used
   with ast.parse, but it's a different scenario.)
2) DSLs in general are grossly unpythonic, hiding loads of magic
   and unfamiliar semantics behind what looks like a normal Python.
   It's not something to be ashamed of, but the benefit
   rarely justifies the (maintenance) cost.

To be clear: I'm NOT advocating for ordered kwargs. Embedding
DSLs into Python is generally a bad idea.

PS. __prepare__ enables many DSL tricks. In fact, it's difficult to
imagine a use case that's not related to some attempt at DSL.
Keyword-only args also help: ordered part of the definition can go
into *args, while attributes/options are kw-only args.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 487: Simpler customization of class creation

[Nikita Nemkin]

On Fri, Jun 17, 2016 at 2:36 AM, Nick Coghlan  wrote:
> On 16 June 2016 at 14:17, Martin Teichmann  wrote:

> An implementation like PyPy, with an inherently ordered standard dict
> implementation, can just rely on that rather than being obliged to
> switch to their full collections.OrderedDict type.

I didin't know that PyPy has actually implemented packed ordered dicts!
https://morepypy.blogspot.ru/2015/01/faster-more-memory-efficient-and-more.html
https://mail.python.org/pipermail/python-dev/2012-December/123028.html

This old idea by Raymond Hettinger is vastly superior to __definition_order__
duct tape (now that PyPy has validated it).
It also gives kwarg order for free, which is important in many metaprogramming
scenarios.
Not to mention memory usage reduction and dict operations speedup...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 520: Ordered Class Definition Namespace

[Nikita Nemkin]

I'll reformulate my argument:

Ordered class namespaces are a minority use case that's already covered
by existing language features (custom metaclasses) and doesn't warrant
the extension of the language (i.e. making OrderedDict a builtin type).
This is about Python-the-Language, not CPython-the-runtime.

If you disagree with this premise, there's no point arguing about
the alternatives. That being said, below are the answers to your objections
to specific alternatives.

On Thu, Jun 16, 2016 at 1:30 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 14 June 2016 at 02:41, Nikita Nemkin <nik...@nemkin.ru> wrote:
>
> Adding metaclasses to an existing class can break compatibility with
> third party subclasses, so making it possible for people to avoid that
> while still gaining the ability to implicitly expose attribute
> ordering to class decorators and other potentially interested parties
> is a recurring theme behind this PEP and also PEPs 422 and 487.

The simple answer is "don't do that", i.e. don't pile an ordered metaclass
on top of another metaclass. Such use case is hypothetical anyway.

Also, namespace argument to the default metaclass doesn't cause conflicts.

>> 3. Making compiler fill in __definition_order__ for every class
>> (just like __qualname__) without touching the runtime.
>> ?
>
> Class scopes support conditionals and loops, so we can't necessarily
> be sure what names will be assigned without running the code. It's
> also possible to make attribute assignments via locals() that are
> entirely opaque to the compiler, but visible to the interpreter at
> runtime.

All explicit assignments in the class body can be detected statically.
Implicit assignments via locals(), sys._frame() etc. can't be detected,
BUT they are unlikely to have a meaningful order!
It's reasonable to exclude them from __definition_order__.

This also applies to documentation tools. If there really was a need,
they could have easily extracted static order, solving 99.% of
the problem.

> The rationale for "Why not make this configurable, rather than
> switching it unilaterally?" is that it's actually *simpler* overall to
> just make it the default - we can then change the documentation to say
> "class bodies are evaluated in a collections.OrderedDict instance by
> default" and record the consequences of that, rather than having to
> document yet another class customisation mechanism.

It would have been a "simpler" default if it was the core dict that
became ordered. Instead, it brings in a 3rd party (OrderedDict).

Documenting an extra metaclass or an extra type kward would hardly
take more space. And it's NOT yet another mechanism. It's the good old
metaclass mechanism.

> It also eliminates boilerplate from class decorator usage
> instructions, where people have to write "to use this class decorator,
> you must also specify 'namespace=collections.OrderedDict' in your
> class header"

Statically inferred __definition_order__ would work here.
Order-dependent decorators don't seem to be important enough
to worry about their usability.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 520: Ordered Class Definition Namespace

[Nikita Nemkin]

Is there any rationale for rejecting alternatives like:

1. Adding standard metaclass with ordered namespace.
2. Adding `namespace` or `ordered` args to the default metaclass.
3. Making compiler fill in __definition_order__ for every class
(just like __qualname__) without touching the runtime.
?

To me, any of the above seems preferred to complicating
the core part of the language forever.

The vast majority of Python classes don't care about their member
order, this is minority use case receiving majority treatment.

Also, wiring OrderedDict into class creation means elevating it
from a peripheral utility to indispensable built-in type.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] MAKE_FUNCTION simplification

[Nikita Nemkin]

On Thu, Apr 14, 2016 at 8:27 PM, Guido van Rossum  wrote:
> Great analysis! What might stand in the way of adoption is concern for
> bytecode manipulation libraries that would have to be changed. What
> might encourage adoption would be a benchmark showing this saves a lot
> of time.
>
> Personally I'm expecting it won't make much of a difference for real
> programs since almost always the cost of creating the function is
> dwarfed by the (total) cost of running it. But Python does create a
> lot of functions, and there's also lambdas.

This change alone is very unlikely to have a measurable performance impact.
The intention is to clean up ceval.c/compile.c a bit, nothing more.
If many other opcodes were somehow slimmed down in the similar fashion,
then we might (or might not) see perf gains.

For example, most slot dispatch opcodes can be compressed into a single
opcode+slot index with inlined dispatch logic, instead of each one individually
calling C API functions...

> There's also talk of switching to wordcode, in a different thread.
> Maybe the idea would be easier to introduce there? (Bytecode libraries
> would have to change anyways, so the additional concern for this
> change would be minimal.)

Wordcode can benefit from this change, because it guarantees
single-byte MAKE_FUNCTION oparg.

I think that Python should make bytecode explicitly unstable and subject
to change with any major release. The potential for a faster Python
interpreter (or simple JIT) is huge; requiring bytecode compatibility
will slow down any progress in this area.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] MAKE_FUNCTION simplification

[Nikita Nemkin]

On Thu, Apr 14, 2016 at 8:32 PM, Victor Stinner
 wrote:
>
> Would you like to work on a patch to implement that change?

I'll work on a patch. Should I post it to bugs.python.org?

> Since Python 3.6 may get a new bytecode format  (wordcode, see the
> other thread on this mlailing list), I think that it's ok to change
> MAKE_FUNCTION in the same release.

Wordcode looks like pure win from (projected) 25% bytecode size
reduction alone.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] MAKE_FUNCTION simplification

[Nikita Nemkin]

MAKE_FUNCTION opcode is complex due to the way it receives
input arguments:

 1) default args, individually;
 2) default kwonly args, individual name-value pairs;
 3) a tuple of parameter names (single constant);
 4) annotation values, individually;
 5) code object;
 6) qualname.

The counts for 1,2,4 are packed into oparg bitfields, making oparg large.

My suggestion is to pre-package 1-4 before calling MAKE_FUNCTION,
i.e. explicitly emit BUILD_TUPLE for defaults args and BUILD_MAPs
for keyword defaults and annotations.

Then, MAKE_FUNCTION will become a dramatically simpler
5 argument opcode, taking

 1) default args tuple (optional);
 2) default keyword only args dict (optional);
 3) annotations dict (optional);
 4) code object;
 5) qualname.

These arguments correspond exactly to __annotations__, __kwdefaults__,
__defaults__, __code__ and __qualname__ attributes.

For optional args, oparg bits should indicate individual arg presence.
(This also saves None checks in opcode implementation.)

If we add another optional argument (and oparg bit) for __closure__
attribute, then separate MAKE_CLOSURE opcode becomes unnecessary.

Default args tuple is likely to be a constant and can be packaged whole,
compensating for the extra size of explicit BUILD_* instructions.

Compare the current implementation:

https://github.com/python/cpython/blob/master/Python/ceval.c#L3262

with this provisional implementation (untested):

TARGET(MAKE_FUNCTION) {
PyObject *qualname = POP();
PyObject *codeobj = POP();
PyFunctionObject *func;
func = (PyFunctionObject *)PyFunction_NewWithQualName(
   codeobj, f->f_globals, qualname);
Py_DECREF(codeobj);
Py_DECREF(qualname);
if (func == NULL)
goto error;

/* NB: Py_None is not an acceptable value for these. */
if (oparg & 0x08)
func->func_closure = POP();
if (oparg & 0x04)
func->func_annotations = POP();
if (oparg & 0x02)
func->func_kwdefaults = POP();
if (oparg & 0x01)
func->func_defaults = POP();

PUSH((PyObject *)func);
DISPATCH();
}

compile.c also gets a bit simpler, but not much.

What do you think?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com