Re: [Python-ideas] Python docs page: In what ways is None special

[C Anthony Risinger]

On Jul 23, 2018 8:43 PM, "Chris Barker - NOAA Federal via Python-ideas" <
python-ideas@python.org> wrote:


> Procedures return None
> ==
 a = [3,1,2]
 b = a.sort()
 a, b
> ([1, 2, 3], None)

This is less about None than about the convention that mutating
methods return None. Maybe that discussion belongs elsewhere.


> None is default return value
> =
 def fn(): pass
> ...
 fn() # No response!
 print(fn()) # Here's why.
> None

Yup.


I believe these two are related and an artifact of how code/function
objects always leave *something* on TOS/top-of-stack.

IIRC even module objects have a discarded "return value" (in CPython at
least). This internal, unseen, and very-much-special-syntax-worthy value,
is None other.

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[C Anthony Risinger]

On Mon, Jul 23, 2018 at 7:46 AM, Grégory Lielens 
wrote:

>  Paul Moore wrote:
>>
>> This is my impression, as well. It seems like something that's helpful
>> in dealing with unstructured object hierarchies with lots of optional
>> attributes - which is where JSON tends to be used.
>>
>> But given that, I'm really much more interested in seeing the new
>> operators compared against a well-written "JSON object hierarchy
>> traversal" library than against raw Python code.
>>
>
> Good point, I did not think about that when suggesting to give some
> context, but indeed if it's linked to a library in particular, there is
> always the possibility to choose another object than None as the "nothing
> here" marker.
>
>From what I gather, disallowing reassignment all but cements None as the
"nothing here" marker (the originally intent?), _especially_ when "nothing
here" is candidate for replacement by a more appropriate, type- or domain-
or context-specific "nothing here" marker. Imbuing None with any other
meaning brings nothing but headache. It's an attractive nuisance. None is
None is None, there can be only one! You don't know what it is, only that
it's not anything else. What possible meaning can such an object usefully
have in an application built on disparate libraries with their own ideas on
the matter?

Every API I've used (apologies for coming up blank on a concrete example!)
granting None meaning is awkward to consume. `.get()` interfaces are less
useful (must carry your own internal sentinels) or more try/except blocks
are required to achieve the same end (not a bad thing per se, but
a diminished experience to be sure). Early APIs I wrote in this "well it's
None, but but but, with this convenient meaning, see?! SEE??"-style were
later regarded -- quite thoroughly -- as a bad idea by myself, my peers,
and downstream consumers.

I'd personally use these new operators both frequently and judiciously.
They align very well with a "set 'em up and knock 'em down"-style I use:
normalized, progressively conditioned input values fast-followed by
aggressive short-circuits and clear early returns. IME this pattern
generates clean, readable, and performant code. Honestly the
short-circuiting capability alone is enough to sell me :-) This PEP would
find legitimate use by me every day. I'm not 100% sold on `?[` (then again,
attributes are pulled from an object namespace via `?.` and namespaces are
containers by definition) but `?.` and `??` deliver immense value.

Not sure if useful, but this discussion reminds me of a pattern prevalent
in the Elixir community. They use `?` and `!` in function definitions to
denote variants differing only on return behavior (not function clauses!
This is by convention only, they're entirely new functions with a symbol in
their name). It looks something like this:

# Default function.
# Return a tuple {interesting-or-nil, error-or-nil}.
def open(path) do ... end

# Maybe variant.
# Return a boolean, or less often, interesting-or-nil (replaces `is_` or
`can_` methods in Python).
def open?(path) do ... end

# Forceful variant.
# Return interesting or die trying (inverse of `.get()` methods in Python;
raising is not the default expectation in Elixir).
def open!(path) do ... end

The `?.`-operator reminds me of this. It offers to perform an extremely
common operation (simple attribute access) while short-circuiting on the
most frequently triggered guard condition (AttributeError).

I don't think the utility here is restricted to deeply nested JSON
`loads()` or one-off scripts. It better aligns the community on semantics,
encouraging more idiomatic -- and uniform! -- interactions with None.

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A "local" pseudo-function

[C Anthony Risinger]

let[x](EXPR)
x == EXPR

let[x](a=1)
x == 1

let[x](a=1, EXPR)
x == EXPR

let[x, y](a=1, EXPR)
x == 1
y == EXPR


let[x, y](a=1, b=2, EXPR)
x == 2
y == EXPR

z = let[x, y](a=1, EXPR)
x == 1
y == EXPR
z == (1, EXPR)

Anybody seeing how the above might be useful, and address some of the
concerns I've read? I don't recall seeing this suggested prior.

I like the idea behind pseudo-function let/local, especially when paired
with the explanation of equal sign precedence changes within paren, but I'm
having a really hard time getting over the name binding leaking out of the
paren.

I like this item-ish style because it puts the name itself outside the
parentheses while still retaining the benefits in readability. It also
allows capturing the entire resultset, or individual parts.

On Sat, Apr 28, 2018, 6:00 PM Tim Delaney 
wrote:

> ​​
> On Sat, 28 Apr 2018 at 12:41, Tim Peters  wrote:
>
> My big concern here involves the:
>
> ​​if local(m = re.match(regexp, line)):
> print(m.group(0))
>
> example. The entire block needs to be implicitly local for that to work -
> what happens if I assign a new name in that block? Also, what happens with:
>
> ​​if local(m = re.match(regexp1, line)) or local(m = re.match(regexp2,
> line)):
> print(m.group(0))
>
> Would the special-casing of local still apply to the block? Or would you
> need to do:
>
> ​​if local(m = re.match(regexp1, line) or re.match(regexp2, line)):
> print(m.group(0))
>
> ​This might just be lack of coffee and sleep talking, but maybe new
> "scoping delimiters" could be introduced. Yes - I'm suggesting introducing
> curly braces for blocks, but with a limited scope (pun intended).  Within
> a local {} block statements and expressions are evaluated exactly like they
> currently are, including branching statements, optional semi-colons, etc.
> The value returned from the block is from an explicit return, or the last
> evalauted expression.
>
> a = 1
>> b = 2
>> c =
>> ​​
>> local(a=3) * local(b=4)
>>
>
> ​c = ​local { a=3 } * local { b=4 }
>
> c =
>> ​​
>> local(a=3
>> ​,
>>  b=4
>> ​,​
>> a*b)
>
>
> ​​
> ​c = ​
> ​
> ​
> ​
> local
> ​ {
> a=3
> ​;​
> b=4
> ​;​
> a*b
> ​ }​
> ​
>
> ​
> ​c = ​
> ​
> ​
> local
> ​ {
> a = 3
> b = 4
> a * b
> ​
> }​
>
> c = local(a=3,
>> ​​
>> b=local(a=2, a*a), a*b)
>
> ​
> ​
> ​c = ​
> ​
> ​
> local
> ​ {
> a = 3
> b = local(a=2, a*a)
> return a * b
> ​
> }​
>
>
>> ​​
>> r1, r2 = local(D = b**2 - 4*a*c,
>>sqrtD = math.sqrt(D),
>>twoa = 2*a,
>>((-b + sqrtD)/twoa, (-b - sqrtD)/twoa))
>>
>
> ​
> ​
> r1, r2 = local {
> D = b**2 - 4*a*c
> sqrtD = math.sqrt(D)
> twoa = 2*a
> return ((-b + sqrtD)/twoa, (-b - sqrtD)/twoa)
> }
>
> ​​
>> if local(m = re.match(regexp, line)):
>> print(m.group(0))
>>
>
> ​
> if local { m = re.match(regexp, line) }:
> print(m.group(0))
>
> And a further implication:
>
> a = lambda a, b: local(c=4, a*b*c)
>
> a = lambda a, b: local {
> c = 4
> return a * b * c
> }
>
> Tim Delaney
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>

On Apr 28, 2018 6:00 PM, "Tim Delaney"  wrote:

​​
On Sat, 28 Apr 2018 at 12:41, Tim Peters  wrote:

My big concern here involves the:

​​if local(m = re.match(regexp, line)):
print(m.group(0))

example. The entire block needs to be implicitly local for that to work -
what happens if I assign a new name in that block? Also, what happens with:

​​if local(m = re.match(regexp1, line)) or local(m = re.match(regexp2, line)
):
print(m.group(0))

Would the special-casing of local still apply to the block? Or would you
need to do:

​​if local(m = re.match(regexp1, line) or re.match(regexp2, line)):
print(m.group(0))

​This might just be lack of coffee and sleep talking, but maybe new
"scoping delimiters" could be introduced. Yes - I'm suggesting introducing
curly braces for blocks, but with a limited scope (pun intended).  Within a
local {} block statements and expressions are evaluated exactly like they
currently are, including branching statements, optional semi-colons, etc.
The value returned from the block is from an explicit return, or the last
evalauted expression.

a = 1
> b = 2
> c =
> ​​
> local(a=3) * local(b=4)
>

​c = ​local { a=3 } * local { b=4 }

c =
> ​​
> local(a=3
> ​,
>  b=4
> ​,​
> a*b)


​​
​c = ​
​
​
​
local
​ {
a=3
​;​
b=4
​;​
a*b
​ }​
​

​
​c = ​
​
​
local
​ {
a = 3
b = 4
a * b
​
}​

c = local(a=3,
> ​​
> b=local(a=2, a*a), a*b)

​
​
​c = ​
​
​
local
​ {
a = 3
b = local(a=2, a*a)
return a * b
​
}​


> ​​
> r1, r2 = local(D = b**2 - 4*a*c,
>sqrtD = math.sqrt(D),
>twoa = 2*a,
>((-b + sqrtD)/twoa, (-b - 

Re: [Python-ideas] Non-intrusive debug logging

[C Anthony Risinger]

On Thu, Jan 25, 2018 at 5:09 PM, Nick Timkovich 
wrote:

> I think part of the reason that logging appears complicated is because
> logging actually is complicated. In the myriad different contexts a Python
> program runs (daemon, command line tool, interactively), the logging output
> should be going to all sorts of different places. Thus was born handlers.
> If you want "all the logs", do you really want all the logs from some
> library and the library it calls? Thus was born filters.
>
> For better or worse, the "line cost" of a logging call encourages them to
> be useful.
>

I think that last bit is the OP's primary ask. Truly great and useful logs
are genuinely hard to write. They want a cross-cutting, differentially
updated context that no single section of code a) cares about or b)
willingly wants to incur the costs of... especially when unused.

In my mind, the most significant barrier to fantastic logging -- DEBUG in
particular -- is you must 100% understand, ahead-of-time, which data and in
which situations will yield solutions to unknown future problems, and then
must limit that extra data to relevant inputs only (eg. only DEBUG logging
a specific user or condition), ideally for a defined capture window, so you
avoid writing 100MBps to /dev/null all day long. While this proposal does
limit the line noise I don't think it makes logging any more accessible, or
useful.

The desire to tap into a running program and dynamically inspect useful
data at the time it's needed is what led to this:

Dynamic logging after the fact (injects "logging call sites" into a target
function's __code__)
https://pypi.python.org/pypi/retrospect/0.1.4

It never went beyond a basic POC and it's not meant to be a plug, only
another point of interest. Instead of explicitly calling out to some
logging library every 0.1 lines of code, I'd rather attach "something" to a
function, as needed, and tell it what I am interested in (function args,
function return, symbols, etc). This makes logging more like typing, where
you could even move logging information to a stub file of sorts and bind it
with the application, using it... or not! Sort of like a per-function
sys.settrace.

I believe this approach (ie. with something like __code__.settrace) could
fulfill the OP's original ask with additional possibilities.

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] How assignment should work with generators?

[C Anthony Risinger]

On Nov 29, 2017 10:09 PM, "Steven D'Aprano"  wrote:

> On Thu, Nov 30, 2017 at 11:21:48AM +1300, Greg Ewing wrote:
>
> > It seems that many people think about unpacking rather
> > differently from the way I do. I think the difference
> > is procedural vs. declarative.
> >
> > To my way of thinking, something like
> >
> >a, b, c = x
> >
> > is a pattern-matching operation. It's declaring that
> > x is a sequence of three things, and giving names to
> > those things. It's not saying to *do* anything to x.
>
> I hadn't thought of that interpretation before, but now that Greg
> mentions it, its so obvious-in-hindsight that I completely agree with
> it. I think that we should promote this as the "one obvious"
> interpretation.
>
> Obviously as a practical matter, there are some x (namely iterators)
> where you cannot extract items without modifying x, but in all other
> cases I think that the pattern-matching interpretation is superiour.
>

This conversation about suitcases, matching, and language assumptions is
interesting. I've realized two concrete things about how I understand
unpacking, and perhaps, further explain the dissonance we have here:

* Unpacking is destructuring not pattern matching.
* Tuple syntax is commas, paren, one, or both.

For the former, destructuring, this reply conveys my thoughts verbatim:
https://groups.google.com/forum/#!topic/clojure/SUoThs5FGvE

"There are two different concerns in what people refer to as "pattern
matching": binding and flow-control. Destructuring only addresses binding.
Pattern matching emphasizes flow control, and some binding features
typically come along for free with whatever syntax it uses. (But you could
in principle have flow control without binding.)"

The only part of unpacking that is 'pattern matching' is the fact that it
blows up spectacularly when the LHS doesn't perfectly match the length of
RHS, reversing flow via exception:

>>> 0,b = 0,1
  File "", line 1
SyntaxError: can't assign to literal

If Python really supported pattern matching (which I would 100% love! yes
please), and unpacking was pattern matching, the above would succeed
because zero matches zero. Pattern matching is used extensively in
Erlang/Elixir for selecting between various kinds of clauses (function,
case, etc), but you also see *significant* use of the `[a, b | _] = RHS`
construct to ignore "the remainder" because 99% of the time what you really
want is to [sometimes!] match a few things, bind a few others, and ignore
what you don't understand or need. This is why destructuring Elixir maps or
JS objects never expect (or even support AFAIK) exact-matching the entire
object... it would render this incredible feature next to useless!
*Destructuring is opportunistic if matching succeeds*.

For the latter, tuples-are-commas-unless-they-are-parens :-), I suspect I'm
very much in the minority here. While Python is one of my favorite
languages, it's only 1 of 10, and I didn't learn it until I was already 4
languages deep. It's easy to forget how odd tuples are because they are so
baked in, but I've had the "well, ehm, comma is the tuple constructor...
usually" or "well, ehm, you are actually returning 1 tuple... not 2 things"
conversation with *dozens* of seasoned developers. Even people
professionally writing Python for months or more. Other languages use more
explicit, visually-bounded object constructors. This makes a meaningful
difference in how a human's intuition interprets the meaning of a new
syntax. *Objects start and end but commas have no inherent boundaries*.

These two things combined lead to unpacking problems because I look at all
assignments through the lens of destructuring (first observation)
and unpacking almost never uses parentheses (second observation). To
illustrate this better, the following is how my mind initially parses
different syntax contrasted with what's actually happening (and thus the
correction I've internalized over a decade):

>>> a, b = 5, 6
CONCEPT ... LHS[0] `a` bound to RHS[0] `5`, LHS[1] `b` bound to RHS[1] `6`.
REALITY ... LHS[:] single tuple destructured to RHS[:] single tuple.

>>> a, b = 5, 6, 7
CONCEPT ... LHS[0] `a` bound to RHS[0] `5`, LHS[1] `b` bound to RHS[1]
`6`, RHS[2] `6` is unbound expression.
REALITY ... LHS[:] single tuple destructured to RHS[:] single tuple, bad
match, RAISE ValueError!

>>> (a, b) = 5, 6
>>> [a, b] = 5, 6
CONCEPT ... LHS[0] `(a, b)` bound to RHS[0] `5`, bad match, RAISE TypeError!
REALITY ... LHS[:] single tuple destructured to RHS[:] single tuple.

>>> a, b = it
>>> [a], b = it
CONCEPT ... LHS[0] `a` bound with RHS[0] `it`, LHS[1] is bad match, RAISE
UnboundLocalError/NameError!
REALITY ... `a` bound to `it[0]` and `b` bound to `it[1]` (`it[N]` for
illustration only!)

The tuple thing in particular takes non-zero time to internalize. I
consider it one of Python's warts, attributed to times explained and
comparisons with similar languages. Commas at the top-level, with no other

Re: [Python-ideas] How assignment should work with generators?

[C Anthony Risinger]

On Mon, Nov 27, 2017 at 3:49 PM, Greg Ewing <greg.ew...@canterbury.ac.nz>
wrote:

> C Anthony Risinger wrote:
>
>> * Perhaps existence of `__len__` should influence unpacking? There is a
>> semantic difference (and typically a visual one too) between 1-to-1
>> matching a fixed-width sequence/container on the RHS to identifiers on the
>> LHS, even if they look similar (ie. "if RHS has a length make it fit,
>> otherwise don't").
>>
>
> -1. There's a convention that an iterator can implement __len__
> to provide a hint about the number of items it will return
> (useful for preallocating space, etc.) It would be very
> annoying if such an iterator behaved differently from other
> iterators when unpacking.
>

Is __len__ a viable option now that __length_hint__ has been identified for
hints? IOW, if it's possible to know the full length of RHS ahead of time,
and the LHS is also fixed width, then unpack like today else unpack lazily.

This does make unpacking behave slightly different if the RHS is `Sized`
(per ABC, has __len__), but that doesn't seem too unreasonable. It's
different after all.

Using Ellipsis, eg. `a, b, ... = it()`, seems like a OK approach too but
it's unfortunate we are effectively working around the current
force-matching behavior of unpacking... is this the appropriate default? Is
there precedence or guidance elsewhere?

`a, b, ...` to me says "pull out a and b and throw away the rest", sort of
like the spread operator in JS/ECMA. The mere presence of more characters
(...) implies something else will *happen* to the remaining items, not that
they will be skipped.

What about the explicit RHS unpacking idea? Kirill commented on this
approach but I'm not sure others did:

>>> a, b = iter([1, 2, 3]); a, b
(1, 2)

>>> a, b = *iter([1, 2, 3]); a, b
Traceback (most recent call last):
  File "", line 1, in 
ValueError: too many values to unpack (expected 2)

This reads very clearly to me that the RHS is expected to be 100%
unraveled, and would work with any iterable in the same way.

Thanks,
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] How assignment should work with generators?

[C Anthony Risinger]

On Nov 28, 2017 12:32 AM, "Steven D'Aprano"  wrote:

On Tue, Nov 28, 2017 at 06:15:47PM +1300, Greg Ewing wrote:
> Steven D'Aprano wrote:
> >How does "stop iterating here" equate to a wildcard?
>
> The * means "I don't care what else the sequence has in it".
>
> Because I don't care, there's no need to iterate any further.

I'll grant you that. But I don't see how that relates to being a
wildcard. I'm not seeing the connection. I mean, you wouldn't interpret

   from module import *

to mean "I don't care what's in the module, so don't bother importing
anything" would you? So the concept of wildcard here seems to be the
opposite to its use here:

- in imports, it means "import everything the module offers";
- in extended iterable unpacking, it means "collect everything";

both of which (to me) seem related to the concept of a wildcard; but in
this proposed syntax, we have

   x, y, * = iterable

which means the opposite to "collect everything", instead meaning
"collect nothing and stop".

Anyway, given that * = is visually ambiguous with *= I don't think this
syntax is feasible (even though * = is currently a syntax error, or at
least it is in 3.5).


If not already considered, what if the RHS had to be explicitly unpacked?

Something like:

a, b, c = *iterator

Which would essentially be:

a, b, c = (*iterator,)

This enables lazy assignment by default but `*` can force complete
expansion (and exact matching) of the RHS.

It's a breaking change, but it does have a straightforward fix (simply wrap
and unpack any relevant RHS).

Thanks,

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] How assignment should work with generators?

[C Anthony Risinger]

This proposal resonates with me. I've definitely wanted to use unpacking to
crank an iterator a couple times and move on without exhausting the
iterator. It's a very natural and intuitive meaning for unpacking as it
relates to iterators.

In my mind, this ask is aligned with, and has similar motivation to, lazy
zip(), map(), and keys() in Python 3. Syntax support for unpacking as it
stands today is not very conducive to iterables and conflicts with a
widespread desire to use more of them.

Couple thoughts:

* Perhaps existence of `__len__` should influence unpacking? There is a
semantic difference (and typically a visual one too) between 1-to-1
matching a fixed-width sequence/container on the RHS to identifiers on the
LHS, even if they look similar (ie. "if RHS has a length make it fit,
otherwise don't").

* (related to above) Perhaps the "rigidity"(?) of both RHS and LHS should
influence unpacking? If they are both fixed-width, expect exact match. If
either is variable-width, then lazily unravel until a different problem
happens (eg. LHS is fixed-width but RHS ran out of values... basically we
always unravel lazily, but keep track of when LHS or RHS become variable,
and avoid checking length if they do).

* Python 4 change as the language moves towards lazy iterators everywhere?
If `__len__` influenced behavior like mentioned above, then a mechanical
fix to code would simply be `LHS = tuple(*RHS)`, similar to keys().

While I like the original proposal, adding basic slice support to iterables
is also a nice idea. Both are independently useful, eg.
`gen.__getitem__(slice())` delegates to islice(). This achieves the goal of
allowing meaningful unpacking of an iterator window, using normal syntax,
without breaking existing code.

The fixed/variable-width idea makes the most sense to me though. This
enables things like:

>>> a, b, c = (1, *range(2, 100), 3)
(1, 2, 3)

Since both sides are not explicitly sized unpacking is not forcibly sized
either.

Expecting LHS/RHS to exactly match 100% of the time is the special case
here today, not the proposed general unpacking rules that will work well
with iterators. This change also has a Lua-like multiple return value feel
to it that appeals to me.

Thanks,

On Nov 27, 2017 10:23 AM, "Paul Moore"  wrote:

> On 27 November 2017 at 16:05, Chris Angelico  wrote:
> > On Tue, Nov 28, 2017 at 2:35 AM, Steven D'Aprano 
> wrote:
> >> In this case, there is a small but real benefit to counting the
> >> assignment targets and being explicit about the number of items to
> >> slice. Consider an extension to this "non-consuming" unpacking that
> >> allowed syntax like this to pass silently:
> >>
> >> a, b = x, y, z
> >>
> >> That ought to be a clear error, right? I would hope you don't think that
> >> Python should let that through. Okay, now we put x, y, z into a list,
> >> then unpack the list:
> >>
> >> L = [x, y, z]
> >> a, b = L
> >>
> >> That ought to still be an error, unless we explicity silence it. One way
> >> to do so is with an explicit slice:
> >>
> >> a, b = L[:2]
> >
> > I absolutely agree with this for the default case. That's why I am
> > ONLY in favour of the explicit options. So, for instance:
> >
> > a, b = x, y, z # error
> > a, b, ... = x, y, z # valid (evaluates and ignores z)
>
> Agreed, only explicit options are even worth considering (because of
> backward compatibility if for no other reason). However, the unpacking
> syntax is already complex, and hard to search for. Making it more
> complex needs a strong justification. And good luck in doing a google
> search for "..." if you saw that code in a project you had to
> maintain. Seriously, has anyone done a proper investigation into how
> much benefit this proposal would provide? It should be reasonably easy
> to do a code search for something like "=.*islice", to find code
> that's a candidate for using the proposed syntax. I suspect there's
> very little code like that.
>
> I'm -1 on this proposal without a much better justification of the
> benefits it will bring.
> Paul
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] LOAD_NAME/LOAD_GLOBAL should be use getattr()

[C Anthony Risinger]

On Thu, Sep 14, 2017 at 8:07 AM, Steven D'Aprano 
wrote:

> On Wed, Sep 13, 2017 at 12:24:31PM +0900, INADA Naoki wrote:
> > I'm worring about performance much.
> >
> > Dict has ma_version from Python 3.6 to be used for future optimization
> > including global caching.
> > Adding more abstraction layer may make it difficult.
>
> Can we make it opt-in, by replacing the module __dict__ when and only if
> needed? Perhaps we could replace it on the fly with a dict subclass that
> defines __missing__? That's virtually the same as __getattr__.
>
> Then modules which haven't replaced their __dict__ would not see any
> slow down at all.
>
> Does any of this make sense, or am I talking nonsense on stilts?
>

This is more or less what I was describing here:

https://mail.python.org/pipermail/python-ideas/2017-September/047034.html

I am also looking at Neil's approach this weekend though.

I would be happy with a __future__ that enacted whatever concessions are
necessary to define a module as if it were a class body, with import
statements maybe being implicitly global. This "new-style" module would
preferably avoid the need to populate `sys.modules` with something that
can't possibly exist yet (since it's being defined!). Maybe we allow module
bodies to contain a `return` or `yield`, making them a simple function or
generator? The presence of either would activate this "new-style" module
loading:

* Modules that call `return` should return the completed module. Importing
yourself indirectly would likely cause recursion or be an error (lazy
importing would really help here!). Could conceptually expand to something
like:

```
global __class__
global __self__

class __class__:
def __new__(... namespace-dunders-and-builtins-passed-as-kwds ...):
# ... module code ...
# ... closures may access __self__ and __class__ ...
return FancyModule(__name__)

__self__ = __class__(__builtins__={...}, __name__='fancy', ...)
sys.modules[__self__.__name__] = __self__
```

* Modules that call `yield` should yield modules. This could allow defining
zero modules, multiple modules, overwriting the same module multiple times.
Module-level code may then yield an initial object so self-referential
imports, in lieu of deferred loading, work better. They might decide to
later upgrade the initial module's __class__ (similar to today) or replace
outright. Could conceptually expand to something like:

```
global __class__
global __self__

def __hidden_TOS(... namespace-dunders-and-builtins-passed-as-kwds ...):
# ... initial module code ...
# ... closures may access __self__ and __class__ ...
module = yield FancyModuleInitialThatMightRaiseIfUsed(__name__)
# ... more module code ...
module.__class__ = FancyModule

for __self__ in __hidden_TOS(__builtins__={...}, __name__='fancy', ...):
__class__ = __self__.__class__
sys.modules[__self__.__name__] = __self__
```

Otherwise I still have a few ideas around using what we've got, possibly in
a backwards compatible way:

```
global __builtins__ = {...}
global __class__
global __self__

# Loader dunders.
__name__ = 'fancy'

# Deferred loading could likely stop this from raising in most cases.
# globals is a deferred import dict using __missing__.
# possibly sys.modules itself does deferred imports using __missing__.
sys.modules[__name__] = RaiseIfTouchedElseReplaceAllRefs(globals())

class __class__:
[global] import current_module # ref in cells replaced with __self__
[global] import other_module

def bound_module_function(...):
pass

[global] def simple_module_function(...):
pass

# ... end module body ...

# Likely still a descriptor.
__dict__ = globals()

__self__ = __class__()
sys.modules[__self__.__name__] = __self__
 ```

Something to think about.

Thanks,

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] lazy import via __future__ or compiler analysis

[C Anthony Risinger]

On Mon, Sep 11, 2017 at 1:09 PM, Neil Schemenauer <
nas-python-id...@arctrix.com> wrote:

> On 2017-09-11, C Anthony Risinger wrote:
> > I'm not sure I follow the `exec(code, module)` part from the other
> thread.
> > `exec` needs a dict to exec code into [..]
> [..]
> > How do you handle lazy loading when a defined function requests a global
> > via LOAD_NAME? Are you suggesting to change function.__globals__ to
> > something not-a-dict, and/or change LOAD_NAME to bypass
> > function.__globals__ and instead do something like:
>
> I propose to make function.__namespace__ be a module (or other
> namespace object).  function.__globals__ would be a property that
> calls vars(function.__namespace__).
>

Oh interesting, I kinda like that.


> Doing it while perserving backwards compatibility will be a
> challenge.  Doing it without losing performance (LOAD_GLOBAL using
> the fact that f_globals is an honest 'dict') is also hard.  It this
> point, I think there is a chance we can do it.  It is a conceptual
> simplification of Python that gives the language more consistency
> and more power.
>

I do agree it makes module access more uniform if both defined functions
and normal code end up effectively calling getattr(...), instead of
directly reaching into __dict__.


> > All this chatter about modifying opcodes, adding future statements, lazy
> > module opt-in mechanisms, special handling of __init__ or __getattr__ or
> > SOME_CONSTANT suggesting modules-are-almost-a-class-but-not-quite feel
> like
> > an awful lot of work to me, adding even more cognitive load to an already
> > massively complex import system. They seem to make modules even less like
> > other objects or types.
>
> I disagree.  It would make for less cognitive load as LOAD_ATTR
> would be very simlar to LOAD_NAME/LOAD_GLOBAL.  It makes modules
> *more* like other objects and types.
>

I'm not sure about this though. Anything that special cases dunder methods
to sort of look like their counter part on types, eg. __init__ or
__getattr__ or __getattribute__ or whatever else, is a hack to me. The only
way I see to remedy this discrepancy is to make modules a real subclass of
ModuleType, giving them full access to the power of the type system:

```
DottedModuleName(ModuleType, bound_methods=False):
# something like this:
# sys.modules[__class__.__name__] =
__class__._proxy_during_import() ???
# ... module code here ...
sys.modules[DottedModuleName.__name__] =
DottedModuleName(DottedModuleName.__name__, DottedModuleName.__doc__)
```

I've done this a few times in the past, and it works even better on python3
(python2 function.__globals__ didn't trigger __missing__ IIRC). I guess all
I'm getting at, is can we find a way to make modules a real type? So dunder
methods are activated? This would make modules phenomenally powerful
instead of just a namespace (or resorting to after the fact __class__
reassignment hacks).


> I'm busy with "real work" this week and so can't follow the
> discussion closely or work on my proof-of-concept prototype.  I hope
> we can come up with an elegant solution and not some special hack
> just to make module properties work.


Agree, and same, but take a look at what I posted prior. I have a ton of
interest around lazy/deferred module loading, have made it work a few times
in a couple ways, and am properly steeping in import lore. I have bandwidth
to work towards a goal that gives modules full access to dunder methods.
I'll also try to properly patch Python in the way I described.

Ultimately I want deferred loading everywhere, even if it means modules
can't do all the other things types can do. I'm less concerned with how we
get there :-)

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] lazy import via __future__ or compiler analysis

[C Anthony Risinger]

On Fri, Sep 8, 2017 at 11:36 AM, Neil Schemenauer <
nas-python-id...@arctrix.com> wrote:

> On 2017-09-09, Chris Angelico wrote:
> > Laziness has to be complete - or, looking the other way, eager
> > importing is infectious. For foo to be lazy, bar also has to be lazy;
>
> Not with the approach I'm proposing.  bar will be loaded in non-lazy
> fashion at the right time, foo can still be lazy.
>

I'll bring the the conversation back here instead of co-opting the PEP 562
thread.

On Sun, Sep 10, 2017 at 2:45 PM, Neil Schemenauer  wrote:
>
> I think the key is to make exec(code, module) work as an alternative
> to exec(code, module.__dict).  That allows module singleton classes
> to define properties and use __getattr__.  The
> LOAD_NAME/STORE_NAME/DELETE_NAME opcodes need to be tweaked to
> handle this.  There will be a slight performance cost.  Modules that
> don't opt-in will not pay.
>

I'm not sure I follow the `exec(code, module)` part from the other thread.
`exec` needs a dict to exec code into, the import protocol expects you to
exec code into a module.__dict__, and even the related type.__prepare__
requires a dict so it can `exec` the class body there. Code wants a dict so
functions created by the code string can bind it to function.__globals__.

How do you handle lazy loading when a defined function requests a global
via LOAD_NAME? Are you suggesting to change function.__globals__ to
something not-a-dict, and/or change LOAD_NAME to bypass
function.__globals__ and instead do something like:

getattr(sys.modules[function.__globals__['__name__']], lazy_identifier) ?

All this chatter about modifying opcodes, adding future statements, lazy
module opt-in mechanisms, special handling of __init__ or __getattr__ or
SOME_CONSTANT suggesting modules-are-almost-a-class-but-not-quite feel like
an awful lot of work to me, adding even more cognitive load to an already
massively complex import system. They seem to make modules even less like
other objects or types. It would be really *really* nice if ModuleType got
closer to being a simple class, instead of farther away. Maybe we start
treating new modules like a subclass of ModuleType instead of all the
half-way or special case solutions... HEAR ME OUT :-) Demo below.

(also appended to end)
https://gist.github.com/anthonyrisinger/b04f40a3611fd7cde10eed6bb68e8824

```
# from os.path import realpath as rpath
# from spam.ham import eggs, sausage as saus
# print(rpath)
# print(rpath('.'))
# print(saus)

$ python deferred_namespace.py

/home/anthony/devel/deferred_namespace
Traceback (most recent call last):
  File "deferred_namespace.py", line 73, in 
class ModuleType(metaclass=MetaModuleType):
  File "deferred_namespace.py", line 88, in ModuleType
print(saus)
  File "deferred_namespace.py", line 48, in __missing__
resolved = deferred.__import__()
  File "deferred_namespace.py", line 9, in __import__
module = __import__(*self.args)
ModuleNotFoundError: No module named 'spam'
```

Lazy-loading can be achieved by giving modules a __dict__ namespace that is
import-aware. This parallels heavily with classes using __prepare__ to make
their namespace order-aware (ignore the fact they are now order-aware by
default). What if we brought the two closer together?

I feel like the python object data model already has all the tools we need.
The above uses __prepare__ and a module metaclass, but it could also use a
custom __dict__ descriptor for ModuleType that returns an import-aware
namespace (like DeferredImportNamespace in my gist). Or ModuleType.__new__
can reassign its own __dict__ (currently read-only). In all these cases we
only need to make 2 small changes to Python:

* Change `__import__` to call `globals.__defer__` (or similar) when
appropriate instead of importing.
* Create a way to make a non-binding class type so
`module.function.__get__` doesn't create a bound method.

The metaclass path also opens the door for passing keyword arguments to
__prepare__ and __new__:

from spam.ham import eggs using methods: True

... which might mean:

GeneratedModuleClassName(ModuleType, methods=True):
# module code ...
# methods=True passed to __prepare__ and __new__,
# allowing the module to implement bound methods!

... or even:

import . import CustomMetaModule
from spam.ham import (
eggs,
sausage as saus,
) via CustomMetaModule using {
methods: True,
other: feature,
}

... which might mean:

GeneratedModuleClassName(ModuleType, metaclass=CustomMetaModule,
methods=True, other=feature):
# module code ...

Making modules work like a real type/class means we we get __init__,
__getattr__, and every other __*__ method *for free*, especially when
combined with an extension to the import protocol allowing methods=True (or
similar, like above). We could even subclass the namespace for each module,
allowing us to effectively revert the module's __dict__ to a normal dict,
and completely remove any possible overhead.

Re: [Python-ideas] PEP 562

[C Anthony Risinger]

I'd really love to find a way to enable lazy loading by default, maybe with
a way to opt-out old/problem/legacy modules instead of opt-in via
__future__ or anything else. IME easily 95%+ of modules in the wild, today,
will not even notice (I wrote an application bundler in past that enabled
it globally by default without much fuss for several years). The only small
annoyance is when it does cause problems the error can jump around,
depending on how the lazy import was triggered.

module.__getattr__ works pretty well for normal access, after being
imported by another module, but it doesn't properly trigger loading by
functions defined in the module's own namespace. These functions are bound
to module.__dict__ as their __globals__ so lazy loading of this variety is
really dependant on a custom module.__dict__ that implements __getitem__ or
__missing__.

I think this approach is the ideal path over existing PEPs. I've done it in
the past and it worked very well. The impl looked something like this:

* Import statements and __import__ generate lazy-load marker objects
instead of performing imports.
* Marker objects record the desired import and what identifiers were
supposed to be added to namespace.
* module.__dict__.__setitem__ recognizes markers and records their
identifiers as lazily imported somewhere, but **does not add them to
namespace**.
* module.___getattribute__ will request the lazy attribute via
module.__dict__ like regular objects and functions will request via their
bound __globals__.
* Both will trigger module.__dict.__missing__, which looks to see if the
requested identifier was previously marked as a lazy import, and if so,
performs the import, saves to namespace properly, and returns the real
import.

-- 


C Anthony


On Sep 10, 2017 1:49 PM, "Ivan Levkivskyi"  wrote:

I have written a short PEP as a complement/alternative to PEP 549.
I will be grateful for comments and suggestions. The PEP should
appear online soon.

--
Ivan

***

PEP: 562
Title: Module __getattr__
Author: Ivan Levkivskyi 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Sep-2017
Python-Version: 3.7
Post-History: 09-Sep-2017


Abstract


It is proposed to support ``__getattr__`` function defined on modules to
provide basic customization of module attribute access.


Rationale
=

It is sometimes convenient to customize or otherwise have control over
access to module attributes. A typical example is managing deprecation
warnings. Typical workarounds are assigning ``__class__`` of a module object
to a custom subclass of ``types.ModuleType`` or substituting ``sys.modules``
item with a custom wrapper instance. It would be convenient to simplify this
procedure by recognizing ``__getattr__`` defined directly in a module that
would act like a normal ``__getattr__`` method, except that it will be
defined
on module *instances*. For example::

  # lib.py

  from warnings import warn

  deprecated_names = ["old_function", ...]

  def _deprecated_old_function(arg, other):
  ...

  def __getattr__(name):
  if name in deprecated_names:
  warn(f"{name} is deprecated", DeprecationWarning)
  return globals()[f"_deprecated_{name}"]
  raise AttributeError(f"module {__name__} has no attribute {name}")

  # main.py

  from lib import old_function  # Works, but emits the warning

There is a related proposal PEP 549 that proposes to support instance
properties for a similar functionality. The difference is this PEP proposes
a faster and simpler mechanism, but provides more basic customization.
An additional motivation for this proposal is that PEP 484 already defines
the use of module ``__getattr__`` for this purpose in Python stub files,
see [1]_.


Specification
=

The ``__getattr__`` function at the module level should accept one argument
which is a name of an attribute and return the computed value or raise
an ``AttributeError``::

  def __getattr__(name: str) -> Any: ...

This function will be called only if ``name`` is not found in the module
through the normal attribute lookup.

The reference implementation for this PEP can be found in [2]_.


Backwards compatibility and impact on performance
=

This PEP may break code that uses module level (global) name
``__getattr__``.
The performance implications of this PEP are minimal, since ``__getattr__``
is called only for missing attributes.


References
==

.. [1] PEP 484 section about ``__getattr__`` in stub files
   (https://www.python.org/dev/peps/pep-0484/#stub-files)

.. [2] The reference implementation
   (https://github.com/ilevkivskyi/cpython/pull/3/files)


Copyright
=

This document has been placed in the public domain.

___
Python-ideas mailing list
Python-ideas@python.org

Re: [Python-ideas] namedtuple literals [Was: RE a new namedtuple]

[C Anthony Risinger]

On Sun, Jul 23, 2017 at 8:54 PM, Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> C Anthony Risinger writes:
>
>  > A tuple is a tuple is a tuple. No types. Just convenient accessors.
>
> That's not possible, though.  A *tuple* is an immutable collection
> indexed by the natural numbers, which is useful to define as a single
> type precisely because the natural numbers are the canonical
> abstraction of "sequence".  You can use the venerable idiom
>
> X = 0
> Y = 1
>
> point = (1.0, 1.0)
> x = point[X]
>
> to give the tuple "named attributes", restricting the names to Python
> identifiers.  Of course this lacks the "namespace" aspect of
> namedtuple, where ".x" has the interpretation of "[0]" only in the
> context of a namedtuple with an ".x" attribute.  But this is truly an
> untyped tuple-with-named-attributes.
>
> However, once you attach specific names to particular indexes, you
> have a type.  The same attribute identifiers may be reused to
> correspond to different indexes to represent a different "convenience
> type".  Since we need to be able to pass these objects to functions,
> pickle them, etc, that information has to be kept in the object
> somewhere, either directly (losing the space efficiency of tuples) or
> indirectly in a class-like structure.
>
> I see the convenience of the unnamed-type-typed tuple, but as that
> phrase suggests, I think it's fundamentally incoherent, a high price
> to pay for a small amount of convenience.
>
> Note that this is not an objection to a forgetful syntax that creates
> a namedtuple subtype but doesn't bother to record the type name
> explicitly in the program.  In fact, we already have that:
>
> >>> from collections import namedtuple
> >>> a = namedtuple('_', ['x', 'y'])(0,1)
> >>> b = namedtuple('_', ['x', 'y'])(0,1)
> >>> a == b
> True
> >>> c = namedtuple('_', ['a', 'b'])(0,1)
>
> This even gives you free equality as I suppose you want it:
>
> >>> a == c
> True
> >>> a.x == c.a
> True
> >>> a.a == c.x
> Traceback (most recent call last):
>   File "", line 1, in 
> AttributeError: '_' object has no attribute 'a'
> >>> c.x == a.a
> Traceback (most recent call last):
>   File "", line 1, in 
> AttributeError: '_' object has no attribute 'x'
>
> Bizarre errors are the inevitable price to pay for this kind of abuse,
> of course.
>
> I'm not a fan of syntaxes like "(x=0, y=1)" or "(x:0, y:1)", but I'll
> leave it up to others to decide how to abbreviate the abominably ugly
> notation I used.
>

Sure sure, this all makes sense, and I agree you can't get the accessors
without storing information, somewhere, that links indexes to attributes,
and it makes complete sense it might be implemented as a subtype, just like
namedtuple works today.

I was more commenting on what it conceptually means to have the designation
"literal". It seems surprising to me that one literal has a different type
from another literal with the same construction syntax. If underneath the
hood it's technically a different type stored in some cached and hidden
lookup table, so be it, but on the surface I think most just want a basic
container with simpler named indexes.

Every time I've used namedtuples, I've thought it more of a chore to pick a
name for it, because it's only semi-useful to me in reprs, and I simply
don't care about the type, ever. I only care about the shape for comparison
with other tuples. If I want typed comparisons I can always just use a
class. I'd also be perfectly fine with storing the "type" as a field on the
tuple itself, because it's just a value container, and that's all I'll ever
want from it.

Alas, when I think I want a namedtuple, I usually end up using a dict
subclass that assigns `self.__dict__ = self` within __new__, because this
makes attribute access (and assignment) work automagically, and I care
about that more than order (though it can be made to support both).

At the end of the day, I don't see a way to have both a literal and
something that is externally "named", because the only ways to pass the
name I can imagine would make it look like a value within the container
itself (such as using a literal string for the first item), unless even
more new syntax was added.

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] namedtuple literals [Was: RE a new namedtuple]

[C Anthony Risinger]

On Jul 23, 2017 1:56 PM, "MRAB"  wrote:

On 2017-07-23 17:08, Todd wrote:


> On Jul 20, 2017 1:13 AM, "David Mertz" > wrote:
>
> I'm concerned in the proposal about losing access to type
> information (i.e. name) in this proposal.  For example, I might
> write some code like this now:
>
>  >>> from collections import namedtuple
>  >>> Car = namedtuple("Car", "cost hp weight")
>  >>> Motorcycle = namedtuple("Motorcycle", "cost hp weight")
>  >>> smart = Car(18_900, 89, 949)
>  >>> harley = Motorcyle(18_900, 89, 949)
>  >>> if smart==harley and type(smart)==type(harley):
> ... print("These are identical vehicles")
>
> The proposal to define this as:
>
>  >>> smart = (cost=18_900, hp=89, weight=949)
>  >>> harley = (cost=18_900, hp=89, weight=949)
>
> Doesn't seem to leave any way to distinguish the objects of
> different types that happen to have the same fields.  Comparing
> `smart._fields==harley._fields` doesn't help here, nor does any type
> constructed solely from the fields.
>
>
> What about making a syntax to declare a type? The ones that come to mind
> are
>
>  name = (x=, y=)
>
> Or
>
>  name = (x=pass, y=pass)
>
> They may not be clear enough, though.
>
> Guido has already declared that he doesn't like those bare forms, so it'll
probably be something like ntuple(...).


Not exactly a literal in that case.

If this is true, why not simply add keyword arguments to tuple(...)?

Something like (a=1, b=2, c=3) makes very clear sense to me, or even (1, 2,
c=3), where the first two are accessible by index only. Or even (1, 2, c:
3), which reminds me of Elixir's expansion of tuple and list keywords.

A tuple is a tuple is a tuple. No types. Just convenient accessors.

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Dictionary destructing and unpacking.

[C Anthony Risinger]

On Jun 8, 2017 1:35 AM, "Greg Ewing" <greg.ew...@canterbury.ac.nz> wrote:

C Anthony Risinger wrote:

> Incredibly useful and intuitive, and for me again, way more generally
> applicable than iterable unpacking. Maps are ubiquitous.
>

Maps with a known, fixed set of keys are relatively uncommon
in Python, though. Such an object is more likely to be an
object with named attributes.


I would generally agree, but in the 3 languages I mentioned at least, map
destructuring does not require the LHS to exactly match the RHS (a complete
match is required for lists and tuples though).

Because pattern matching is central to the language, Elixir takes it even
further, providing syntax that allows you to choose whether a variable on
the LHS is treated as a match (similar to the None constant in my example)
or normal variable binding.

In all cases though, the LHS need only include the attributes you are
actually interested in matching and/or binding. I need to review the linked
thread still, but the way ECMAScript does it:

const {one, two} = {one: 1, two: 2};

I think could also be useful in Python, especially if we defined some
default handling of objects and dicts via __getattr__ and/or __getitem__,
because people could use this to destructure `self` (I've seen this
requested a couple times too):

self.one = 1
self.two = 2
self.three = 3
{one, two} = self

Or maybe even:

def fun({one, two}, ...):

Which is also supported (and common) in those 3 langs, but probably less
pretty to Python eyes (and I think a bit less useful without function
clauses).

-- 

C Anthony [mobile]
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Dictionary destructing and unpacking.

[C Anthony Risinger]

On Jun 7, 2017 5:54 PM, "Erik" <pyt...@lucidity.plus.com> wrote:

On 07/06/17 23:42, C Anthony Risinger wrote:

> Neither of these are really comparable to destructuring.
>

No, but they are comparable to the OP's suggested new built-in method
(without requiring each mapping type - not just dicts - to implement it).
That was what _I_ was responding to.


No worries, I only meant to emphasize that destructuring is much much more
powerful and less verbose/duplicative than anything based on functions. It
could readily apply/fallback against any object's __dict__ because maps
underpin the entire Python object system.

-- 

C Anthony [mobile]
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Dictionary destructing and unpacking.

[C Anthony Risinger]

On Jun 7, 2017 5:42 PM, "C Anthony Risinger" <c...@anthonyrisinger.com> wrote:

On Jun 7, 2017 5:15 PM, "Matt Gilson" <m...@getpattern.com> wrote:



On Wed, Jun 7, 2017 at 3:11 PM, Erik <pyt...@lucidity.plus.com> wrote:

> On 07/06/17 19:14, Nick Humrich wrote:
>
>> a, b, c = mydict.unpack('a', 'b', 'c')
>>
>
> def retrieve(mapping, *keys):
>return (mapping[key] for key in keys)
>
>
>
Or even:

from operator import itemgetter

retrieve = itemgetter('a', 'b', 'c')

a, b, c = retrieve(dictionary)


Neither of these are really comparable to destructuring. If you take a look
at how Erlang and Elixir do it, and any related code, you'll find it used
constantly, all over the place. Recent ECMAScript is very similar, allowing
both destructuring into vars matching the key names, or arbitrary var
names. They both allow destructuring in the function header (IIRC python
can do this with at least tuples). Erlang/Elixir goes beyond this by using
the pattern matching to select the appropriate function clause within a
function definition, but that's less relevant to Python.

This feature has been requested before. It's easily one of the most, if not
the top, feature I personally wish Python had. Incredibly useful and
intuitive, and for me again, way more generally applicable than iterable
unpacking. Maps are ubiquitous.


Also in the Erlang/Elixir (not sure about ECMAScript) the destructuring is
about both matching *and* assignment. So something like this (in Python):

payload = {"id": 123, "data": {...}}
{"id": None, "data": data} = payload

Would raise a MatchError or similar. It's a nice way to assert some values
and bind others in one shot. Those languages often use atoms for keys
though, which typically don't require quoting (and ECMAScript is more lax),
so that extended format is less useful and pretty if the Python variant
expected quotes all over the place.

-- 

C Anthony [mobile]
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Relative import of self or parent package?

[C Anthony Risinger]

I occasionally want to do something like this:

import .. as parent

or:

import . as self

The pattern is a little less useful for namespace packages (which I've been
trying to use almost exclusively, along with relative imports) than it is
for __init__.py based packages, but there is still some utility.

Is there a reason one can't import dots directly and bind them to a name? I
did not find any text around this and the closest I've come up with is:

from .. import __name__ as parent_module_name
parent = __import__(parent_module_name)

Could we support relative imports without `from` so long as there is an
`as`?

Thanks,

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Passive tracing of function objects (was: Efficient debug logging)

[C Anthony Risinger]

On Wed, Feb 15, 2017 at 10:45 AM, C Anthony Risinger <anth...@xtfx.me>
wrote:

> On Tue, Feb 14, 2017 at 3:55 PM, Barry <ba...@barrys-emacs.org> wrote:
>>
>>
>> The point is that the cost of creating the msg argument can be very high.
>>
>> At the point that logging decides to skip output it is to late to save
>> the cost of creating the arg tuple.
>>
>
> Awhile back I was playing with some code for the purpose of adding debug
> logs "after the fact":
>
> https://github.com/anthonyrisinger/retrospect/blob/master/README.rst
>
> I'm not trying to plug it (It's POC, broken, and Python 2) but maybe there
> is some value because I was trying to solve a similar problem of fast and
> easy debugging and logs.
>

I meant to note that I did in fact use this (plus a wrapper that would take
an ENV var with the function to target) a few times to inspect code running
deep in Django without ever editing any Django code. It also works for
inspecting code you can't edit, say modules installed globally into /usr.

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Passive tracing of function objects (was: Efficient debug logging)

[C Anthony Risinger]

On Tue, Feb 14, 2017 at 3:55 PM, Barry  wrote:
>
>
> The point is that the cost of creating the msg argument can be very high.
>
> At the point that logging decides to skip output it is to late to save the
> cost of creating the arg tuple.
>

Awhile back I was playing with some code for the purpose of adding debug
logs "after the fact":

https://github.com/anthonyrisinger/retrospect/blob/master/README.rst

I'm not trying to plug it (It's POC, broken, and Python 2) but maybe there
is some value because I was trying to solve a similar problem of fast and
easy debugging and logs.

The idea here is we target a function for "retrospection" by replacing it's
code object with a modified variant that implements call outs at different
points. Such interesting points might be "on symbol table changes" (new
vars) or "on identifier rebinding" (updating vars) or "on line changes" or
even "every single tick" to inspect TOS.

I like this approach of latching onto the function/code object directly
because it does not require a wrapper, but it can have be tricky with
decorators depending on what the decorator does with the function it
receives (we need access to the function object to replace it's __code__).

I don't much like writing logs. They are verbose in code and almost always
either much more or less than I need to deal with the problem at hand, and
it's not always clear what's important ahead-of-time. With all the
implementation details in play, you might end up wrapping code in context
managers and whatnot for the sole purpose of logging. Or you have to jump
thru hoops to get the exact info you need using many log levels or flags. I
think having a way to target functions for real time introspection, then
perfectly disable it, could allow for interesting solution to large,
cross-cutting concerns like application logging.

Is there any interest in this? Say a new function method like
fun.trace(vars=[a,b,c])? Is there already a solution or past discussion on
this topic?

Thanks,

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Using Python for end user applications

[C Anthony Risinger]

On Sat, Jan 28, 2017 at 5:26 AM, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 28 January 2017 at 02:11, C Anthony Risinger <anth...@xtfx.me> wrote:
> > I can't articulate it we'll, or even fully isolate the reasons for it.
> All I
> > really know is how I feel when peers ask me about Python or the reading I
> > get when others speak about their experience using it. Python is
> absolutely
> > one of my favorite languages to write, yet I find myself recommending
> > against it, and watching others do the same. Python comes with caveats
> and
> > detailed explanations out the gate and people simply perceive higher
> > barriers and more chores.
>
> Picking up on this and the comment you made in the original post
>
> > With a still difficult distribution/compatibility story, I've watched
> dozens of instances
> > where people choose something else, usually Node or Golang.
>
> Can you explain why you recommend against Python, in a bit more
> detail? If you are an enthusiastic Python user, but you are steering
> people away from Python, then it would be worth understanding why.
>
> As you mention end user applications and distribution, one of my first
> questions would be what platform you work on. Following on from that,
> what sort of end user applications are you looking at? If we're
> talking here about games for iOS, then that's a much different
> situation than GUI apps for Windows or command line tools for Linux.
>

I'm working on Linux myself, with my typical audience being other
developers or occasionally leadership. Builds usually include Windows, but
always Linux and OSX. I'd like a one-click solution to:

How do I redistribute and successfully install Python, dependencies, and an
application with the least possible steps for the end user? For any
platform or persona?

I prefer dynamic applications redistribute their runtime, and not depend on
the system in any significant way aside from major libraries or support
files. It's more obnoxious sometimes but I believe it lowers the number of
unexpected incidents. I also don't want to setup support infrastructure
(pypi proxy cache or self hosted pypi) myself or require coordination with
ops teams. Simple authenticated download for publishing is ideal, and
direct execution of that download is even better.

Containers are have greatly impacted how people think about distribution.
There is a trend to use fast/small containers as CLI tools. I think there
is still benefit to empowering a way to "remix python" with your local
environment and produce a redistributable binary for downstream users. I
have far less experience with Go but I like how easy it is to generate such
executables.

My personal feeling is that Python happily works in the "Command line
> tools for Linux" area (except possibly with regard to C extensions
> where the plethora of Linux ABIs makes things hard). But other areas
> less so. I've been having good experiences making standalone
> applications with the new Windows "embedded" distributions, but that
> is relatively new, and still has a lot of rough edges. I'm working on
> a project to bundle a working zipapp with the embedded distribution to
> make a standalone exe - would having something like that make any
> difference in your environment?
>

Yes I believe it would. I implemented something to this end for Linux
(defunct, no Python 3, uses abandoned PEP 425):

https://github.com/anthonyrisinger/zippy

about 4-5 years ago for this purpose, before similar technologies such as
PEX and zipapp existed. It worked well for the time it was used.

The project is dead, but it had a few ideas I think might be worth further
exploration. In a nutshell, "zippy" was a build tool (waf and distlib
because no interface to pip) that always output a single python binary
built from source. All python code (stdlib, deps, app) was appended in a
zipfile. All C-extensions (including dotted) were observed during .so
generation and re-linked statically into the binary as builtins instead
(more on this below).

The end result was a self-contained executable that would only recognize
it's own stdlib and could properly run embedded C-modules. The
implementation also changed entrypoints based on argv[0], so this could
thing could perform as a multicall binary like BusyBox (the symlink name
was resolved to an entrypoint). It was easy to send this to other
developers where they could unpack it for development or use it as-is. A
future goal was to allow respinning an unpacked environment into a new
redistributable.

To make this work, I changed python in a few specific ways:

Set LANDMARK file at build time to something other than `os.py`. Python
uses this file to recognize the stdlib during early boot. Customizing it
[zippy-SOMEHASH.json] ensures a different unpacked build's stdlib 

Re: [Python-ideas] A more readable way to nest functions

[C Anthony Risinger]

On Fri, Jan 27, 2017 at 3:28 PM, Ethan Furman  wrote:

> On 01/27/2017 01:07 PM, Brent Brinkley wrote:
>
>> Suggested structure:
>>
>>   print() <| some_func() <| another_func("Hello")
>>
>
> My first question is what does this look like when print() and some_func()
> have other parameters?  In other words, what would this look like?
>
> print('hello', name, some_func('whatsit', another_func('good-bye')),
> sep=' .-. ')


The Elixir pipe operator looks pretty close to the suggested style, but the
argument order is reversed:

another_func('good-bye') |> some_func('whatsit') |> print('hello', name,
sep=' .-. ')

This isn't exactly equivalent to the example though because the result of
each call is passed as the first argument to the next function. I think it
looks nice when it's the right fit, but it's limited to the first argument.

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Is it Python 3 yet?

[C Anthony Risinger]

On Jan 27, 2017 4:51 PM, "Random832" <random...@fastmail.com> wrote:

On Fri, Jan 27, 2017, at 12:54, C Anthony Risinger wrote:
> I know the scientific community is a big and important part of the
> Python ecosystem, but I honestly believe other parts of Python are
> suffering from any dragging of feet at this point. Python 3 has been
> out nearly a decade, and I think it would be super for the community
> to take a bold stance (is it still bold 9 years later?) and really
> stand behind Python 3, prominently, almost actively working to
> diminish Python 2.

This particular subthread is regarding whether to make a 64-bit version
of python 2 and/or 3 (whatever is done regarding the other question) the
default download button for users coming from Win64 browsers. At least,
the bits you're responding to are talking about 32-bit libraries rather
than Python 2.


Yeah, I guess I was trying to push against any further stagnation, of any
kind, on forward-facing questions like 32/64 bit and 2/3 version. I
hesitated to say anything because I don't feel I'm adding much concrete or
even useful information to the conversation, but it's something that's been
building internally for a long time while observing the overarching tone
and outcomes of Python threads.

I can't articulate it we'll, or even fully isolate the reasons for it. All
I really know is how I feel when peers ask me about Python or the reading I
get when others speak about their experience using it. Python is absolutely
one of my favorite languages to write, yet I find myself recommending
against it, and watching others do the same. Python comes with caveats and
detailed explanations out the gate and people simply perceive higher
barriers and more chores.

I don't have any truly constructive input so I'll stop here; I only wanted
to voice that in my tiny tiny bubble, I'm watching market share diminish,
it's unfortunate, and I'm not sure what to do about it.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Is it Python 3 yet?

[C Anthony Risinger]

So I realize this is subjective and just a personal experience, but over
the last 3-5 years I've really watched Python usage and popularity decline
in the "hearts and minds" of my peers, across a few different companies I
work with. At my current gig we don't even use Python anymore for tools
that will be distributed to an end user; we only use Python for internal
tooling.

With a still difficult distribution/compatibility story, I've watched
dozens of instances where people choose something else, usually Node or
Golang. The primary uses here are api and microservice-type applications,
developer tooling, and CLI apps. Even recent additions like `async` keyword
are causing more problems because it's not a useful general-purpose
concurrency primitive eg. like a goroutine or greenlets.

I know the scientific community is a big and important part of the Python
ecosystem, but I honestly believe other parts of Python are suffering from
any dragging of feet at this point. Python 3 has been out nearly a decade,
and I think it would be super for the community to take a bold stance (is
it still bold 9 years later?) and really stand behind Python 3,
prominently, almost actively working to diminish Python 2.

I've been hearing and reading about both for a long time, and honestly I'd
love one of them to go away! I don't even care which :-)

On Fri, Jan 27, 2017 at 4:25 AM, Terry Reedy  wrote:

> On 1/27/2017 4:38 AM, Nathaniel Smith wrote:
>
>> On Fri, Jan 27, 2017 at 1:32 AM, Stephan Houben <
>> stephanh42-re5jqeeqqe8avxtiumw...@public.gmane.org> wrote:
>>
>>> Hi all,
>>>
>>> FWIW, I got the following statement from here:
>>>
>>> https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows
>>>
>>> "Standard numpy and scipy binary releases on Windows use pre-compiled
>>> ATLAS
>>> libraries and are 32-bit only because of the difficulty of compiling
>>> ATLAS
>>> on 64-bit Windows. "
>>>
>>> Might want to double-check with the numpy folks; it would
>>> be too bad if numpy wouldn't work on the preferred Windows Python.
>>>
>>
>> That's out of date
>>
>
> Would be nice if it were updated...
>
>  -- official numpy releases have switched from ATLAS
>
>> to OpenBLAS (which requires some horrible frankencompiler system, but
>> it seems to work for now...), and there are 32- and 64-bit Windows
>> wheels up on PyPI: https://pypi.python.org/pypi/numpy/
>>
>
> and from
>
> NumPy, a fundamental package needed for scientific computing with Python.
> Numpy+MKL is linked to the Intel® Math Kernel Library and includes
> required DLLs in the numpy.core directory.
>
> numpy‑1.11.3+mkl‑cp27‑cp27m‑win32.whl
> numpy‑1.11.3+mkl‑cp27‑cp27m‑win_amd64.whl
> etc.
>
> All the several packages that require numpy also come in both versions.
>
> 64-bit is definitely what I'd recommend as a default to someone
>> wanting to use numpy, because when working with arrays it's too easy
>> to hit the 32-bit address space limit.
>>
>> -n
>>
>>
>
> --
> Terry Jan Reedy
>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Order of loops in list comprehension

[C Anthony Risinger]

On Oct 22, 2016 2:51 AM, "Alexander Heger"  wrote:
>>>
>>>
 For me the current behaviour does not seem unreasonable as it
resembles the order in which you write out loops outside a comprehension
>>>
>>>
>>> That's true, but the main reason for having comprehensions
>>> syntax in the first place is so that it can be read
>>> declaratively -- as a description of the list you want,
>>> rather than a step-by-step sequence of instructions for
>>> building it up.
>>>
>>> If you have to stop and mentally transform it into nested
>>> for-statements, that very purpose is undermined.
>>
>> Exactly.
>
>
> Well, an argument that was often brought up on this forum is that Python
should do things consistently, and not in one way in one place and in
another way in another place, for the same thing.  Here it is about the
order of loop execution.  The current behaviour in comprehension is that is
ts being done the same way as in nested for loops.  Which is easy enough to
remember.  Same way, everywhere.

A strict interpretation by this logic would also require the [x ...] part
to be at the end, like [... x] since that's how it would look in a nested
for loop (inside deepest loop).

I personally agree with what many others have said, in that comprehension
order is not intuitive as is. I still page fault about it after many years
of using.

Is there a way to move the expression bit to the end in a backcompat way?
It might be a completely different syntax though (I think both colons and
semicolons were suggested).

FWIW, Erlang/Elixir (sorry after 6 years python this is what I do now!)
does it the same way as python:

>>> [{X, Y} || X <- [1,2,3], Y <- [a,b]].
[{1,a},{1,b},{2,a},{2,b},{3,a},{3,b}]

Here X is the outer loop.

I think the confusion stems from doing it both ways at the same time. We
retain the for loop order but then hoist the expression to the top. Ideally
we'd either not do that, or reverse the for loop order.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[C Anthony Risinger]

On Oct 3, 2016 7:09 PM, "Stephen J. Turnbull" <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:
>
> Rene Nejsum writes:
>
>  > I believe that you should be able to code concurrent code, without
>  > being to explicit about it, but let the runtime handle low-level
>  > timing, as long as you know your code will execute in the intended
>  > order.
>
> Isn't "concurrent code whose order of execution you know" an oxymoron?

They are referring to the synchronous nature of any independent control
state. Whether it's a thread, a coroutine, a continuation, or whatever else
doesn't really matter much. When a thing runs concurrently along side other
things, it's still synchronous with respect to itself regardless of how
many context switches occur before completion. Such things only need
mechanisms to synchronize in order to cooperate.

People want to know how they are suppose to write unified,
non-insane-and-ugly code in this a/sync python 2/3 world we now find
ourselves in. I've been eagerly watching this thread for the answer, thus
far to no avail.

Sans-io suggests we write bite-sized synchronous code that can be driven by
a/sync consumers. While this is all well and good, how does one write said
consuming library for both I/O styles without duplication?

The answer seems to be "write everything you ever wanted as async and throw
some sync wrappers around it". Which means all the actual code I write will
be peppered with async and await keywords.

In Go I can spawn a new control state (goroutine) at any time against any
function. This is clear in the code. In Erlang I can spawn a new control
state (Erlang process) at any time and it's also clear. Erlang is a little
different because it will preempt me, but the point is I am simply choosing
a target function to run in a new context. Gevent and even threading module
is another example of this pattern.

In all reality you don't typically need many suspension points other than
around I/O, and occasionally heavy CPU, so I think folks are struggling to
understand (I admit, myself included) why the runtime doesn't want to be
more help and instead punts back to the developer.

-- 

C Anthony
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] if-statement in for-loop

[C Anthony Risinger]

On Sep 11, 2016 7:11 AM, "Chris Angelico"  wrote:
>
> That said, though, filtered iteration isn't common enough to demand
> its own syntax IMO. I do it fairly often,

I do it often enough to want this. When I first started writing Python this
struck me as an inconsistency... if it's useful in comprehensions, why not
regular loops? I realize comprehensions are all about construction of the
list itself, but the parallel still exists.

Also feels similar to guard statements. I'd love to see Python gain more
pattern matching and destructuring features because they are wonderful in
Erlang/Elixir.

> but it's usually fine to
> just have a condition on a separate line. (I do use ": continue"
> rather than making it two lines.)

FWIW, code I write or review would mandate this be two lines followed by a
blank line, so 3 total. I require any abrupt change or termination in the
current flow of control to be followed by a blank line so the reader
clearly sees the possible jump (continue, break, and return especially).
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/