Re: [Python-ideas] Add closing and iteration to threading.Queue

[Guido van Rossum]

On Sun, Oct 21, 2018 at 6:08 PM Nathaniel Smith  wrote:
I'm not sure if this is an issue the way Queue is used in practice, but in
general you have to be careful with this kind of circular flow because if
your queue communicates backpressure (which it should) then circular flows
can deadlock.

Nathaniel, would you be able to elaborate more on the issue of
backpressure? I think a lot of people here are not really familiar with the
concepts and its importance, and it changes how you have to think about
queues and the like.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add closing and iteration to threading.Queue

[Cameron Simpson]

On 21Oct2018 18:06, Nathaniel Smith  wrote:

On Sun, Oct 21, 2018, 16:48 MRAB  wrote:

On 2018-10-21 22:30, Antoine Pitrou wrote:
> Ah.  This is the one statement that makes me favorable to this 
> idea.

> When there is a single consumer, it's easy enough to send a sentinel.
> But when there are multiple consumers, suddenly you must send exactly
> the right number of sentinels (which means you also have to careful
> keep track of their number, which isn't always easy).  There's some
> delicate code doing exactly that in concurrent.futures.
>
You don't need more than one sentinel. When a consumer sees the
sentinel, it just needs to put it back for the other consumers.


Yes, this is exactly what my own IterableQUeue does.


I'm not sure if this is an issue the way Queue is used in practice, but in
general you have to be careful with this kind of circular flow because if
your queue communicates backpressure (which it should) then circular flows
can deadlock.


Haven't come across this myself. A closeable queue doesn't seem circular 
to me. The handling of the sentinel is internal to the IterableQueue, so 
external users never see it.


Cheers,
Cameron Simpson 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add closing and iteration to threading.Queue

[Cameron Simpson]

On 21Oct2018 21:19, Vladimir Filipović  wrote:

On Sun, Oct 21, 2018 at 8:45 PM MRAB  wrote:

FTR, this has been discussed before:
[Python-ideas] `__iter__` for queues?
https://mail.python.org/pipermail/python-ideas/2010-January/006711.html


Thank you!


Hmm, yes. My post there is this one:

 https://mail.python.org/pipermail/python-ideas/2010-January/006716.html

I want to point out that in my code the single consumer of a Queue is 
incredibly common, so common that I can't off the top of my head think 
of _any_ uses of Queue directly: I _always_ make an IterableQueue and 
simply have the consumer iterate over the iterable queue.


This is _exactly_ like Vladimir's proposal to my mind: my IterableQueue 
is iterable, and has a .close method just like his (prevent further puts 
and indicates end of stream) mediated with a sentinel internally.  
Arbitrary number of putters, _usually_ only one getter but of course 
there's no need for that.


So to my mind his proposal is very simple and sensible, and matches 
almost universally my own use of Queues.


Cheers,
Cameron Simpson 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add closing and iteration to threading.Queue

[Nathaniel Smith]

On Sun, Oct 21, 2018, 16:48 MRAB  wrote:

> On 2018-10-21 22:30, Antoine Pitrou wrote:
> > On Sun, 21 Oct 2018 19:58:05 +0200
> > Vladimir Filipović 
> > wrote:
> >>
> >> To anticipate a couple more possible questions:
> >>
> >> - What would this proposal do about multiple producers/consumers
> >> needing to jointly decide _when_ to close the queue?
> >>
> >> Explicitly nothing.
> >>
> >> The queue's state is either closed or not, and it doesn't care who
> >> closed it. It needs to interact correctly with multiple consumers and
> >> multiple producers, but once any one piece of code closes it, the
> >> correct interaction is acting like a closed queue for everybody.
> >
> > Ah.  This is the one statement that makes me favorable to this idea.
> > When there is a single consumer, it's easy enough to send a sentinel.
> > But when there are multiple consumers, suddenly you must send exactly
> > the right number of sentinels (which means you also have to careful
> > keep track of their number, which isn't always easy).  There's some
> > delicate code doing exactly that in concurrent.futures.
> >
> You don't need more than one sentinel. When a consumer sees the
> sentinel, it just needs to put it back for the other consumers.
>

I'm not sure if this is an issue the way Queue is used in practice, but in
general you have to be careful with this kind of circular flow because if
your queue communicates backpressure (which it should) then circular flows
can deadlock.

-n

>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Python Enhancement Proposal for List methods

[David Mertz]

I have changed my opinion to -1 on list.replace(). While max=n is a useful
argument, a plain function could work equally well on every sequence and
not be specific to lists.

On Sun, Oct 21, 2018, 10:18 AM David Mertz  The list comprehensions are not very hard, and are more general. EXCEPT
> with the limited number of occurrences. We have this for str.replace(...,
> max=n), and it is useful fairly often.
>
> I'm +0.5 on .replace() with that capability. But -1 on .removeall() that
> adds nothing to an easy listcomp.
>
> On Sun, Oct 21, 2018, 9:01 AM Siva Sukumar Reddy 
>> Hey everyone,
>>
>> I am really new to Python contribution community want to propose below
>> methods for List object. Forgive me if this is not the format to send an
>> email.
>>
>> 1. *list.replace( item_to_be_replaced, new_item )*: which replaces all
>> the occurrences of an element in the list instead of writing a new list
>> comprehension in place.
>> 2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences )*:
>> which replaces the occurrences of an element in the list till specific
>> number of occurrences of that element. The number_of_occurrences can
>> defaulted to 0 which will replace all the occurrences in place.
>> 3. *list.removeall( item_to_be_removed )*: which removes all the
>> occurrences of an element in a list in place.
>>
>> What do you think about these features?
>> Are they PEP-able? Did anyone tried to implement these features before?
>> Please let me know.
>>
>> Thank you,
>> Sukumar
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Chris Angelico]

On Mon, Oct 22, 2018 at 11:40 AM Steven D'Aprano  wrote:
>
> On Mon, Oct 22, 2018 at 10:25:19AM +1100, Chris Angelico wrote:
> > On Mon, Oct 22, 2018 at 10:16 AM Terry Reedy  wrote:
>
> > > Except for relatively trivial expressions, this is a bad thing.  All
> > > functions created from lambda expressions get the same pseudo-name
> > > ''.  This can make tracebacks worse.  Perhaps more importantly,
> > > proper testing may become harder.
> >
> > The same considerations bite comprehensions, too, but we don't
> > discourage their use. So I don't think this should be a killer - not
> > on its own, anyhow.
>
> But we *do* discourage large, complex comprehensions, especially nested
> ones. And more importantly, comprehensions are also limited to a single
> expression, like lambda, and if you need a multi-statement comprehension
> we say "turn it into a for-loop".

This is true. But nobody ever says "oh but it's impossible to
unit-test that for-loop". It's just a block of code within a function,
and doesn't NEED to be separately tested. A lambda function should
also behave as a subsection of its surrounding code, and thus not need
to be unit-tested. I don't know why this is a problem for lambda
functions and not for comprehensions or statements. *shrug*

> > I do not currently support any proposed syntax for multi-statement
> > lambda functions, mainly because they've all been ugly. But maybe
> > there'll be one, somewhere, some day, that makes sense.
>
> After 25 years, I think the odds of that are pretty slim.

Also very true... but I'm trying to be welcoming rather than just
rudely say "it hasn't been solved for 25 years so there's no way you
could possibly have solved this". After all, we've had plenty of other
proposals that took many years to finally happen.

Slim is nonzero and I'm open to the possibility that there is now a
new syntax that would work.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Steven D'Aprano]

On Mon, Oct 22, 2018 at 10:25:19AM +1100, Chris Angelico wrote:
> On Mon, Oct 22, 2018 at 10:16 AM Terry Reedy  wrote:

> > Except for relatively trivial expressions, this is a bad thing.  All
> > functions created from lambda expressions get the same pseudo-name
> > ''.  This can make tracebacks worse.  Perhaps more importantly,
> > proper testing may become harder.
> 
> The same considerations bite comprehensions, too, but we don't
> discourage their use. So I don't think this should be a killer - not
> on its own, anyhow.

But we *do* discourage large, complex comprehensions, especially nested 
ones. And more importantly, comprehensions are also limited to a single 
expression, like lambda, and if you need a multi-statement comprehension 
we say "turn it into a for-loop".


> I do not currently support any proposed syntax for multi-statement
> lambda functions, mainly because they've all been ugly. But maybe
> there'll be one, somewhere, some day, that makes sense.

After 25 years, I think the odds of that are pretty slim.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] TypeHinting: From variable name to type

[Steven D'Aprano]

On Sat, Oct 20, 2018 at 07:53:18PM +0200, Thomas Güttler Lists wrote:
> 
> Am 19.10.18 um 12:15 schrieb Steven D'Aprano:

> >>One solution is the do typehinting everywhere the veriable gets used.
> > 
> >You shouldn't need to. Your IDE or type-checker should be able to do
> >type-inference and infer the type. You only need to add a hint if it
> >cannot infer the type.
> >
> >If your IDE doesn't do type inference, get a better IDE *wink*
> 
> 
> I use the free community version of pyCharm, and it often does not know
> which type a variable has. A simple name2type mapping would improve
> the situation a lot.

If the free community version of PyCharm doesn't support type-inference, 
what makes you think it would support this "name2type" mapping?

I don't use PyCharm, and I don't know if this feature applies to the 
free version, but according to their blog, PyCharm not only supports 
type-inference but also uses dynamic runtime information to enhance 
static type checking:

https://blog.jetbrains.com/pycharm/2013/02/dynamic-runtime-type-inference-in-pycharm-2-7/

If you find yourself needing to annotate lots of variables, I suspect 
that you're fighting the IDE rather than using it correctly. Without 
seeing your code, and knowing a lot more about PyCharm, I cannot say 
further.

But the critical point here is that we should not add a language feature 
to make up for the limitations of a single IDE. If the free version of 
PyCharm is underpowered, perhaps you ought to try the paid version, or 
another IDE, or submit a feature request to PyCharm, *before* turning to 
the Python language.



> >>But why does the human brain not need this?
> >>
> >>Because it is intelligent?
> >Yes.
> >
> >  
> >>I would not call this intelligence. There is a simple
> >>dictionary in the brain of the developer, which maps:
> >>
> >>variable-name --> Type
> >Is that a fact? How do you know?
> >
> >How does the information in this dict get filled in?
> 
> 
> as I already said before a docstring in the __init__.py file would
> be a pragmatic solution.


You misunderstand me. You claim that there is a dictionary in the brain 
of the developer, right now. How does that dictionary *in the brain* 
get there, if it isn't through the use of human intelligence?

I believe you are too focused on a single specific implementation:

"use a dict to map variable name to type -- I know how to 
implement this, therefore that is what the brain must do"

whereas I think that the reality is more complex: what you see as a 
mapping, I expect is just a special case of more general, and far more 
powerful, general reasoning intelligence.

We can infer types from variable names, because we know what the name 
means, not because "there is a simple dictionary in the brain of the 
developer" mapping name to type.

exchange_rate # probably a float
num_rows  # number of rows, therefore an int
name  # a string
phone_number  # not a float! should be a string
response  # a response from something (depends on context)

If we read Greek, we'd probably even recognise that a variable called 
"όνομα" was likely to be a string.


[...]
> >When the context is different, we interpret the name differently:
> >
> > response = input("Continue? Y/n ")
> > response = chatbot.lookup(question)
> > response = mapping[challenge]
> >
> >
> >would all be interpreted differently, even if the module used Django. I
> >doubt any reasonable programmer would imagine that they were
> >HttpResponse objects.
> 
> relax. I mean this like I said. Please relax.


Please don't patronise me. Just because I disagree with your suggestion 
doesn't mean I am freaking out and need to be told to relax.

We should not implement new features after ONLY considering when the 
feature would be useful. We should also consider when the feature is not 
needed, and when it would be actively harmful, and weigh it on the 
balance.

Your suggestion is useful when:

- you use the same variable name over and over again;

- always (or almost always) to mean the same type;

- and type inference won't work;

- and you don't want to annotate the variable each time;

- and the IDE supports this new feature;

- and there is little or no chance that you might want to 
  use the same name for a different type in the same module.


It is not useful, maybe even harmful, if any of those conditions are not 
met. I'm especially concerned about the last one: type-checking false 
alarms due to a clash between the name2type mapping and the programmer's 
intention.

But on the other hand, if this is a convention purely implemented by 
IDEs, then I don't give a damn about this one bit. (Except to say, take 
the discussion to the "code-quality" mailing list, not here.) It doesn't 
sound like something that needs language support.


> >What if you are using *both* django and requests in the same module? You
> >could have both of these:
> >
> > response = 

Re: [Python-ideas] Add closing and iteration to threading.Queue

[MRAB]

On 2018-10-21 22:30, Antoine Pitrou wrote:

On Sun, 21 Oct 2018 19:58:05 +0200
Vladimir Filipović 
wrote:


To anticipate a couple more possible questions:

- What would this proposal do about multiple producers/consumers
needing to jointly decide _when_ to close the queue?

Explicitly nothing.

The queue's state is either closed or not, and it doesn't care who
closed it. It needs to interact correctly with multiple consumers and
multiple producers, but once any one piece of code closes it, the
correct interaction is acting like a closed queue for everybody.


Ah.  This is the one statement that makes me favorable to this idea.
When there is a single consumer, it's easy enough to send a sentinel.
But when there are multiple consumers, suddenly you must send exactly
the right number of sentinels (which means you also have to careful
keep track of their number, which isn't always easy).  There's some
delicate code doing exactly that in concurrent.futures.

You don't need more than one sentinel. When a consumer sees the 
sentinel, it just needs to put it back for the other consumers.


[snip]
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Python Enhancement Proposal for List methods

[Chris Angelico]

On Mon, Oct 22, 2018 at 7:58 AM Terry Reedy  wrote:
>
> An indefinite number of in-place removes is a generally a bad idea
> unless done carefully.
> alist.remove is inherently O(n).
> alist.removeall implemented naively would be O(n*n)/
> alist = [x for x in alist if x == old] or
> alist = list(x for x in alist if x == old) or
> alist = list(filter(lambda x: x == old)) is O(n).
>

That's actually a good reason to make it part of the language or
stdlib - it's very easy to get it wrong. All of the above are not
in-place. If I were to make a stdlib function to do removeall, it
would be:

seq[:] = [x for x in seq if x != removeme]

(And that one probably missed some subtlety somewhere too.) So it'd be
less buggy for people to reach for a standard solution than to try to
craft their own.

That said, though, I think this probably belongs as a recipe, not as a
stdlib function or method.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Chris Angelico]

On Mon, Oct 22, 2018 at 10:16 AM Terry Reedy  wrote:
>
> On 10/21/2018 12:28 PM, Andreas Winschu wrote
>
> > A def function has to be named.
>
> In general, this is a good thing.  It often improves tracebacks.
> Perhaps more importantly, name facilitate testing and mocking.
>
> > Wheres a lambda expression can be passed anonymously to any other
> > function as an argument.
>
> Except for relatively trivial expressions, this is a bad thing.  All
> functions created from lambda expressions get the same pseudo-name
> ''.  This can make tracebacks worse.  Perhaps more importantly,
> proper testing may become harder.

The same considerations bite comprehensions, too, but we don't
discourage their use. So I don't think this should be a killer - not
on its own, anyhow.

I do not currently support any proposed syntax for multi-statement
lambda functions, mainly because they've all been ugly. But maybe
there'll be one, somewhere, some day, that makes sense.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Chris Angelico]

On Mon, Oct 22, 2018 at 8:05 AM Vladimir Filipović  wrote:
> From one popular library:
>
> ws = websocket.WebSocketApp(
> url,
> on_open = (lambda ws: ws.send('subscribe'); conn_counter += 1),
> on_message = (lambda ws, msg: print(msg); msg_counter += 1))
>
> Because they include statements, I again need to create those
> functions separately. Naming isn't so much a problem here, but it
> would turn kludgey if I needed to make several such calls in the same
> scope.

Cool! We have an example. Next step: Offer a variety of alternate
syntaxes that *do* currently work, and then the proposed
multi-statement lambda, and show how the current syntaxes are all
ugly.

One syntax that currently can be used to great effect is decorators. I
don't know the specific websocket library that you're using, but it's
generally not difficult to design something so that a function call
can do the assignment. It'd end up looking something like:

ws = websocket.WebSocketApp(url)

@ws.event
def on_open(sock):
sock.send('subscribe')
nonlocal conn_counter # or global, not sure
conn_counter += 1

@ws.event
def on_message(sock, msg):
print(msg)
nonlocal msg_counter
msg_counter += 1

Yes, it's a lot more vertical than your version. However, adding in
the nonlocal declarations definitely makes it look a LOT less
lambda-friendly.

This also takes a bit of extra support. But it can be third-party
support if you need it to. Worst case, you can make a little helper:

events = {}
def event(func):
events[func.__name__] = func
return func

@event
def on_open...
@event
def on_message...

ws = websocket.WebSocketApp(url, **events)

which will work with the same API. Great for larger and more complex setups.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Vladimir Filipović]

On Mon, Oct 22, 2018 at 12:52 AM Steven D'Aprano  wrote:
> > def inc_counter():
> > counter += 1
>
> I don't think that's a real working example.
> ...
> You need to declare counter and sum as global variables.

Good catch!
None of the examples were real, in the sense of being copied directly
from code in actual use.
They were synthesized to illustrate a point while being as brief as possible.

I'm not even really advocating for multi-statement (or statement)
lambdas BTW, I was just answering the question of when one might even
want an anonymous function that can't be expressed using the current
lambda syntax.

> Its not the naming that would make it turn kludgy, but the use of global
> variables. If you had three websockets, would you want them all to share
> the same counter?

No, the naming would turn it kludgey precisely because I'd want
different functions. If there's only one "on_open", it suggests a
natural name for itself. If there are multiple, they can either be
called "on_open1", "on_open2"... or I can keep reusing the same name
in the same scope but that would make the code less clear, not more.

> You've fallen straight into the classic eager versus late binding of
> closures Gotcha.

I did! /facepalm
Thanks. I constructed a very poor example, and it's beginning to look
like the particular point I was trying to illustrate with that doesn't
actually stand.

> If your intention was to demonstrate that multi-statement lambda would
> be a bug-magnet, you have done an excellent job :-)

I don't know why you'd say that. That was a zero-statement lambda :)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Terry Reedy]

On 10/21/2018 12:28 PM, Andreas Winschu wrote


A def function has to be named.


In general, this is a good thing.  It often improves tracebacks. 
Perhaps more importantly, name facilitate testing and mocking.


Wheres a lambda expression can be passed anonymously to any other 
function as an argument.


Except for relatively trivial expressions, this is a bad thing.  All 
functions created from lambda expressions get the same pseudo-name 
''.  This can make tracebacks worse.  Perhaps more importantly, 
proper testing may become harder.



/map(lambda x: x**2, array)/


It is obvious what this trivial expression does, but without a name or 
comment or context indicating intent, how do I know that it is correct? 
Trivial examples evade the issues that arise even with multiline 
expressions, let alone multiple statements.


>>> for i in map(lambda x: x **
 2, 'abc'):
print(i)

Traceback (most recent call last):
  File "", line 2, in 
2, 'abc'):
  File "", line 2, in 
2, 'abc'):
TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int'


vs
/def powers2(x)/
/   x**2/
/map(powers2, array)/


>>> def pow2(x):
return (x **
2)

>>> for i in map(pow2, 'abc'): print(i)

Traceback (most recent call last):
  File "", line 1, in 
for i in map(pow2, 'abc'): print(i)
  File "", line 3, in pow2
2)
TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int'

Given that there might be a hundred functions named '', I think 
the specific name is a bit helpful.


Coming up with a name here is not needed, as the operation is expressive 
enough.


It is clear as to the meaning.  And testing that pow2(3) equals 9 is 
pretty useless.  But you are proposing something else: adding anonymous 
multiple statement functions, where a name *and testing* is usually 
needed.  Justification for what we have today is not justification for a 
major change.


tkinter uses callbacks with widget commands and scheduling.  idlelib has 
77 matches for the re '[( ,]command='.  47 are followed by 
'self.somemethod'; 26 by other function names, and 4 by lambda 
expressions.  All four of the latter are trivial calls of a named 
method: self.somemethod(value).  So when all named non-test functions 
are tested, all non-test callbacks will be tested.


In spite of newbie problems with lambda, I am happy to not have to 
define trivial test functions like this.


def undoNone(): return d.undo_event(None)

But doubt that I would rather write any of the independent 
production-code functions and methods as anonymous lambda constructs.


--
Terry Jan Reedy


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Steven D'Aprano]

On Sun, Oct 21, 2018 at 06:28:40PM +0200, Andreas Winschu wrote:

> A def function has to be named.

That's a feature, not a lack.

As a general rule of thumb, if the body of a function is complex enough 
to require multiple statements, it is probably complex enough to require 
documentation, testing and a nice name to display in tracebacks.

I wish that somebody would do an objective scientific study on the bug 
density of code using named functions versus anonymous code blocks. My 
intuition (which is, of course, completely subjective) tells me that 
code written in languages that allow and encourage anonymous code blocks 
will have more bugs, simply because coders will be less likely to name 
and test their functions.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Steven D'Aprano]

On Sun, Oct 21, 2018 at 11:04:02PM +0200, Vladimir Filipović wrote:

> Sometimes I need that callable to be something very simple, but it
> technically can't be a lambda because it has a statement in it.
> Defining it as a function forces me to give it not only a name, but a
> place too. Defining it just before the point of use is a natural
> location in one sense, but it makes the visual flow of the code worse,
> especially if I need to make several such steps (with different
> almost-lambdas) in a row.
> 
> 
> From the standard library, I can actually think of Thread:
> 
> t1 = Thread(target=(lambda: counter += 1))
> t2 = Thread(target=(lambda: sum += current_quux()))
> 
> Because they include statements, it needs to be:
> 
> def inc_counter():
> counter += 1
> t1 = Thread(target=inc_counter)


I don't think that's a real working example.

py> counter = 0
py> def f():
... counter += 1
...
py> f()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in f
UnboundLocalError: local variable 'counter' referenced before assignment

You need to declare counter and sum as global variables.


> def update_sum():
> sum += current_quux()
> t2 = Thread(target=update_sum)
> 
> 
> From one popular library:
> 
> ws = websocket.WebSocketApp(
> url,
> on_open = (lambda ws: ws.send('subscribe'); conn_counter += 1),
> on_message = (lambda ws, msg: print(msg); msg_counter += 1))
> 
> Because they include statements, I again need to create those
> functions separately. Naming isn't so much a problem here, but it
> would turn kludgey if I needed to make several such calls in the same
> scope.

Its not the naming that would make it turn kludgy, but the use of global 
variables. If you had three websockets, would you want them all to share 
the same counter?


> In any case, that's a common little frustration I often put up with. I
> think most other languages around Python's power level don't make that
> a problem.
> 
> 
> To expand that argument a bit, lambdas have access to names in their
> immediate context that can be more of a pain with less-local
> functions. For a toy example:
> 
> [(lambda a: print(x + a)) for x in l]

You've fallen straight into the classic eager versus late binding of 
closures Gotcha.


py> l = "spam eggs cheese aardvark".split()
py> funcs = [(lambda a: print(x + a)) for x in l]
py> for f in funcs:
... f("")
...
aardvark
aardvark
aardvark
aardvark


If your intention was to demonstrate that multi-statement lambda would 
be a bug-magnet, you have done an excellent job :-)



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add closing and iteration to threading.Queue

[Antoine Pitrou]

On Sun, 21 Oct 2018 19:58:05 +0200
Vladimir Filipović 
wrote:
> 
> To anticipate a couple more possible questions:
> 
> - What would this proposal do about multiple producers/consumers
> needing to jointly decide _when_ to close the queue?
> 
> Explicitly nothing.
> 
> The queue's state is either closed or not, and it doesn't care who
> closed it. It needs to interact correctly with multiple consumers and
> multiple producers, but once any one piece of code closes it, the
> correct interaction is acting like a closed queue for everybody.

Ah.  This is the one statement that makes me favorable to this idea.
When there is a single consumer, it's easy enough to send a sentinel.
But when there are multiple consumers, suddenly you must send exactly
the right number of sentinels (which means you also have to careful
keep track of their number, which isn't always easy).  There's some
delicate code doing exactly that in concurrent.futures.

> - Should multiprocessing.Queue do the same thing?
> 
> I think so, though I'm not proposing it.
> 
> It already has a close() method, whose meaning is very similar but not
> identical to (a subset of) the proposed threading.Queue.close's
> meaning (with resource-management effects not relevant to
> threading.Queue either way).

Not really similar, unfortunately.  mp.Queue.close() isn't a logical
thing, but releases the queue's internal resources.  It doesn't signal
consumers that the producers has finished with the queue.

Perhaps if you renamed close() to something else ("put_eof" or
"put_end" perhaps?) that would allow porting it to mp.Queue?

Regards

Antoine.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Python Enhancement Proposal for List methods

[Terry Reedy]

On 10/21/2018 9:00 AM, Siva Sukumar Reddy wrote:

I am really new to Python contribution community want to propose below 
methods for List object.


For most classes, there is an infinite number of possible functions that 
could be made into methods.  The Python philosophy is to include as 
methods the basic functions needed to write other functions.  So number 
classes include the basic operations of +, -, *, /, and ** as methods. 
The math and cmath modules contain number functions that use these basic 
operations.


The list class also has a fairly small set of methods.  Nearly all are 
either original or were added before list.pop, about 20 years ago. 
list.clear was added more recently, in analogy with dict.clear, and 
there was some opposition because there are other ways to clear a list.


1. *list.replace( item_to_be_replaced, new_item )*: which replaces all 
the occurrences of an element in the list instead of writing a new list 
comprehension in place.
2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences 
)*: which replaces the occurrences of an element in the list till 
specific number of occurrences of that element. The 
number_of_occurrences can defaulted to 0 which will replace all the 
occurrences in place.


I presume these are somewhat in analogy with str.replace(old, new, 
count=-1), except for being limited to one sequence member and being 
done in place.  Note that only 1 method is needed and the default count 
should be -1, not 0.


However, the need for a built-in method is much, much less.  Nearly all 
cases that need to be done inline code can be done like this.


for i, item in enumerate(mylist):
if item == old:
mylist[i] = new

As Serhiy noted from previous discussion, other conditions are possible, 
such as 'item in {old1, old2, old3}' or, given a list of numbers,

if item > max: mylist[i] = max

3. *list.removeall( item_to_be_removed )*: which removes all the 
occurrences of an element in a list in place.


str.replace(old, '') does this (not in place) for strings.  There are 
also str.translate and re.sub and re.subn.


An indefinite number of in-place removes is a generally a bad idea 
unless done carefully.

alist.remove is inherently O(n).
alist.removeall implemented naively would be O(n*n)/
alist = [x for x in alist if x == old] or
alist = list(x for x in alist if x == old) or
alist = list(filter(lambda x: x == old)) is O(n).

Again, as Serhiy noted, 'x == old' is only one possible condition.

The last example illustrates another point.  In Python 3, one can 
usually and often should avoid turning streams of objects into a 
concrete list until one needs one of the particular list operations.


I noted above that we have only added 1 list method (that I know of) 
since .pop.  On the other hand, we added generators and then the 
itertools module with multiple basic functions and multiple recipes 
based on those functions.  Since lists are no longer the basic 
representation of sequences of items, adding list methods is somewhat 
obsolete.


If possible, it is best to filter out multiple items when creating the 
list instead of later.  Or to put it another way, it is best to avoid 
turning a stream into a list for as long as possible.  If space and 
aliasing are not issues, replacing a list is easier and possibly faster 
that editing it in place.  Note that editors for adding, replacing, and 
deleting items tend to use tree structures rather than a single linear 
sequence.


An alternative to removing items is to replace them with a null value, 
such as None, '', 0, 1 (for multiplication), 'pass' (for code 
execution), or sentinal=object().  The choice depends on what is to be 
done later.  In fact, temporary replacement is the key to efficient 
multiple removals.


--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add closing and iteration to threading.Queue

[Nathaniel Smith]

Hi Vladimir,

It's great to see people revisiting these old stdlib tools. Closure
tracking is definitely a big point of awkwardness for Queues. In Trio we
started with a straight copy of threading.Queue, and this turned out to be
a major friction point for users. We just deprecated our version of Queue
and replaced it with a new design. Our new thing is probably more radical
than you want to get in the stdlib (we ended up splitting the object into
two pieces, a sender object and a receiver object), but you might find the
discussions interesting:

Manual:
https://trio.readthedocs.io/en/latest/reference-core.html#using-channels-to-pass-values-between-tasks

A more minimal proposal to add closure tracking to trio.Queue:
https://github.com/python-trio/trio/pull/573

Follow-up issue with design questions we're still thinking about (also
links to earlier design discussions):
https://github.com/python-trio/trio/issues/719

We only started shipping this last week, so we're still getting experience
with it.

-n

On Sun, Oct 21, 2018, 10:59 Vladimir Filipović  wrote:

> Hi!
>
> I originally submitted this as a pull request. Raymond Hettinger
> suggested it should be given a shakeout in python-ideas first.
>
> https://github.com/python/cpython/pull/10018
> https://bugs.python.org/issue35034
>
> --
>
> Briefly:
>
> Add a close() method to Queue, which should simplify many common uses
> of the class and reduce the space for some easy-to-make errors.
>
> Also add an __iter__() method which in conjunction with close() would
> further simplify some common use patterns.
>
> --
>
> At eye-watering length:
>
> Apologies in advance for the length of this message. This isn't a PEP
> in disguise, it's a proposal for a very small, simple and I dare
> imagine uncontroversial feature. I'm new to contributing to Python and
> after the BPO/github submission I didn't manage to come up with a
> better way to present it than this.
>
> The issue
>
> Code using threading.Queue often needs to coordinate a "work is
> finished as far as I care" state between the producing and consuming
> side. Not "work" in the task_done() sense of completion of processing
> of queue items, "work" in the simpler sense of just passing data
> through the queue.
>
> For example, a producer can be driving the communication by enqueuing
> e.g. names of files that need to be processed, and once it's enqueued
> the last filename, it can be useful to inform the consumers that no
> further names will be coming, so after they've retrieved what's
> in-flight currently, they don't need to bother waiting for any more.
> Alternatively, a consumer can be driving the communication, and may
> need to let the producers know "I'm not interested in any more, so you
> can stop wasting resources on producing and enqueuing them".
> Also, a third, coordinating component may need to let both sides know
> that "Our collective work here is done. Start wrapping it up y'all,
> but don't drop any items that are still in-flight."
>
> In practice it's probably the exception, not the rule, when any piece
> of code interacting with a Queue _doesn't_ have to either inform
> another component that its interest in transferring the data has
> ended, or watch for such information.
>
> In the most common case of producer letting consumers know that it's
> done, this is usually implemented (over and over again) with sentinel
> objects, which is at best needlessly verbose and at worst error-prone.
> A recipe for multiple consumers making sure nobody misses the sentinel
> is not complicated, but neither is it obvious the first time one needs
> to do it.
> When a generic sentinel (None or similar) isn't adequate, some
> component needs to create the sentinel object and communicate it to
> the others, which complicates code, and especially complicates
> interfaces between components that are not being developed together
> (e.g. if one of them is part of a library and expects the library-user
> code to talk to it through a Queue).
>
> In the less common cases where the producers are the ones being
> notified, there isn't even a typical solution - everything needs to be
> cooked up from scratch using synchronization primitives.
>
> --
>
> A solution
>
> Adding a close() method to the Queue that simply prohibits all further
> put()'s (with other methods acting appropriately when the queue is
> closed) would simplify a lot of this in a clean and safe way - for the
> most obvious example, multi-consumer code would not have to juggle
> sentinel objects.
>
> Adding a further __iter__() method (that would block as necessary, and
> stop its iteration once the queue is closed and exhausted) would
> especially simplify many unsophisticated consumers.
>
> This is a current fairly ordinary pattern:
>
> # Producer:
> while some_condition:
> q.put(generate_item())
> q.put(sentinel)
>
> # Consumer:
> while True:
> item = q.get()
> if item == sentinel:
> q.put(sentinel)
> break

Re: [Python-ideas] Multi Statement Lambdas

[Tobias Kohn]

 it might just be a detail, but Python does not even have  
single-statement lambdas.  The body of a lambda is an expression  
yielding a value, not a statement.


Function languages (from which the idea of the Lambda in Python  
probably came from) do not have statements at all.  Something like the  
example with "y = x+1; y*2" given earlier is usually expressed as a  
let-expression, which is syntactic sugar for a lambda itself.  Hence,  
the example could actually be written in Python like so (I am not  
saying it is beautiful, but just that it is possible ^_^):

  arr.map( lambda x: (lambda y:y*2)(x+1) )
Or, if you prefer (this seems to me to be syntactically very close to  
the original):

  arr.map( lambda x: (lambda y=x+1: y*2)() )

Moreover, in Python 3.8, we will have assignments in expressions, and  
(even though I obviously can't test this) I wonder if you could then  
write the same thing as:

  arr.map( lambda x: (y := x+1, y*2)[1] )

I guess, the original request is therefore not really about having  
multi-statement lambdas, but more about extending lambdas to anonymous  
functions with the possibility to have full statements inside the body.


Finally, I absolutely agree with that good naming is paramount for  
understandable code.  But IMHO, good naming means that I write a small  
/named/ function (describing its intent) if its body contains more  
than just a simple expression, anyway.


Cheers,
Tobias

Quoting Anders Hovmöller :

A powerful general purpose language should not limit itself to one  
statement in a closure.


Nitpick on language: It doesn't. Lambdas are not the only way to do  
a closure.


It's important to be precise when discussing these things.

Lets add mutli-statement lambdas to python either with just curly  
braces or with an indent based machine.


Mostly it's just better with a function. Do you have some more  
compelling examples where you can't use a function smoothly?


/ Anders
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideasCode of  
Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Ron Reiter]

Just write all your transformations at the top of the file and then write
the mapreduce chain at the end. If you need to use closures that depend on
each other then it may be a bit more ugly but still feasible to implement.

- Ron


[image: Facebook]  [image: Twitter]
 [image: LinkedIn]



On Sun, Oct 21, 2018 at 10:20 PM Marko Ristin-Kaufmann <
marko.ris...@gmail.com> wrote:

> Hi,
> What about writing longer map in pyspark? When I worked with Spark
> and Scala, it was easy to script a longer chain of transformations in
> Scala. Does anybody know how that works in Python without multiline lambdas?
>
> Cheers Marko
>
>
> Le dim. 21 oct. 2018 à 18:40, Ron Reiter  a écrit :
>
>> Multi-line lambdas may be nice for functional languages, but Python is
>> very imperative in nature and prefers clean syntax over conciseness, which
>> means it will make it REALLY hard to fit with Python. I personally find
>> multi-line lambdas an unreadable abomination.
>>
>> As for some more concrete reasons on why this should not be a part of
>> Python, see here for an example which shows why it would even be hard to
>> come up with a plausible syntax for multi-line lambdas:
>>
>> https://stackoverflow.com/questions/1233448/no-multiline-lambda-in-python-why-not
>>
>> Guido's words:
>>
>> "But the complexity of any proposed solution for this puzzle is immense,
>> to me: it requires the parser (or more precisely, the lexer) to be able to
>> switch back and forth between indent-sensitive and indent-insensitive
>> modes, keeping a stack of previous modes and indentation level. Technically
>> that can all be solved (there's already a stack of indentation levels that
>> could be generalized). But none of that takes away my gut feeling that it
>> is all an elaborate Rube Goldberg contraption."
>>
>> If one would find a syntax which is Pythonic and clean then I am in favor
>> of adding it, but I argue it simply does not exist.
>>
>> - Ron
>>
>>
>> [image: Facebook]  [image: Twitter]
>>  [image: LinkedIn]
>> 
>>
>>
>> On Sun, Oct 21, 2018 at 7:34 PM Anders Hovmöller 
>> wrote:
>>
>>>
>>> Wheres a lambda expression can be passed anonymously to any other
>>> function as an argument.
>>>
>>> *map(lambda x: x**2, array)*
>>>
>>> vs
>>>
>>> *def powers2(x)*
>>> *   x**2*
>>> *map(powers2, array)*
>>>
>>> Coming up with a name here is not needed, as the operation is expressive
>>> enough.
>>>
>>>
>>> Sure, but there is a well known convention for such non-names already:
>>> meta syntactical variables. In this case the name could be "foo" and all
>>> readers will interpret this as "no name".
>>>
>>> I admit to have wanted lambdas that are just as powerful as normal
>>> functions, but often when I want it the code is nicer when a normal
>>> function is used.
>>>
>>> / Anders
>>> ___
>>> Python-ideas mailing list
>>> Python-ideas@python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add closing and iteration to threading.Queue

[Vladimir Filipović]

On Sun, Oct 21, 2018 at 8:45 PM MRAB  wrote:
> FTR, this has been discussed before:
>
> [Python-ideas] `__iter__` for queues?
> https://mail.python.org/pipermail/python-ideas/2010-January/006711.html

Thank you!

For the sake of clarity, I want to outline a few differences between
that discussion and my proposal:

1. Much of the discussion there seemed to implicitly limit itself to
consideration of FIFO queues. This proposal works cleanly for child
classes too, including any (API-compliant) user-written children.

2. Throughout that discussion, iteration is the A feature, and closing
is occasionally mentioned as a possible prerequisite. In this
proposal, the A feature is closing, which enables sensible iteration
(as a B feature) but is useful even if iteration isn't used.

3. There's naturally a lot of quick spitballing of various
mutually-incompatible ideas there, whereas this is one rounded
self-consistent proposal. Most of what I've come up with has already
been anticipated there but it's all mixed up textually.

4. This proposal sidesteps a lot of the doubts and difficulties by
just not using sentinels at all. Being closed is part of the queue's
state that can be queried at any time, and will affect put() calls
immediately, without waiting for a sentinel to float up to the front.
(With recognition that your (MRAB's) message towards that thread's end
already proposed the same approach.)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Marko Ristin-Kaufmann]

Hi,
What about writing longer map in pyspark? When I worked with Spark
and Scala, it was easy to script a longer chain of transformations in
Scala. Does anybody know how that works in Python without multiline lambdas?

Cheers Marko


Le dim. 21 oct. 2018 à 18:40, Ron Reiter  a écrit :

> Multi-line lambdas may be nice for functional languages, but Python is
> very imperative in nature and prefers clean syntax over conciseness, which
> means it will make it REALLY hard to fit with Python. I personally find
> multi-line lambdas an unreadable abomination.
>
> As for some more concrete reasons on why this should not be a part of
> Python, see here for an example which shows why it would even be hard to
> come up with a plausible syntax for multi-line lambdas:
>
> https://stackoverflow.com/questions/1233448/no-multiline-lambda-in-python-why-not
>
> Guido's words:
>
> "But the complexity of any proposed solution for this puzzle is immense,
> to me: it requires the parser (or more precisely, the lexer) to be able to
> switch back and forth between indent-sensitive and indent-insensitive
> modes, keeping a stack of previous modes and indentation level. Technically
> that can all be solved (there's already a stack of indentation levels that
> could be generalized). But none of that takes away my gut feeling that it
> is all an elaborate Rube Goldberg contraption."
>
> If one would find a syntax which is Pythonic and clean then I am in favor
> of adding it, but I argue it simply does not exist.
>
> - Ron
>
>
> [image: Facebook]  [image: Twitter]
>  [image: LinkedIn]
> 
>
>
> On Sun, Oct 21, 2018 at 7:34 PM Anders Hovmöller 
> wrote:
>
>>
>> Wheres a lambda expression can be passed anonymously to any other
>> function as an argument.
>>
>> *map(lambda x: x**2, array)*
>>
>> vs
>>
>> *def powers2(x)*
>> *   x**2*
>> *map(powers2, array)*
>>
>> Coming up with a name here is not needed, as the operation is expressive
>> enough.
>>
>>
>> Sure, but there is a well known convention for such non-names already:
>> meta syntactical variables. In this case the name could be "foo" and all
>> readers will interpret this as "no name".
>>
>> I admit to have wanted lambdas that are just as powerful as normal
>> functions, but often when I want it the code is nicer when a normal
>> function is used.
>>
>> / Anders
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add closing and iteration to threading.Queue

[MRAB]

On 2018-10-21 18:58, Vladimir Filipović wrote:

Hi!

I originally submitted this as a pull request. Raymond Hettinger
suggested it should be given a shakeout in python-ideas first.

https://github.com/python/cpython/pull/10018
https://bugs.python.org/issue35034


[snip]
FTR, this has been discussed before:

[Python-ideas] `__iter__` for queues?
https://mail.python.org/pipermail/python-ideas/2010-January/006711.html
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Add closing and iteration to threading.Queue

[Vladimir Filipović]

Hi!

I originally submitted this as a pull request. Raymond Hettinger
suggested it should be given a shakeout in python-ideas first.

https://github.com/python/cpython/pull/10018
https://bugs.python.org/issue35034

--

Briefly:

Add a close() method to Queue, which should simplify many common uses
of the class and reduce the space for some easy-to-make errors.

Also add an __iter__() method which in conjunction with close() would
further simplify some common use patterns.

--

At eye-watering length:

Apologies in advance for the length of this message. This isn't a PEP
in disguise, it's a proposal for a very small, simple and I dare
imagine uncontroversial feature. I'm new to contributing to Python and
after the BPO/github submission I didn't manage to come up with a
better way to present it than this.

The issue

Code using threading.Queue often needs to coordinate a "work is
finished as far as I care" state between the producing and consuming
side. Not "work" in the task_done() sense of completion of processing
of queue items, "work" in the simpler sense of just passing data
through the queue.

For example, a producer can be driving the communication by enqueuing
e.g. names of files that need to be processed, and once it's enqueued
the last filename, it can be useful to inform the consumers that no
further names will be coming, so after they've retrieved what's
in-flight currently, they don't need to bother waiting for any more.
Alternatively, a consumer can be driving the communication, and may
need to let the producers know "I'm not interested in any more, so you
can stop wasting resources on producing and enqueuing them".
Also, a third, coordinating component may need to let both sides know
that "Our collective work here is done. Start wrapping it up y'all,
but don't drop any items that are still in-flight."

In practice it's probably the exception, not the rule, when any piece
of code interacting with a Queue _doesn't_ have to either inform
another component that its interest in transferring the data has
ended, or watch for such information.

In the most common case of producer letting consumers know that it's
done, this is usually implemented (over and over again) with sentinel
objects, which is at best needlessly verbose and at worst error-prone.
A recipe for multiple consumers making sure nobody misses the sentinel
is not complicated, but neither is it obvious the first time one needs
to do it.
When a generic sentinel (None or similar) isn't adequate, some
component needs to create the sentinel object and communicate it to
the others, which complicates code, and especially complicates
interfaces between components that are not being developed together
(e.g. if one of them is part of a library and expects the library-user
code to talk to it through a Queue).

In the less common cases where the producers are the ones being
notified, there isn't even a typical solution - everything needs to be
cooked up from scratch using synchronization primitives.

--

A solution

Adding a close() method to the Queue that simply prohibits all further
put()'s (with other methods acting appropriately when the queue is
closed) would simplify a lot of this in a clean and safe way - for the
most obvious example, multi-consumer code would not have to juggle
sentinel objects.

Adding a further __iter__() method (that would block as necessary, and
stop its iteration once the queue is closed and exhausted) would
especially simplify many unsophisticated consumers.

This is a current fairly ordinary pattern:

# Producer:
while some_condition:
q.put(generate_item())
q.put(sentinel)

# Consumer:
while True:
item = q.get()
if item == sentinel:
q.put(sentinel)
break
process(item)

(This consumer could be simplified a little with an assignment
expression or an iter(q.get, sentinel), but one of those is super new
and the other seems little-known in spite of being nearly old enough
to vote.)

With the close() and __iter__(), this would become:

# Producer:
with closing(q):
while some_condition:
q.put(generate_item())

# Consumer:
for item in q:
process(item)

Apart from it being shorter and less error-prone (e.g. if
generate_item raises), the implied interface for initializing the two
components is also simplified, because there's no sentinel to pass
around.

More complex consumers that couldn't take advantage of the __iter__()
would still benefit from being able to explicitly and readably find
out (via exception or querying) that the queue has been closed and
exhausted, without juggling the sentinel.

I honestly think this small addition would be an unqualified win. And
it would not change anything for code that doesn't want to use it.

--

I've got a sample implementation ready for Queue and its children.
(Link is at the start of this message. It includes documentation
updates too, in case those clarify anything further at this stage.)

If this is going in the right 

Re: [Python-ideas] Multi Statement Lambdas

[Ron Reiter]

Multi-line lambdas may be nice for functional languages, but Python is very
imperative in nature and prefers clean syntax over conciseness, which means
it will make it REALLY hard to fit with Python. I personally find
multi-line lambdas an unreadable abomination.

As for some more concrete reasons on why this should not be a part of
Python, see here for an example which shows why it would even be hard to
come up with a plausible syntax for multi-line lambdas:
https://stackoverflow.com/questions/1233448/no-multiline-lambda-in-python-why-not

Guido's words:

"But the complexity of any proposed solution for this puzzle is immense, to
me: it requires the parser (or more precisely, the lexer) to be able to
switch back and forth between indent-sensitive and indent-insensitive
modes, keeping a stack of previous modes and indentation level. Technically
that can all be solved (there's already a stack of indentation levels that
could be generalized). But none of that takes away my gut feeling that it
is all an elaborate Rube Goldberg contraption."

If one would find a syntax which is Pythonic and clean then I am in favor
of adding it, but I argue it simply does not exist.

- Ron


[image: Facebook]  [image: Twitter]
 [image: LinkedIn]



On Sun, Oct 21, 2018 at 7:34 PM Anders Hovmöller 
wrote:

>
> Wheres a lambda expression can be passed anonymously to any other function
> as an argument.
>
> *map(lambda x: x**2, array)*
>
> vs
>
> *def powers2(x)*
> *   x**2*
> *map(powers2, array)*
>
> Coming up with a name here is not needed, as the operation is expressive
> enough.
>
>
> Sure, but there is a well known convention for such non-names already:
> meta syntactical variables. In this case the name could be "foo" and all
> readers will interpret this as "no name".
>
> I admit to have wanted lambdas that are just as powerful as normal
> functions, but often when I want it the code is nicer when a normal
> function is used.
>
> / Anders
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Andreas Winschu]

A def function has to be named.

Wheres a lambda expression can be passed anonymously to any other function
as an argument.

*map(lambda x: x**2, array)*

vs

*def powers2(x)*
*   x**2*
*map(powers2, array)*

Coming up with a name here is not needed, as the operation is expressive
enough.



Am So., 21. Okt. 2018 um 18:17 Uhr schrieb Anders Hovmöller <
bo...@killingar.net>:

>
> >  A powerful general purpose language should not limit itself to one
> statement in a closure.
>
> Nitpick on language: It doesn't. Lambdas are not the only way to do a
> closure.
>
> It's important to be precise when discussing these things.
>
> > Lets add mutli-statement lambdas to python either with just curly braces
> or with an indent based machine.
>
> Mostly it's just better with a function. Do you have some more compelling
> examples where you can't use a function smoothly?
>
> / Anders



-- 
Gruß

Andreas Winschu
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Anders Hovmöller]

>  A powerful general purpose language should not limit itself to one statement 
> in a closure.

Nitpick on language: It doesn't. Lambdas are not the only way to do a closure. 

It's important to be precise when discussing these things. 

> Lets add mutli-statement lambdas to python either with just curly braces or 
> with an indent based machine. 

Mostly it's just better with a function. Do you have some more compelling 
examples where you can't use a function smoothly?

/ Anders
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Multi Statement Lambdas

[Chris Angelico]

On Mon, Oct 22, 2018 at 3:07 AM Andreas Winschu  wrote:
> Guido had a hard opinion on this, but i hope the python community does not. A 
> powerful general purpose language should not limit itself to one statement in 
> a closure.
> Lets add mutli-statement lambdas to python either with just curly braces or 
> with an indent based machine.
>

Just to be clear here, a *closure* is fully capable of having multiple
statements in it. You can use 'def' inside a function, and it will
create a closure.

Proposals like this stand or fall on the syntax to be used. Can you be
completely specific about exactly what the syntax is that you're
proposing, and give some examples? Don't forget that this is currently
valid code:

func(lambda x: {})

so you'll need to clearly distinguish your block delimiters from
set/dict display.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] TypeHinting: From variable name to type

[Kyle Lahnakoski]

On 2018-10-21 10:44, David Mertz wrote:
> On Fri, Oct 19, 2018 at 3:18 AM Thomas Güttler
> mailto:guettl...@thomas-guettler.de>>
> wrote:
>
> Now my idea: Per module and/or per file type hinting from variable
> name.
> Maybe a magic docstring in the __init__.py file:
> variable-name-mapping:
>   {
>     request: django.http.HttpRequest,
> }
>
>
> In general, I like this idea; but really only on a per-file basis.  A
> short header at the top would be easy enough for an IDE or linter to
> scan.  But imposing the conventions project-wide feels too broad.
>

Maybe in a .editorconfig file?  This would allow type hinting at a
number of directory levels.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Multi Statement Lambdas

[Andreas Winschu]

There was an extensive discussion about this in the past.

https://www.artima.com/weblogs/viewpost.jsp?thread=147358

In the end Guido felt that his effort seems to him like building a Rube
Goldberg machine.

By looking at the amount of people still stumbling on this issue
we definitely feel, that it is not
the. Such a machine is a useful one.

"Naming things" was from the beginning one the two hardest things in
programming. Good naming is important for understandable code. The balance
of extracted, named functions, as most of things in programming, is
subjective though.

The python style guide itself encourages people to use simple data
transformations like `map`, `filter`, `reduce` for expressiveness. Those
transformations can still be simple with 2 or 3 statements and will not
require naming. Naming the transformation becomes harder, than reading the
actual code.

Coffeescript, which is another indent based language, handles multi
statement lambdas
without curly braces.

*arr = []*
*arr.map((x)-> *
*  y = x+1*
*  y * 2*
*)*

There is also an argument in this discussion, that just using curly braces
for lambdas (which is easier to implement) would be pretty unpythonic.

I really agree that python curly braces free block style is really helpful
for readability in contrast to javascript for example. However using curly
braces for lambdas in ruby is not hurting the eyes that much. Normally
blocks in ruby  use `do .. end` notation. Thus lambdas are clearly visible
and also people try not to nest multiple levels of lamda transfromations.
We strive to keep such a chain flat.


*def transform   arr*
* .map {  ... }*

* .filter  { ... } .reduce { ... }*

In the end curly branes for lambdas is also a mathematical notation.

Guido had a hard opinion on this, but i hope the python community does not.
A powerful general purpose language should not limit itself to one
statement in a closure.
Lets add mutli-statement lambdas to python either with just curly braces or
with an indent based machine.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Python Enhancement Proposal for List methods

[Serhiy Storchaka]

21.10.18 16:00, Siva Sukumar Reddy пише:
I am really new to Python contribution community want to propose below 
methods for List object. Forgive me if this is not the format to send an 
email.


1. *list.replace( item_to_be_replaced, new_item )*: which replaces all 
the occurrences of an element in the list instead of writing a new list 
comprehension in place.
2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences 
)*: which replaces the occurrences of an element in the list till 
specific number of occurrences of that element. The 
number_of_occurrences can defaulted to 0 which will replace all the 
occurrences in place.
3. *list.removeall( item_to_be_removed )*: which removes all the 
occurrences of an element in a list in place.


What do you think about these features?
Are they PEP-able? Did anyone tried to implement these features before?
Please let me know.


See the previous discussion about this three years ago:
https://mail.python.org/pipermail/python-ideas/2015-October/036770.html

The conclusion is that this feature is too niche. It works only with 
lists, and supports only one test predicate (equality to a specified 
value). filter() and comprehensions are more general.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] TypeHinting: From variable name to type

[David Mertz]

On Fri, Oct 19, 2018 at 3:18 AM Thomas Güttler 
wrote:

> Now my idea: Per module and/or per file type hinting from variable name.
> Maybe a magic docstring in the __init__.py file:
> variable-name-mapping:
>   {
> request: django.http.HttpRequest,
> }
>

In general, I like this idea; but really only on a per-file basis.  A short
header at the top would be easy enough for an IDE or linter to scan.  But
imposing the conventions project-wide feels too broad.

There might, of course, be cases where the same name is used for different
purposes in the same file.  But the tool can alert the developer of that...
and in that one file, she could either remove the header of refactor the
names used, as made sense for that particular code.

This is certainly not something that requires language support.  It can
easily be purely a convention, as long as different IDEs, linters, type
checkers, etc. agree on what the convention is.  Maybe at some point in the
future, if the convention becomes adopted, there might be some help in
having a standard library module, or even minimal language recognition, of
the convention.  But let's work on adopting a convention first.



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Python Enhancement Proposal for List methods

[David Mertz]

The list comprehensions are not very hard, and are more general. EXCEPT
with the limited number of occurrences. We have this for str.replace(...,
max=n), and it is useful fairly often.

I'm +0.5 on .replace() with that capability. But -1 on .removeall() that
adds nothing to an easy listcomp.

On Sun, Oct 21, 2018, 9:01 AM Siva Sukumar Reddy  Hey everyone,
>
> I am really new to Python contribution community want to propose below
> methods for List object. Forgive me if this is not the format to send an
> email.
>
> 1. *list.replace( item_to_be_replaced, new_item )*: which replaces all
> the occurrences of an element in the list instead of writing a new list
> comprehension in place.
> 2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences )*:
> which replaces the occurrences of an element in the list till specific
> number of occurrences of that element. The number_of_occurrences can
> defaulted to 0 which will replace all the occurrences in place.
> 3. *list.removeall( item_to_be_removed )*: which removes all the
> occurrences of an element in a list in place.
>
> What do you think about these features?
> Are they PEP-able? Did anyone tried to implement these features before?
> Please let me know.
>
> Thank you,
> Sukumar
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Python Enhancement Proposal for List methods

[Siva Sukumar Reddy]

Hey everyone,

I am really new to Python contribution community want to propose below
methods for List object. Forgive me if this is not the format to send an
email.

1. *list.replace( item_to_be_replaced, new_item )*: which replaces all the
occurrences of an element in the list instead of writing a new list
comprehension in place.
2. *list.replace( item_to_be_replaced, new_item, number_of_occurrences )*:
which replaces the occurrences of an element in the list till specific
number of occurrences of that element. The number_of_occurrences can
defaulted to 0 which will replace all the occurrences in place.
3. *list.removeall( item_to_be_removed )*: which removes all the
occurrences of an element in a list in place.

What do you think about these features?
Are they PEP-able? Did anyone tried to implement these features before?
Please let me know.

Thank you,
Sukumar
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] IPLike objects

[Andrew Svetlov]

Should we support bytes? Why? For path name, the bytes are sometimes the
only way to encode the file name, Posix operates with bytes internally.
But IP addresses are always ASCII strings. I doubt if we shall support
bytes for it, complicating the implementation.

Support for `int` in `get_addr()` is event more questionable and
error-prone.

The main consumer of IPLIke API is socket module.
Moving IPLike from ipaddress.py into socket.py makes sense for me.

Calling `ip_address` from `get_ipaddr()` is another question.
I doubt if `get_ipaddr()` is required at all.
In my mind `socket.ipaddr(arg)` should be very similar to `os,fspath` and
implemented in the same way:
1. Return argument as is if it is a string,
2. Otherwise try to return `type(arg).__ipaddr__(arg)`

`socket.ipaddr` should be as fast as possible, don't process extra
conversions and have a C Accelerated version used by default.
Basically it is a mirror of `os.fspath()`, `socket.IPLike` is a mirror of
`os.PathLike`.

Your thoughts?

On Sun, Oct 21, 2018 at 1:59 PM Damla Altun 
wrote:

> Hi,
>
> Python's ipaddress module is really good for working with network / IP
> related thing.
> I would like to use ipaddress.IPv4/v6Address in creating network
> interfaces. Like a socket but i can't,
> the internal things and some standard libraries doesn't support them (even
> socket).
>
> >>> server_address = (ipaddress.IPv4Address('127.0.0.1'), 1)
> >>> sock.bind(server_address)
> Traceback (most recent call last):
> File "", line 1, in 
> TypeError: str, bytes or bytearray expected, not IPv4Address
>
>
> Proposal
> 
>
> The best solution we've got[0] is implementing something like PathLike[1]
> for IP Address objects. For PoC[2]
> we called it IPLike. It is a protocol (some kind of interface) created
> with Abstract Base Classes and
> it forces class to implement __ipaddr__ method (like __fspath__ method).
> The __ipaddr__ method should return
> string or bytes formed notation of IP address. It is dotted notation in
> IPv4, long formed notation in IPv6.
>
> The __ipaddr__ method is also an identifier for IPLike classes. IPLike
> interface's subclasshook
> checks the __ipaddr__ method of given class when issubclass called.
>
> >>> class SomethingRepresentIP:
> ... def __ipaddr__(self) -> str:
> ... return '127.0.0.1'
> ...
> >>> issubclass(SomethingRepresentIP, IPLike)
> True
> >>> isinstance(SomethingRepresentIP(), IPLike)
> True
>
> For PoC[2] we implented __ipaddr__ method to ipaddress._BaseAddress class.
> It is parent class of IPv4/v6Adress.
>
> >>> issubclass(IPv4Address, IPLike)
> True
> >>> IPv4Address('127.0.0.1').__ipaddr__()
> '127.0.0.1'
>
> Like os.fspath we need a general-purpose ip-getter. So we implemented
> ipaddress.get_ipaddr(address) function.
> When you give it an valid IP like object it returns the value of
> __ipaddr__ method.
>
> >>> get_ipaddr(IPv4Address('127.0.0.1'))
> '127.0.0.1'
>
> When you give it string or bytes or integer object it calls
> ipaddr.ip_address(address) and re-calls itself with
> the value it got from ip_address. It helps get_ipaddr to convert
> integer-formed / byte-formed IPs to string formed IPs
> with a valid notation.
>
> >>> get_ipaddr('127.0.0.1')
> '127.0.0.1'
> >>> get_ipaddr(3232235521)
> '192.168.0.1'
> >>> get_ipaddr(b'\xC0\xA8\x00\x01')
> '192.168.0.1'
>
> If you give any type else it raises TypeError. Also if you give it a IP
> like object (an object that has __ipaddr__ method),
> but the object's __ipaddr__ method returns something that is not
> string/bytes it raises TypeError.
>
> For PoC[2] i added socket to support IP like objects.
>
> >>> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> >>> server_address = (SomethingRepresentIP(), 1)
> >>> sock.bind(server_address)
> >>> sock.listen(1)
> >>> conn, client = sock.accept()
> >>> conn
>  type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 1),
> raddr=('127.0.0.1', 49452)>
>
> I didn't implement a C API for ipaddress.get_ipaddr. The function is too
> small for creating a new module, new header and the ipaddress library
> currently has no
> C-side module. If needed i can write a function inside the socketmodule.c .
>
> Proof-of-Concept (Implementation)
> =
>
> The implementation can be found on github (DamlaAltun/cpython[2]) We have
> a Abstract Base Class for IP Address protocol called IPLike.
>
> (ipaddress.py[3])
>
> class IPLike(abc.ABC):
> """
> Abstract base class for IP (address) like objects.
> """
>
> @abc.abstractmethod
> def __ipaddr__(self):
> """
> If it is a IPv4 return in the dotted form,
> If it is a IPv6 return in the long form.
> """
> raise NotImplementedError
>
> @classmethod
> def __subclasshook__(cls, subclass):
> return 

[Python-ideas] IPLike objects

[Damla Altun]

Hi,

Python's ipaddress module is really good for working with network / IP
related thing.
I would like to use ipaddress.IPv4/v6Address in creating network
interfaces. Like a socket but i can't,
the internal things and some standard libraries doesn't support them (even
socket).

>>> server_address = (ipaddress.IPv4Address('127.0.0.1'), 1)
>>> sock.bind(server_address)
Traceback (most recent call last):
File "", line 1, in 
TypeError: str, bytes or bytearray expected, not IPv4Address


Proposal


The best solution we've got[0] is implementing something like PathLike[1]
for IP Address objects. For PoC[2]
we called it IPLike. It is a protocol (some kind of interface) created with
Abstract Base Classes and
it forces class to implement __ipaddr__ method (like __fspath__ method).
The __ipaddr__ method should return
string or bytes formed notation of IP address. It is dotted notation in
IPv4, long formed notation in IPv6.

The __ipaddr__ method is also an identifier for IPLike classes. IPLike
interface's subclasshook
checks the __ipaddr__ method of given class when issubclass called.

>>> class SomethingRepresentIP:
... def __ipaddr__(self) -> str:
... return '127.0.0.1'
...
>>> issubclass(SomethingRepresentIP, IPLike)
True
>>> isinstance(SomethingRepresentIP(), IPLike)
True

For PoC[2] we implented __ipaddr__ method to ipaddress._BaseAddress class.
It is parent class of IPv4/v6Adress.

>>> issubclass(IPv4Address, IPLike)
True
>>> IPv4Address('127.0.0.1').__ipaddr__()
'127.0.0.1'

Like os.fspath we need a general-purpose ip-getter. So we implemented
ipaddress.get_ipaddr(address) function.
When you give it an valid IP like object it returns the value of __ipaddr__
method.

>>> get_ipaddr(IPv4Address('127.0.0.1'))
'127.0.0.1'

When you give it string or bytes or integer object it calls
ipaddr.ip_address(address) and re-calls itself with
the value it got from ip_address. It helps get_ipaddr to convert
integer-formed / byte-formed IPs to string formed IPs
with a valid notation.

>>> get_ipaddr('127.0.0.1')
'127.0.0.1'
>>> get_ipaddr(3232235521)
'192.168.0.1'
>>> get_ipaddr(b'\xC0\xA8\x00\x01')
'192.168.0.1'

If you give any type else it raises TypeError. Also if you give it a IP
like object (an object that has __ipaddr__ method),
but the object's __ipaddr__ method returns something that is not
string/bytes it raises TypeError.

For PoC[2] i added socket to support IP like objects.

>>> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> server_address = (SomethingRepresentIP(), 1)
>>> sock.bind(server_address)
>>> sock.listen(1)
>>> conn, client = sock.accept()
>>> conn


I didn't implement a C API for ipaddress.get_ipaddr. The function is too
small for creating a new module, new header and the ipaddress library
currently has no
C-side module. If needed i can write a function inside the socketmodule.c .

Proof-of-Concept (Implementation)
=

The implementation can be found on github (DamlaAltun/cpython[2]) We have a
Abstract Base Class for IP Address protocol called IPLike.

(ipaddress.py[3])

class IPLike(abc.ABC):
"""
Abstract base class for IP (address) like objects.
"""

@abc.abstractmethod
def __ipaddr__(self):
"""
If it is a IPv4 return in the dotted form,
If it is a IPv6 return in the long form.
"""
raise NotImplementedError

@classmethod
def __subclasshook__(cls, subclass):
return hasattr(subclass, '__ipaddr__')

It has a abstract method __ipaddr__ and a subclasshook. The __ipaddr__
method returns ip representation of object.
The subclasshook checks for __ipaddr__ method.

Then there is a ip-getter called get_ipaddr

(ipaddress.py[4])

def get_ipaddr(address):
"""Return the representation of a ip address like object.

If bytes or string or int passed in, get_ipaddr converts it to
a IPv4/v6 Address and calls it self again for getting,
IP address of IPv4/v6 object. If a IPLike passed it uses IPLike
interface to get the value.
"""

if isinstance(address, (str, bytes, int)):
return get_ipaddr(ip_address(address))

addr_type = type(address)
try:
ip_addr = address.__ipaddr__()
except AttributeError:
if hasattr(address, '__ipaddr__'):
raise
else:
raise TypeError("expected str, bytes or ipaddress.IPLike
object, "
"not {}".format(addr_type.__name__))

if isinstance(ip_addr, (str, bytes)):
return ip_addr
else:
raise TypeError("expected {}.__ipaddr__() to return str or
bytes, "
"not {}".format(addr_type.__name__,