Re: [Python-ideas] make Connections iterable

[Nathaniel Smith]

On Mon, Jan 8, 2018 at 7:27 PM, Amit Green  wrote:

> An argument against this API, is that any caller of recv should be doing
> error handling (i.e.: catching exceptions from the socket).
>

It's still not entirely clear, but I'm pretty sure this thread is talking
about multiprocessing.Connection objects, which don't have anything to do
with sockets. (I think. They might use sockets internally on some
platforms.)

The only documented error from multiprocessing.Connection.recv is EOFError,
which is basically equivalent to a StopIteration.

I'm surprised that multiprocessing.Connection isn't iterable -- it seems
like an obvious oversight.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] make Connections iterable

[Amit Green]

On Mon, Jan 8, 2018 at 10:34 PM, Nick Coghlan  wrote

> It could be useful to include a recipe in the documentation that shows a
> generator with suitable error handling (taking the generic connection
> errors and adapting them to app specific ones) while also showing how to
> adapt the connection to the iterator protocol, though.
>

Agreed +1.

This would be great for three reasons:

   - Error handling should be as close to the source of errors as possible
   (i.e.: in the iterator).
   - Converting error's and adapting them is a great idea;
   - Mostly: Any documentation that shows people how to do better error
   handling, is always welcome.



Virus-free.
www.avast.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] make Connections iterable

[Nick Coghlan]

On 9 January 2018 at 13:27, Amit Green  wrote:

> An argument against this API, is that any caller of recv should be doing
> error handling (i.e.: catching exceptions from the socket).
>
> Changing into an iterator makes it less likely that error handling will be
> properly coded, and makes the error handling more obscure.
>
> Thus although the API would make the code more readable for the [wrong
> case] of not handling errors; the real issue is that it would make the code
> more obscure for the proper case of error handling.
>
> We should focus on the proper use case: using recv with error handling &
> thus not add this API.
>

It could be useful to include a recipe in the documentation that shows a
generator with suitable error handling (taking the generic connection
errors and adapting them to app specific ones) while also showing how to
adapt the connection to the iterator protocol, though.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] make Connections iterable

[Amit Green]

An argument against this API, is that any caller of recv should be doing
error handling (i.e.: catching exceptions from the socket).

Changing into an iterator makes it less likely that error handling will be
properly coded, and makes the error handling more obscure.

Thus although the API would make the code more readable for the [wrong
case] of not handling errors; the real issue is that it would make the code
more obscure for the proper case of error handling.

We should focus on the proper use case: using recv with error handling &
thus not add this API.


Virus-free.
www.avast.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Mon, Jan 8, 2018 at 10:05 PM, Oscar Smith  wrote:

> The arguments for including this API is that it allows easy iteration over
> the results of a connection allowing it to be used with any of the features
> of itertools or any other library accepting iterables. recv is only used in
> places where the iterable protocol could be used, so it makes sense for
> consistency to use the API shared by the rest of Python.
>
> Oscar Smith
>
> On Mon, Jan 8, 2018 at 7:25 PM, Steven D'Aprano 
> wrote:
>
>> On Mon, Jan 08, 2018 at 10:17:30AM -0600, Oscar Smith wrote:
>> > I am currently working on a program where it would be really useful if a
>> > connection had a __next__ method, because then it would be much easier
>> to
>> > iterate over.
>>
>> What sort of connection are you referring to?
>>
>>
>> > It would just be an alias to recv, but would allow you to do
>> > things like merging the results of connections using heapq.merge that
>> > currently are highly non-trivial to accomplish.
>>
>> This gives you an iterator which repeatedly calls connection.recv until
>> it raises a FooException, then ends.
>>
>> def conn_iter(connection):
>> try:
>> while True:
>> yield connection.recv()
>> except FooException:  # FIXME -- what does recv actually raise?
>> return
>>
>> Doesn't seem "highly non-trivial" to me. Have I missed something?
>>
>>
>> > Is there a reason this API isn't supported?
>>
>> You are asking the wrong question. Adding APIs isn't "default allow",
>> where there has to be a reason to *not* support it otherwise it gets
>> added. It is "default deny" -- there has to be a good reason to add it,
>> otherwise it gets left out. YAGNI is an excellent design principle, as
>> it is easier to add a useful API later, than to remove an unnecessary or
>> poorly designed one.
>>
>> So the question needs to be:
>>
>> "Is this a good enough reason to support this API?"
>>
>> Maybe, maybe not. Not every trivial wrapper function needs to be a
>> method.
>>
>> But perhaps this is an exception: perhaps iterability is such a common
>> and useful API for connections that it should be added, for the same
>> reason that files are iterable.
>>
>> Care to elaborate on why this would be useful and why the generator I
>> showed above isn't satisfactory?
>>
>> --
>> Steve
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] make Connections iterable

[Oscar Smith]

The arguments for including this API is that it allows easy iteration over
the results of a connection allowing it to be used with any of the features
of itertools or any other library accepting iterables. recv is only used in
places where the iterable protocol could be used, so it makes sense for
consistency to use the API shared by the rest of Python.

Oscar Smith

On Mon, Jan 8, 2018 at 7:25 PM, Steven D'Aprano  wrote:

> On Mon, Jan 08, 2018 at 10:17:30AM -0600, Oscar Smith wrote:
> > I am currently working on a program where it would be really useful if a
> > connection had a __next__ method, because then it would be much easier to
> > iterate over.
>
> What sort of connection are you referring to?
>
>
> > It would just be an alias to recv, but would allow you to do
> > things like merging the results of connections using heapq.merge that
> > currently are highly non-trivial to accomplish.
>
> This gives you an iterator which repeatedly calls connection.recv until
> it raises a FooException, then ends.
>
> def conn_iter(connection):
> try:
> while True:
> yield connection.recv()
> except FooException:  # FIXME -- what does recv actually raise?
> return
>
> Doesn't seem "highly non-trivial" to me. Have I missed something?
>
>
> > Is there a reason this API isn't supported?
>
> You are asking the wrong question. Adding APIs isn't "default allow",
> where there has to be a reason to *not* support it otherwise it gets
> added. It is "default deny" -- there has to be a good reason to add it,
> otherwise it gets left out. YAGNI is an excellent design principle, as
> it is easier to add a useful API later, than to remove an unnecessary or
> poorly designed one.
>
> So the question needs to be:
>
> "Is this a good enough reason to support this API?"
>
> Maybe, maybe not. Not every trivial wrapper function needs to be a
> method.
>
> But perhaps this is an exception: perhaps iterability is such a common
> and useful API for connections that it should be added, for the same
> reason that files are iterable.
>
> Care to elaborate on why this would be useful and why the generator I
> showed above isn't satisfactory?
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] make Connections iterable

[Steven D'Aprano]

On Mon, Jan 08, 2018 at 10:17:30AM -0600, Oscar Smith wrote:
> I am currently working on a program where it would be really useful if a
> connection had a __next__ method, because then it would be much easier to
> iterate over.

What sort of connection are you referring to?


> It would just be an alias to recv, but would allow you to do
> things like merging the results of connections using heapq.merge that
> currently are highly non-trivial to accomplish.

This gives you an iterator which repeatedly calls connection.recv until 
it raises a FooException, then ends.

def conn_iter(connection):
try:
while True:
yield connection.recv()
except FooException:  # FIXME -- what does recv actually raise?
return

Doesn't seem "highly non-trivial" to me. Have I missed something?


> Is there a reason this API isn't supported?

You are asking the wrong question. Adding APIs isn't "default allow", 
where there has to be a reason to *not* support it otherwise it gets 
added. It is "default deny" -- there has to be a good reason to add it, 
otherwise it gets left out. YAGNI is an excellent design principle, as 
it is easier to add a useful API later, than to remove an unnecessary or 
poorly designed one.

So the question needs to be:

"Is this a good enough reason to support this API?"

Maybe, maybe not. Not every trivial wrapper function needs to be a 
method.

But perhaps this is an exception: perhaps iterability is such a common 
and useful API for connections that it should be added, for the same 
reason that files are iterable.

Care to elaborate on why this would be useful and why the generator I 
showed above isn't satisfactory?

-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] make Connections iterable

[Terry Reedy]

On 1/8/2018 11:17 AM, Oscar Smith wrote:
I am currently working on a program where it would be really useful if a 
connection had a __next__ method, because then it would be much easier 
to iterate over. It would just be an alias to recv, but would allow you 

> to do things like merging the results of connections using heapq.merge
> that currently are highly non-trivial to accomplish.

The reference to recv says that you must be talking about 
multiprocessing.Connection rather than sqlite3.Connection.


Since recv raises EORError when done, an alias does not work.  Try the 
following generator adaptor.


def connect_gen(connection):
try:
while True:
yield connection.recv()
except EOFError:
pass

You could make the above the .__iter__ method of a MyConnecton subclass.

--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] make Connections iterable

[Oscar Smith]

I am currently working on a program where it would be really useful if a
connection had a __next__ method, because then it would be much easier to
iterate over. It would just be an alias to recv, but would allow you to do
things like merging the results of connections using heapq.merge that
currently are highly non-trivial to accomplish. Is there a reason this API
isn't supported?
Oscar Smith
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/