Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Oleg Nesterov]

On 04/03, Stephen J. Turnbull wrote:
>
> Nathaniel Smith writes:
>
>  > > Well, I'm afraid to contact this closed and not-for-mortals list,
>  > > not sure this very basic question should go there ;) perhaps you
>  > > are already a member, feel free to forward.
>  >
>  > core-mentorship is intended as a friendly place for folks who are
>  > starting to study CPython internals. I'm not sure where you got the
>  > impression that it's not-for-mortals but I suspect the people
>  > running it would like to know so they can fix it :-).
>
> I imagine Oleg got that impression because he tried to access the
> archives, which are closed to non-members.

Yes, so I came to conclusion that I should not bother this list with the
very simple questions.

> If it's something else, we would *definitely* like to know.

No, no, sorry for confusion! And thanks to all.

Oleg.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Nick Coghlan]

On 3 April 2017 at 05:23, Stephen J. Turnbull
 wrote:
> Nathaniel Smith writes:
>
>  > > Well, I'm afraid to contact this closed and not-for-mortals list,
>  > > not sure this very basic question should go there ;) perhaps you
>  > > are already a member, feel free to forward.
>  >
>  > core-mentorship is intended as a friendly place for folks who are
>  > starting to study CPython internals. I'm not sure where you got the
>  > impression that it's not-for-mortals but I suspect the people
>  > running it would like to know so they can fix it :-).
>
> I imagine Oleg got that impression because he tried to access the
> archives, which are closed to non-members.  I don't recall the
> rationale for that but there was one, I suspect that won't change.

It relates to these two points in the list's guidelines for participation:

=
* Statements made by core developers can be quoted outside of the list.
* Statements made by others can not be quoted outside the list without
explicit permission.
=

as well as this stated goal:

=
A major goal of this group is to help new contributors feel more
confident about participating in the results focused public
environments of the bug tracker, python-dev and python-ideas.
=

Lots of folks aren't comfortable with having their newbie questions
being archived permanently in reach of online search engines, so
rather than saying "You shouldn't be bothered by that", we instead
offer them a place to ask questions that is at least somewhat private.

There are folks at the other end of the spectrum that don't trust any
communications channel with deliberately closed archives ("What are
they hiding?"), but that's OK - different services are provided for
different audiences.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Stephen J. Turnbull]

Nathaniel Smith writes:

 > > Well, I'm afraid to contact this closed and not-for-mortals list,
 > > not sure this very basic question should go there ;) perhaps you
 > > are already a member, feel free to forward.
 > 
 > core-mentorship is intended as a friendly place for folks who are
 > starting to study CPython internals. I'm not sure where you got the
 > impression that it's not-for-mortals but I suspect the people
 > running it would like to know so they can fix it :-).

I imagine Oleg got that impression because he tried to access the
archives, which are closed to non-members.  I don't recall the
rationale for that but there was one, I suspect that won't change.
IMO, the listinfo
https://mail.python.org/mailman/listinfo/core-mentorship
makes it clear that developers new to Python core code (whether they
are new or advanced developers in other code bases) are welcome.

If it's something else, we would *definitely* like to know.

Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Martin Panter]

On 31 March 2017 at 05:22, Nathaniel Smith  wrote:
>>> On Mon, Mar 20, 2017 at 11:30 AM, Oleg Nesterov  wrote:
>>> > [Aborting "yield" in a "for" loop leaves a sub-generator open, but 
>>> > aborting "yield from" cleans up the sub-generator]
>
> In any case the short answer to your original question is that PEP 342
> says that generator finalization calls the generator's close() method,
> which throws a GeneratorExit into the generator, and PEP 380 says that
> as a special case, when a GeneratorExit is thrown into a yield from,
> then this is propagated by calling .close() on the yielded-from
> iterator (if such a method exists) and then re-raised in the original
> generator.

I think the Python documentation could be improved regarding this.
When I wrote the documentation for coroutine methods
,
I included details about the "close" and "throw" methods delegating to
inner iterators. I thought I was going to propose similar updates to
the generator documentation
,
but it seems I never got around to it. (In the mean time,

was added, to which this may also be relevant, but that is too
complicated for me.)

There is a parallel with another annoyance with Python generator
cleanup: . There are two ways you
can require generators to be used. With a simple generator, you can
partially iterate it and then throw it away without any special
cleaning up. But with more complex generators that "own" expensive
resources, it would be nice to produce a ResourceWarning I you forget
to clean them up.

With the "for / yield" case, the sub-generator is not cleaned up, so
if a resource-intensive sub-generator has to be cleaned up you have to
do it yourself. With "yield from", the cleanup is implicit and
unavoidable, which means you can't use it if you want keep the
sub-generator alive for later. But the automatic cleanup may be useful
in other cases.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Nathaniel Smith]

On Thu, Mar 30, 2017 at 11:05 AM, Oleg Nesterov  wrote:
> On 03/28, Eric Snow wrote:
>>
>> On Mon, Mar 20, 2017 at 11:30 AM, Oleg Nesterov  wrote:
>> > Hello,
>> >
>> > Let me first clarify, I do not claim this is a bug, I am trying to learn
>> > python and now I trying to understand yield-from.
>>
>> Given that you haven't gotten a response here,
>
> and this looks a bit strange. My question is simple, the implementation looks
> clear and straightforward, I am a bit surprised none of cpython devs bothered
> to reply.
>
>> you may want to take
>> this over to the core-mentors...@python.org list.
>
> Well, I'm afraid to contact this closed and not-for-mortals list, not sure
> this very basic question should go there ;) perhaps you are already a member,
> feel free to forward.

core-mentorship is intended as a friendly place for folks who are
starting to study CPython internals. I'm not sure where you got the
impression that it's not-for-mortals but I suspect the people running
it would like to know so they can fix it :-).

In any case the short answer to your original question is that PEP 342
says that generator finalization calls the generator's close() method,
which throws a GeneratorExit into the generator, and PEP 380 says that
as a special case, when a GeneratorExit is thrown into a yield from,
then this is propagated by calling .close() on the yielded-from
iterator (if such a method exists) and then re-raised in the original
generator.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Oleg Nesterov]

On 03/28, Eric Snow wrote:
>
> On Mon, Mar 20, 2017 at 11:30 AM, Oleg Nesterov  wrote:
> > Hello,
> >
> > Let me first clarify, I do not claim this is a bug, I am trying to learn
> > python and now I trying to understand yield-from.
>
> Given that you haven't gotten a response here,

and this looks a bit strange. My question is simple, the implementation looks
clear and straightforward, I am a bit surprised none of cpython devs bothered
to reply.

> you may want to take
> this over to the core-mentors...@python.org list.

Well, I'm afraid to contact this closed and not-for-mortals list, not sure
this very basic question should go there ;) perhaps you are already a member,
feel free to forward.

I downloaded micropython and it doesn't propagate .close() in this case. but
it seem to differ very much from cpython, not sure this matters at all.

Thanks,

Oleg.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Eric Snow]

On Mon, Mar 20, 2017 at 11:30 AM, Oleg Nesterov  wrote:
> Hello,
>
> Let me first clarify, I do not claim this is a bug, I am trying to learn
> python and now I trying to understand yield-from.

Given that you haven't gotten a response here, you may want to take
this over to the core-mentors...@python.org list.

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Oleg Nesterov]

Hello,

Let me first clarify, I do not claim this is a bug, I am trying to learn
python and now I trying to understand yield-from.

This simple test-case

g = (x for x in range(10))

def fg():
for x in g:
yield x

print(next(fg()))
print(next(g))

works as expected and prints:

0
1

However, if I change fg() to use yield-from

g = (x for x in range(10))

def fg():
yield from g

print(next(fg()))
print(next(g))

then next(g) raises StopIteration:

0
Traceback (most recent call last):
  File "/tmp/T.py", line 10, in 
print(next(g))
StopIteration

because g.close() is called by destructor of the object returned by fg().

To me this looks strange and confusing. I tried to google, and found
https://docs.python.org/3/whatsnew/3.3.html#pep-380 but it doesn't document
this behaviour. I understand that yield-from should propagate .close(), but
why _PyGen_Finalize() should send close() to the gi_yieldfrom object?

I applied the patch below just to verify that I actually understand what's
going on, and with this patch the 2nd test-case works as I'd expect. But
since I am very new to python I'd suspect that the code is fine and I simply
do not understand why it works this way.

So. could someone please explain the rationale behind this behaviour? And
probably update the docs should be updated?

Thanks in advance.

Oleg.
---

diff --git a/Objects/genobject.c b/Objects/genobject.c
index 24a1da6..d5152eb 100644
--- a/Objects/genobject.c
+++ b/Objects/genobject.c
@@ -6,6 +6,7 @@
 #include "opcode.h"
 
 static PyObject *gen_close(PyGenObject *, PyObject *);
+static PyObject *do_gen_close(PyGenObject *, PyObject *);
 static PyObject *async_gen_asend_new(PyAsyncGenObject *, PyObject *);
 static PyObject *async_gen_athrow_new(PyAsyncGenObject *, PyObject *);
 
@@ -71,7 +72,7 @@ _PyGen_Finalize(PyObject *self)
 }
 }
 else {
-res = gen_close(gen, NULL);
+res = do_gen_close(gen, NULL);
 }
 
 if (res == NULL) {
@@ -373,10 +374,9 @@ _PyGen_yf(PyGenObject *gen)
 }
 
 static PyObject *
-gen_close(PyGenObject *gen, PyObject *args)
+do_gen_close(PyGenObject *gen, PyObject *yf)
 {
 PyObject *retval;
-PyObject *yf = _PyGen_yf(gen);
 int err = 0;
 
 if (yf) {
@@ -407,6 +407,11 @@ gen_close(PyGenObject *gen, PyObject *args)
 return NULL;
 }
 
+static PyObject *
+gen_close(PyGenObject *gen, PyObject *args)
+{
+return do_gen_close(gen, _PyGen_yf(gen));
+}
 
 PyDoc_STRVAR(throw_doc,
 "throw(typ[,val[,tb]]) -> raise exception in generator,\n\

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com