[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Jason R. Coombs]

Jason R. Coombs  added the comment:

I believe I encountered this issue today on Python 2.7.14 
(https://ci.appveyor.com/project/jaraco/jaraco-windows/build/31/job/lenh5l4dcmj137b9).
 In this case, I have an iterable (in itertools.imap) that raises a TypeError 
when evaluated, not a generator.

The issue doesn't happen on Python 3, where 'map' is used instead of 
itertools.imap.

Does this patch need to be extended to support any iterable?

--
nosy: +jason.coombs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Martin Panter]

Changes by Martin Panter :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Roundup Robot]

Roundup Robot added the comment:

New changeset eef8f72ddb00 by Martin Panter in branch '2.7':
Issue #4806: Avoid masking TypeError when *-unpacking a generator
https://hg.python.org/cpython/rev/eef8f72ddb00

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Michel Desmoulin]

Michel Desmoulin added the comment:

We fixed our bug days ago, but I would have expected [*gen] to have triggered 
an exception before it even got to gather().

The real code was something like:

>>> l = (ensure_awaitable(callable_obj) for callable_obj in callable_list)
>>> gather(*l)

ensure_awaitable() is just using inspect to turn coroutine functions into 
coroutines, and wraps non coroutine callables with 
asyncio.coroutine(callable)().

In the end, ensure_awaitable did raise TypeError if the argument passed was not 
a callable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Roundup Robot]

Roundup Robot added the comment:

New changeset 1a8dc350962b by Martin Panter in branch '3.5':
Issue #4806: Avoid masking original TypeError in call with * unpacking
https://hg.python.org/cpython/rev/1a8dc350962b

New changeset 1261f5f6fc95 by Martin Panter in branch 'default':
Issue #4806: Merge * unpacking fix from 3.5
https://hg.python.org/cpython/rev/1261f5f6fc95

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Martin Panter]

Martin Panter added the comment:

I think this is ready to push for Python 3. Python 2 might need an extra 
Py_TPFLAGS_HAVE_ITER check, perhaps reusing some code from the Issue 5218 patch.

--
stage: patch review -> commit review
versions: +Python 3.6 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Samuel BOVÉE]

Samuel BOVÉE added the comment:

Up for this bug. I got this bug in my code and loose one hour to understand 
what happened. Please review Martin's patch !

--
nosy: +Samuel BOVÉE

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Martin Panter]

Martin Panter added the comment:

I am posting a new version of Daniel’s patch as 
issue4806-star-TypeError.v2.patch:

* Merged with recent “default” (3.5) branch
* Dropped the documentation fix, since revision a8aa918041c2 already fixed that 
in an identical fashion
* Changed to a “look before you leap” style check before attempting 
PySequence_Tuple(). This way we get to keep other valid errors such as “iter() 
returned non-iterator”.

I am comfortable with the code changes previous patch as well as mine. Checking 
the “tp_iter” field and PySequence_Check() is basically what happens behind the 
scenes, via PyObject_GetIter() anyway. So I think either patch is valid and a 
worthwhile improvement.

--
Added file: http://bugs.python.org/file38219/issue4806-star-TypeError.v2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[R. David Murray]

R. David Murray added the comment:

Sounds like we just need someone comfortable with modifying ceval.c to apply 
this ;)

--
nosy: +benjamin.peterson, r.david.murray
versions: +Python 3.4, Python 3.5 -Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[pfctdayelise]

Changes by pfctdayelise brianna.laug...@gmail.com:


--
nosy: +pfctdayelise

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Chris Kaynor]

Changes by Chris Kaynor ckay...@zindagigames.com:


--
nosy: +DragonFireCK

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Jon Brandvein]

Changes by Jon Brandvein jon.brandv...@gmail.com:


--
nosy: +brandj

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

#13904 is another dup, with patches to test and ceval. I asked that they be 
reloaded to this issue.

--
nosy: +ncoghlan
versions:  -Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Terry J. Reedy]

Changes by Terry J. Reedy tjre...@udel.edu:


--
nosy: +ron_adam

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

#5218 is a 4th duplicate, also with a patch to the ceval loop and test.

--
nosy: +georg.brandl, gpolo, pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

Sorry Martin, I see you already said that. Anyway, I closed other three in 
favor of this one.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Martin Panter]

Changes by Martin Panter vadmium...@gmail.com:


--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Martin Panter]

Martin Panter vadmium...@gmail.com added the comment:

I haven’t tried to understand what the patches do, but Issue 5218 looks like a 
very similar problem with a patch including a test case.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

#11944 is probably a duplicate of this and should be checked when this is fixed

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Daniel Urban]

Changes by Daniel Urban urban.dani...@gmail.com:


--
nosy: +durban

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Daniel Urban]

Daniel Urban urban.dani...@gmail.com added the comment:

I think the patch isn't entirely correct.  It uses PyIter_Check for detecting 
the case when an *iterable* raises TypeError, but that function actually checks 
for an *iterator*.  The check on the tp_iter member mentioned by Amaury Forgeot 
d'Arc probably would be better, but even that wouldn't detect every iterable: 
Its presence normally signals that the instances of this type are iterable 
(although sequences may be iterable without this function). 
(http://docs.python.org/dev/py3k/c-api/typeobj.html#PyTypeObject.tp_iter)  
(Apparently any object with a __getitem__ is iterable.  By the way, 
collections.abc.Iterable also doesn't detect this case.)

--
versions: +Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Daniel Urban]

Daniel Urban urban.dani...@gmail.com added the comment:

I'm attaching an updated patch. Instead !PyIter_Check() this patch checks for 
tp_iter == NULL  !PySequence_Check.  If this condition is false, 
PyObject_GetIter has a chance to succeed (and if it fails, we shouldn't mask 
the exception).  I also added more tests which show why the previous patch was 
incorrect.

--
Added file: http://bugs.python.org/file21028/issue4806.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Hagen Fürstenau]

Hagen Fürstenau hfuerste...@gmx.net added the comment:

IIUC, the only change you suggest for my patch is using finite iterable 
instead of sequence or iterable, right?

I've looked at the docs and there seems to be no precedent for finite 
iterable. I think it's just as obvious that the iterable has to yield a 
correct (and finite) number of parameters as the fact that 
list(itertools.count()) is a bad idea. So for consistency I would like to 
keep iterable.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

Ok, leave iterable as is. More important, you have two disjoint patches that I 
believe should be combined.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Hagen Fürstenau]

Hagen Fürstenau hfuerste...@gmx.net added the comment:

Attaching a combined patch against the current py3k.

--
Added file: http://bugs.python.org/file18404/combined.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

I verified with 3.1 the two OP cases and that generators work fine as long as 
they supply the correct number of values.

def f(x): return x
def broken(): return 1
print(f(*(broken() for x in (0,
# prints 1

Change (0,) to (0,1) the normal arg num mismatch message appears.

test_extcall tests version of Nothing() that follow both the old and new 
iteration protocol. It is possible that 'sequence' in meant in the broader 
sense of finite iterable rather that the narrow sense of
5.6. Sequence Types — str, bytes, bytearray, list, tuple, range

Since that is confusing, I would replace 'sequence' with 'finite iterable'. 
(Infinite iterables, obviously, are bad, just as in any other uncontrolled 
situation, such as a,*b = itertools.count().)

So, combine the correction and the suggestion above with original and diff 
against current trunk (py3k branch) if you can or at least 3.1.2.

--
nosy: +terry.reedy
versions: +Python 3.2 -Python 2.5, Python 2.6, Python 3.0

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Bruce Frederiksen]

Bruce Frederiksen dangy...@gmail.com added the comment:

I have also hit this error.  I went to report it but found it already entered 
(good news), but not resolved from nearly a year ago (bad news).

The error masked another bug that I had in my program and it took me quite 
awhile to figure out what the real problem was.

I use *generator arguments quite a lot, so was surprised to see the error.  So 
I, for one, can say that if you disable *generator arguments, you will break 
existing code.

If anybody cares, I have verified that this error also appears in Python2.5 and 
Python2.4 and am attempting to add python2.5 to the Versions list.  (And yes, 
*generators were allowed in Python2.4!)

Is this headed for resolution?  Progress on it seems to have stalled nearly a 
year ago.  Can I vote to revive this?

--
nosy: +dangyogi
versions: +Python 2.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Alexander Belopolsky]

Changes by Alexander Belopolsky belopol...@users.sourceforge.net:


--
keywords: +needs review
stage:  - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Hagen Fürstenau]

Hagen Fürstenau hfuerste...@gmx.net added the comment:

I added a simple check for iterables. This is not very elegant, but
performance is only affected in the case of an exception. Patch and
corresponsing test are attached as TypeError.patch.

As I pointed out above, the actual error message must be a sequence is
also inconsistent with the implementation (and tests) which allows any
kind of iterable. The attached and independent patch
message_and_docs.patch changes this to must be an iterable and
corrects docs and tests accordingly.

--
keywords: +patch
versions: +Python 2.7, Python 3.1
Added file: http://bugs.python.org/file14166/TypeError.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

This patch leaks a reference on each call to PyObject_GetIter(). And I'm
not sure it is a good idea to call this function again: it may be very
expensive! I'd prefer a simple check on the tp_iter member.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Hagen Fürstenau]

Hagen Fürstenau hfuerste...@gmx.net added the comment:

Sorry, I had meant to use PyIter_Check instead of PyObject_GetIter.
Don't know why I didn't do so... ;-)

I corrected the patch.

--
Added file: http://bugs.python.org/file14169/TypeError2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Hagen Fürstenau]

Changes by Hagen Fürstenau hfuerste...@gmx.net:


Removed file: http://bugs.python.org/file14166/TypeError.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Kenneth Arnold]

Kenneth Arnold kenneth.arn...@gmail.com added the comment:

I can confirm that (a) this exact behavior is happening and (b) it quite
confused me (most of the time it works!). What would be a good
TypeError? I'd vote for generators to be explicitly supported even if it
required a special case. Thanks!

--
nosy: +kcarnold

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Hagen Fürstenau]

Hagen Fürstenau hfuerste...@gmx.net added the comment:

I'm getting confused about whether it's actually desired behaviour that
generators can be star arguments.

The error message seems to say it's not: argument after * must be a
sequence. The docs seem to agree: If the syntax *expression appears in
the function call, expression must evaluate to a sequence. However
test_extcall specifically tests function calls with (non-sequence)
iterables as star arguments.

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Hagen Fürstenau]

New submission from Hagen Fürstenau hfuerste...@gmx.net:

If we call some function f with a generator as star argument and this
generator raises a TypeError, we get the following exception:

 def f(x): pass
... 
 def broken(): raise TypeError
... 
 f(*(broken() for x in (0,)))
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: f() argument after * must be a sequence, not generator

This is a misleading error message, as it's usually no problem to use a
generator as a star argument. Even just replacing the TypeError by some
other exception leads to the expected result, i.e. the exception gets
correctly propagated.

The problem seems to be around line 3710 of Python/ceval.c where the
generator is converted to a tuple. If this conversion raises a
TypeError, then the error message is replaced, which will mask any
TypeError raised by the generator.

I'm not sure how to solve this. We probably can't distinguish a good
TypeError from a bad TypeError at this point, so we might have to make
a special case for the conversion of generators.

--
components: Interpreter Core
messages: 78788
nosy: hagen
severity: normal
status: open
title: Function calls taking a generator as star argument can mask TypeErrors 
in the generator
type: behavior
versions: Python 2.6, Python 3.0

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

The issue1615 has the same kind of problems: an AttributeError is masked
by another one.

--
nosy: +amaury.forgeotdarc

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4806
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com