[issue16202] sys.path[0] security issues

[Nick Coghlan]

Nick Coghlan added the comment:

Also, what's up with that weird fallback code in distutils? When is 
tempfile.mkdtemp ever missing?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16216] Arithmetic operations with NULL

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The Standard is not guarantee that result of converting null pointer to integer 
is zero or even it is a even number. I remember the ancient C implementation, 
where it was not so (but Python is not supports these platform).

Of course, such a method of obtaining aligned pointers is one great trick. I'm 
not sure whether to worry about that.

--
nosy: +mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16201] socket.gethostbyname incorrectly parses ip

[Michele Orrù]

Michele Orrù added the comment:

>
>> Buggy due to the use of scanf at Modueles/socketmodule.c:868
>
> I don't think so. The following test fails because sscanf() returns 5
> instead of 4:
You are right, sorry for didn't notice. 


> If you consider that '192.168.1.1  ' is an invalid name, you should
> report the issue to the vendor of the C library of your OS.
Yep, I'm going to ask on #glibc if there is any specific reason for accepting 
trailing spaces. But this concerns the specific underlying implementation if 
getaddrinfo(), so IMO we should just update the int overflow issue for now, 
maintaining the coherence with other utilities having the same "problem". For 
this, I am attaching a new patch which which just do not test for trailing 
whitespaces. 

-- 
ù

--
Added file: http://bugs.python.org/file27549/issue16201.1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16201] socket.gethostbyname incorrectly parses ip

[Michele Orrù]

Michele Orrù added the comment:

Reviewed with Ezio.

--
Added file: http://bugs.python.org/file27550/issue16201.2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15936] Add link from os.urandom to random.SystemRandom

[Mike Hoy]

Changes by Mike Hoy :


--
nosy: +mikehoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16202] sys.path[0] security issues

[Volker Braun]

Volker Braun added the comment:

> When is tempfile.mkdtemp ever missing

It was added in Python 2.3, in the dark ages before that there was only 
tempfile.mktemp. Though I guess we can remove the fallback now...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16206] dict() docs should display multiple signatures

[Roundup Robot]

Roundup Robot added the comment:

New changeset ed6da2a8361c by Chris Jerdonek in branch '3.2':
Issue #16206: Improve the documentation of the dict constructor.
http://hg.python.org/cpython/rev/ed6da2a8361c

New changeset 02de13d69149 by Chris Jerdonek in branch '3.3':
Issue #16206: Merge dict documentation improvements from 3.2.
http://hg.python.org/cpython/rev/02de13d69149

New changeset 392e09ba9feb by Chris Jerdonek in branch 'default':
Issue #16206: Merge dict documentation improvements from 3.3.
http://hg.python.org/cpython/rev/392e09ba9feb

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16206] dict() docs should display multiple signatures

[Roundup Robot]

Roundup Robot added the comment:

New changeset 5fc6f47974db by Chris Jerdonek in branch '2.7':
Issue #16206: Backport dict documentation improvements from 3.2.
http://hg.python.org/cpython/rev/5fc6f47974db

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16206] dict() docs should display multiple signatures

[Chris Jerdonek]

Changes by Chris Jerdonek :


--
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16215] Possible double memory free in str.replace

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16217] Tracebacks are unnecessarily verbose when using 'python -m'

[Matthew Woodcraft]

New submission from Matthew Woodcraft:

If I run my code using 'python -m' and there is an unhandled exception, the 
tracebacks include lines from runpy.py (and now sometimes from 
importlib._bootstrap) which don't provide useful information, and tend to 
overwhelm the valuable part of the traceback.

I suppose this is in some sense a regression from Python 2.4.


echo fail > fail.py
python3.3 -m fail

python -m fail
Traceback (most recent call last):
  File "/usr/lib/python3.3/runpy.py", line 160, in _run_module_as_main
"__main__", fname, loader, pkg_name)
  File "/usr/lib/python3.3/runpy.py", line 73, in _run_code
exec(code, run_globals)
  File "./fail.py", line 2, in 
fail
NameError: name 'fail' is not defined

Here I think it would be better to omit the first two traceback entries.


Syntax errors are worse:
echo syntax error > fail.py
python3.3 -m fail

Traceback (most recent call last):
  File "/usr/lib/python3.3/runpy.py", line 140, in _run_module_as_main
mod_name, loader, code, fname = _get_module_details(mod_name)
  File "/usr/lib/python3.3/runpy.py", line 114, in _get_module_details
code = loader.get_code(mod_name)
  File "", line 981, in get_code
  File "", line 313, in _call_with_frames_removed
  File "./fail.py", line 1
syntax error

Here there are four traceback entries which could be omitted.

--
messages: 172806
nosy: mattheww
priority: normal
severity: normal
status: open
title: Tracebacks are unnecessarily verbose when using 'python -m'
type: enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16217] Tracebacks are unnecessarily verbose when using 'python -m'

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16061] performance regression in string replace for 3.3

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16218] Python launcher does not support non ascii characters

[Turn]

New submission from Turn:

If there are non ASCII character in the py.exe arguments, the execution will 
fail. The script file name or path may contain non ASCII characters.

--
components: Windows
messages: 172807
nosy: turncc
priority: normal
severity: normal
status: open
title: Python launcher does not support non ascii characters
type: crash
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16218] Python launcher does not support non ascii characters

[Jeremy Kloth]

Changes by Jeremy Kloth :


--
nosy: +jkloth

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16217] Tracebacks are unnecessarily verbose when using 'python -m'

[Nick Coghlan]

Nick Coghlan added the comment:

Agreed. We do have a mechanism in place to deal with this specifically for 
importlib in 3.3 (which is why the first traceback is cleaner, although it's 
not triggering in the SyntaxError case for some reason), but it makes sense to 
hide the noise from runpy as well.

Flagging 3.3 for the moment, but if the change is too intrusive it will end up 
being 3.4 only.

--
assignee:  -> ncoghlan
nosy: +brett.cannon
versions: +Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16106] antigravity tests

[Ramchandra Apte]

Ramchandra Apte added the comment:

@Antoine Aw.. no funny bone in you..

--
nosy: +ramchandra.apte

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8402] glob returns empty list with "[" character in the folder name

[Michele Orrù]

Michele Orrù added the comment:

The attached patch adds support for '\\' escaping to fnmatch, and consequently 
to glob.

--
keywords: +patch
nosy: +maker
versions: +Python 3.4 -Python 3.2
Added file: http://bugs.python.org/file27551/issue8402.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15298] _sysconfigdata is generated in srcdir, not builddir

[Trent Nelson]

Changes by Trent Nelson :


--
nosy: +trent

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16145] Abort in _csv module

[Andrew Svetlov]

Andrew Svetlov added the comment:

LGTM

--
nosy: +asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16202] sys.path[0] security issues

[Eric Snow]

Eric Snow added the comment:

> For 3.4, I plan to have a look at the organically-grown-over-time mess
> that is CPython's current interpreter initialisation system and see if I
> can figure out something a bit more sane and easier to configure/control
> (especially when embedding Python in a larger application) :P

I'm totally on board with any effort in this regard.  Sign me up if you need a 
hand.

--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15298] _sysconfigdata is generated in srcdir, not builddir

[Trent Nelson]

Trent Nelson added the comment:

The previous 'alt_sysconfigdata.patch' didn't apply cleanly to 3.3/3.x.  I've 
attached an updated patch that does.

Any objections to applying this to 3.3 -> 3.x now that the freeze is over?

--
versions: +Python 3.4
Added file: http://bugs.python.org/file27552/issue15298-alt_sysconfigdata2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14774] _sysconfigdata.py doesn't support multiple build configurations

[Trent Nelson]

Trent Nelson added the comment:

I'm pretty sure this is a duplicate of http://bugs.python.org/issue15298 (which 
should be fixed shortly).

Feel free to re-open if I'm wrong.

--
nosy: +trent
resolution:  -> duplicate
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16219] segfault when using xml.etree.ElementTree.XMLTreeBuilder

[Scott Maxwell]

New submission from Scott Maxwell:

I upgraded the meld3 module (used by Supervisord) to work on Py3K and 
discovered this segfault. It happens every time. I have seen this on the 
pre-built Mac 3.3.0 and a source-built 3.3.0 on Linux. It does not occur in 
3.2.2. It appears to happen in native code so I can pinpoint the exact location 
of the crash. But it is easy to reproduce.

It is easy to reproduce. Just grab meld3.py and test_meld3.py from my GitHub 
and run test_meld3. Files are here:
https://raw.github.com/scottkmaxwell/meld3/master/meld3/meld3.py
https://raw.github.com/scottkmaxwell/meld3/master/meld3/test_meld3.py

On just clone my depot.

--
components: Library (Lib)
messages: 172815
nosy: scottmax
priority: normal
severity: normal
status: open
title: segfault when using xml.etree.ElementTree.XMLTreeBuilder
type: crash
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16203] Proposal: add re.fullmatch() method

[Matthew Barnett]

Matthew Barnett added the comment:

Tim, my point is that if the MULTILINE flag happens to be turned on, '$' won't 
just match at the end of the string (or slice), it'll also match at a newline, 
so wrapping the pattern in (?:...)$ in that case could give the wrong answer, 
but wrapping it in (?:...)\Z would give the right answer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15833] most failures to write byte-compiled file no longer suppressed

[Trent Nelson]

Trent Nelson added the comment:

FWIW, I just ran into the following on Solaris 10:

% dbx python
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.9' in your .dbxrc
Reading python
Reading ld.so.1
Reading libsocket.so.1
Reading libnsl.so.1
Reading libintl.so.1
Reading librt.so.1
Reading libdl.so.1
Reading libsendfile.so.1
Reading libm.so.2
Reading libthread.so.1
Reading libc.so.1
Reading libaio.so.1
Reading libmd.so.1
(dbx) run
Running: python 
(process id 17819)
Reading libc_psr.so.1
Could not find platform dependent libraries 
Consider setting $PYTHONHOME to [:]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
Traceback (most recent call last):
  File "", line 1558, in _find_and_load
  File "", line 1525, in _find_and_load_unlocked
  File "", line 586, in _check_name_wrapper
  File "", line 1023, in load_module
  File "", line 1004, in load_module
  File "", line 562, in module_for_loader_wrapper
  File "", line 854, in _load_module
  File "", line 990, in get_code
  File "", line 1051, in _cache_bytecode
  File "", line 1065, in set_data
OSError: [Errno 30] Read-only file system: 
'/home/cpython/nfs/src/3.x/Lib/encodings/__pycache__'
t@1 (l@1) signal ABRT (Abort) in __lwp_kill at 0x7e2dc2c0
0x7e2dc2c0: __lwp_kill+0x0008:  bcc,a,pt  %icc,__lwp_kill+0x18  ! 
0x7e2dc2d0
Current function is Py_FatalError
 2360   abort();
(dbx) q 
q: not found
(dbx) quit


Testing the 'import_error.diff' patch now...

--
nosy: +trent

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16219] segfault when using xml.etree.ElementTree.XMLTreeBuilder

[Christian Heimes]

Christian Heimes added the comment:

Thanks for your report.It's a know issue and has already been fixed in HG. 
#16089 contains a patch if you like to give it a shot.

--
nosy: +christian.heimes
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16203] Proposal: add re.fullmatch() method

[Antoine Pitrou]

Antoine Pitrou added the comment:

> Tim, my point is that if the MULTILINE flag happens to be turned on,
> '$' won't just match at the end of the string (or slice), it'll also
> match at a newline, so wrapping the pattern in (?:...)$ in that case
> could give the wrong answer, but wrapping it in (?:...)\Z would give
> the right answer.

This means Tim and Guido are right that a dedicated fullmatch() method
is desireable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15833] most failures to write byte-compiled file no longer suppressed

[Trent Nelson]

Trent Nelson added the comment:

That... didn't work.  Also applied rpetrov's patch and that made no difference:

% ./python Could not find platform dependent libraries 
Consider setting $PYTHONHOME to [:]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
Traceback (most recent call last):
  File "", line 1558, in _find_and_load
  File "", line 1525, in _find_and_load_unlocked
  File "", line 586, in _check_name_wrapper
  File "", line 1023, in load_module
  File "", line 1004, in load_module
  File "", line 562, in module_for_loader_wrapper
  File "", line 854, in _load_module
  File "", line 990, in get_code
  File "", line 1051, in _cache_bytecode
  File "", line 1065, in set_data
OSError: [Errno 30] Read-only file system: 
'/home/cpython/nfs/src/3.x/Lib/encodings/__pycache__'
zsh: IOT instruction (core dumped)  ./python

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16061] performance regression in string replace for 3.3

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

After much experimentation, I suggest the new patch.

Benchmark results (time of replacing 1 of n character (ch1 to ch2) in 10-
char string).

Py3.2Py3.3patch  n ch1 ch2 fill

231 (-13%)   3025 (-93%)  2001 'a' 'b' 'c'
626 (-18%)   2035 (-75%)  5112 'a' 'b' 'c'
444 (-26%)   957 (-66%)   3275 'a' 'b' 'c'
349 (-30%)   530 (-54%)   24310 'a' 'b' 'c'
306 (-40%)   300 (-38%)   18520 'a' 'b' 'c'
280 (-54%)   169 (-23%)   13050 'a' 'b' 'c'
273 (-62%)   123 (-15%)   105100 'a' 'b' 'c'
265 (-70%)   82 (-4%) 79 1000 'a' 'b' 'c'
230 (+4%)3012 (-92%)  2391 '\u010a' '\u010b' '\u010c'
624 (-17%)   1907 (-73%)  5182 '\u010a' '\u010b' '\u010c'
442 (-16%)   962 (-62%)   3705 '\u010a' '\u010b' '\u010c'
347 (-5%)566 (-42%)   33010 '\u010a' '\u010b' '\u010c'
305 (-10%)   357 (-23%)   27520 '\u010a' '\u010b' '\u010c'
285 (-26%)   241 (-12%)   21250 '\u010a' '\u010b' '\u010c'
280 (-33%)   190 (-2%)187100 '\u010a' '\u010b' '\u010c'
263 (-41%)   170 (-8%)1561000 '\u010a' '\u010b' '\u010c'
3355 (-85%)  3309 (-85%)  4981 '\U0001000a' '\U0001000b' '\U0001000c'
2290 (-65%)  2267 (-65%)  8002 '\U0001000a' '\U0001000b' '\U0001000c'
1598 (-62%)  1279 (-52%)  6125 '\U0001000a' '\U0001000b' '\U0001000c'
1313 (-60%)  950 (-45%)   51910 '\U0001000a' '\U0001000b' '\U0001000c'
1195 (-61%)  824 (-44%)   46420 '\U0001000a' '\U0001000b' '\U0001000c'
1055 (-59%)  640 (-32%)   43450 '\U0001000a' '\U0001000b' '\U0001000c'
982 (-55%)   549 (-20%)   439100 '\U0001000a' '\U0001000b' '\U0001000c'
941 (-56%)   473 (-12%)   4171000 '\U0001000a' '\U0001000b' '\U0001000c'

On other platforms other numbers are possible. Especially I'm interested in 
the results on Windows and on 64-bit. For the test I used the script 
replacebench2.py, and compared the results with the help of script 
https://bitbucket.org/storchaka/cpython-stuff/raw/default/bench/bench-diff.py 
.

--
Added file: http://bugs.python.org/file27553/str_replace_1char.patch

___
Python tracker 

___diff -r 6fb1fb550319 Objects/unicodeobject.c
--- a/Objects/unicodeobject.c   Thu Oct 11 18:59:34 2012 -0700
+++ b/Objects/unicodeobject.c   Sat Oct 13 20:19:47 2012 +0300
@@ -9881,6 +9881,80 @@
 return 0;
 }
 
+static void
+replace_1char_inplace(PyObject *u, Py_ssize_t pos, Py_UCS4 u1, Py_UCS4 u2, 
Py_ssize_t maxcount)
+{
+int kind = PyUnicode_KIND(u);
+void *data = PyUnicode_DATA(u);
+Py_ssize_t len = PyUnicode_GET_LENGTH(u);
+if (kind == PyUnicode_1BYTE_KIND) {
+Py_UCS1 *s = (Py_UCS1 *)data;
+Py_UCS1 *end = s + len;
+s += pos;
+*s = u2;
+while (--maxcount && ++s != end) {
+if (*s != u1) {
+int attempts = 10;
+while (1) {
+if (++s == end)
+return;
+if (*s == u1)
+break;
+if (!--attempts) {
+s++;
+s = memchr(s, u1, end - s);
+if (s == NULL)
+return;
+break;
+}
+}
+}
+*s = u2;
+}
+}
+else if (kind == PyUnicode_2BYTE_KIND) {
+Py_UCS2 *s = (Py_UCS2 *)data;
+Py_UCS2 *end = s + len;
+s += pos;
+*s = u2;
+while (--maxcount && ++s != end) {
+if (*s != u1) {
+int attempts = 10;
+while (1) {
+if (++s == end)
+return;
+if (*s == u1)
+break;
+if (!--attempts) {
+Py_ssize_t i;
+s++;
+i = findchar(s, PyUnicode_2BYTE_KIND, end - s, u1, 1);
+if (i < 0)
+return;
+s += i;
+break;
+}
+}
+}
+*s = u2;
+}
+}
+else {
+Py_UCS4 *s = (Py_UCS4 *)data;
+Py_UCS4 *end = s + len;
+s += pos;
+while (1) {
+*s = u2;
+if (!--maxcount)
+return;
+do {
+if (++s == end)
+return;
+} while (*s != u1);
+}
+}
+}
+
 static PyObject *
 replace(PyObject *self, PyObject *str1,
 PyObject *str2, Py_ssize_t maxcount)
@@ -9897,7 +9971,7 @@
 Py_ssize_t len1 = PyUnicode_GET_LENGTH(str1);
 Py_ssize_t len2 = PyUnicode_GET_LENGTH(str2);
 int mayshrink;
-Py_UCS4 maxchar, maxchar_str2;
+Py_UCS4 maxchar, maxchar_str1, maxchar_str2;
 
 if (maxcount < 0)
 maxco

[issue16203] Proposal: add re.fullmatch() method

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Definitely this is not easy issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15833] most failures to write byte-compiled file no longer suppressed

[Trent Nelson]

Trent Nelson added the comment:

Charles: your patch is fine.  +1.

My Solaris failures can be traced back to http://bugs.python.org/issue15819.  
Sorry for the noise.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15958] bytes.join() should allow arbitrary buffer objects

[Antoine Pitrou]

Antoine Pitrou added the comment:

Here is a patch with tests.

--
Added file: http://bugs.python.org/file27554/bytes_join_buffers.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16201] socket.gethostbyname incorrectly parses ip

[Antoine Pitrou]

Antoine Pitrou added the comment:

I don't think uint32_t exists everywhere, you should use "unsigned int" instead.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16212] mmap() dumps core upon resizing the underlying file

[Vladimir Ushakov]

Vladimir Ushakov added the comment:

I know how to avoid the problem in my case, the bug does not really affect me. 
I posted it because I thought that the possibility to crash the interpreter is 
something to be avoided at all costs.

I've found a few examples of handling non-restartable signals with longjmps, 
but not that familiar with the codebase to estimate how reliably it can be done 
on all supported platforms. I don't really know how this code would behave, 
say, on Windows.

I'm gonna do some research on the topic when I have a little more time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16160] subclassing types.SimpleNamespace does not work

[Eric Snow]

Eric Snow added the comment:

Am I good to commit this?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15958] bytes.join() should allow arbitrary buffer objects

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Patch LGTM, however...

$ ./python -m timeit -s "a=[b'a']*10"  "b','.join(a)"

Vanilla: 3.69 msec per loop
Patched: 11.6 msec per loop

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15958] bytes.join() should allow arbitrary buffer objects

[Antoine Pitrou]

Antoine Pitrou added the comment:

> Patch LGTM, however...
> 
> $ ./python -m timeit -s "a=[b'a']*10"  "b','.join(a)"
> 
> Vanilla: 3.69 msec per loop
> Patched: 11.6 msec per loop

True. It is a bit of a pathological case, though.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16220] wsgiref does not call close() on iterable response when client

[Brent Tubbs]

New submission from Brent Tubbs:

When a WSGI application returns an iterable that has a .close() method, the 
server is supposed to call that method once the request has finished.  The 
wsgiref server does not do this when a client disconnects from a streaming 
response.

The attached script allows testing the .close() behavior of various wsgi 
servers (wsgiref, cherrypy, gevent, werkzeug, and gunicorn).  wsgiref is the 
only one of the tested implementations that does not call .close().

--
components: Library (Lib)
files: sleepy_app.py
messages: 172830
nosy: Brent.Tubbs
priority: normal
severity: normal
status: open
title: wsgiref does not call close() on iterable response when client
versions: Python 2.7
Added file: http://bugs.python.org/file27555/sleepy_app.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16220] wsgiref does not call close() on iterable response

[Brent Tubbs]

Changes by Brent Tubbs :


--
title: wsgiref does not call close() on iterable response when client -> 
wsgiref does not call close() on iterable response

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16220] wsgiref does not call close() on iterable response

[Antoine Pitrou]

Antoine Pitrou added the comment:

This sounds like a reasonable expectation. Do you want to provide a patch?

--
nosy: +pitrou, pje
stage:  -> needs patch
type:  -> behavior
versions: +Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13538] Improve doc for str(bytesobject)

[Chris Jerdonek]

Chris Jerdonek added the comment:

Attaching a proposed patch along the lines suggested by Éric.

--
keywords: +patch
versions: +Python 3.4
Added file: http://bugs.python.org/file27556/issue-13538-1-default.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14214] test_concurrent_futures hangs

[Nadeem Vawda]

Changes by Nadeem Vawda :


--
resolution:  -> works for me
stage: needs patch -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14120] ARM Ubuntu 3.x buildbot failing test_dbm

[Nadeem Vawda]

Nadeem Vawda added the comment:

No sign of these failures any more; looks like that fixed it.

--
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15958] bytes.join() should allow arbitrary buffer objects

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is a patch with restored performance. Is not it too complicated?

--
Added file: http://bugs.python.org/file27557/bytes_join_buffers_2.patch

___
Python tracker 

___diff -r 51ce9830d85a Lib/test/test_bytes.py
--- a/Lib/test/test_bytes.pySat Oct 13 11:58:23 2012 -0400
+++ b/Lib/test/test_bytes.pySun Oct 14 01:06:42 2012 +0300
@@ -288,8 +288,22 @@
 self.assertEqual(self.type2test(b"").join(lst), b"abc")
 self.assertEqual(self.type2test(b"").join(tuple(lst)), b"abc")
 self.assertEqual(self.type2test(b"").join(iter(lst)), b"abc")
-self.assertEqual(self.type2test(b".").join([b"ab", b"cd"]), b"ab.cd")
-# XXX more...
+dot_join = self.type2test(b".:").join
+self.assertEqual(dot_join([b"ab", b"cd"]), b"ab.:cd")
+self.assertEqual(dot_join([memoryview(b"ab"), b"cd"]), b"ab.:cd")
+self.assertEqual(dot_join([b"ab", memoryview(b"cd")]), b"ab.:cd")
+self.assertEqual(dot_join([bytearray(b"ab"), b"cd"]), b"ab.:cd")
+self.assertEqual(dot_join([b"ab", bytearray(b"cd")]), b"ab.:cd")
+# Stress it with many items
+seq = [b"abc"] * 1000
+expected = b"abc" + b".:abc" * 999
+self.assertEqual(dot_join(seq), expected)
+# Error handling and cleanup when some item in the middle of the
+# sequence has the wrong type.
+with self.assertRaises(TypeError):
+dot_join([bytearray(b"ab"), "cd", b"ef"])
+with self.assertRaises(TypeError):
+dot_join([memoryview(b"ab"), "cd", b"ef"])
 
 def test_count(self):
 b = self.type2test(b'mississippi')
diff -r 51ce9830d85a Objects/bytearrayobject.c
--- a/Objects/bytearrayobject.c Sat Oct 13 11:58:23 2012 -0400
+++ b/Objects/bytearrayobject.c Sun Oct 14 01:06:42 2012 +0300
@@ -2569,75 +2569,126 @@
 in between each pair, and return the result as a new bytearray.");
 
 static PyObject *
-bytearray_join(PyByteArrayObject *self, PyObject *it)
+bytearray_join(PyObject *self, PyObject *orig)
 {
-PyObject *seq;
-Py_ssize_t mysize = Py_SIZE(self);
-Py_ssize_t i;
-Py_ssize_t n;
-PyObject **items;
-Py_ssize_t totalsize = 0;
-PyObject *result;
-char *dest;
-
-seq = PySequence_Fast(it, "can only join an iterable");
-if (seq == NULL)
+char *sep = PyByteArray_AS_STRING(self);
+const Py_ssize_t seplen = PyByteArray_GET_SIZE(self);
+PyObject *res = NULL;
+char *p;
+Py_ssize_t seqlen = 0;
+Py_ssize_t sz = 0;
+Py_ssize_t i, j, nitems, nbufs;
+PyObject *seq, *item;
+Py_buffer *buffers = NULL;
+#define NB_STATIC_BUFFERS 10
+Py_buffer static_buffers[NB_STATIC_BUFFERS];
+
+seq = PySequence_Fast(orig, "can only join an iterable");
+if (seq == NULL) {
 return NULL;
-n = PySequence_Fast_GET_SIZE(seq);
-items = PySequence_Fast_ITEMS(seq);
-
-/* Compute the total size, and check that they are all bytes */
-/* XXX Shouldn't we use _getbuffer() on these items instead? */
-for (i = 0; i < n; i++) {
-PyObject *obj = items[i];
-if (!PyByteArray_Check(obj) && !PyBytes_Check(obj)) {
-PyErr_Format(PyExc_TypeError,
- "can only join an iterable of bytes "
- "(item %ld has type '%.100s')",
- /* XXX %ld isn't right on Win64 */
- (long)i, Py_TYPE(obj)->tp_name);
+}
+
+seqlen = PySequence_Size(seq);
+if (seqlen == 0) {
+Py_DECREF(seq);
+return PyByteArray_FromStringAndSize("", 0);
+}
+if (seqlen > NB_STATIC_BUFFERS) {
+buffers = PyMem_NEW(Py_buffer, seqlen);
+if (buffers == NULL) {
+Py_DECREF(seq);
+return NULL;
+}
+}
+else {
+buffers = static_buffers;
+}
+
+/* There is at least one thing to join.
+ * Do a pre-pass to figure out the total amount of space we'll
+ * need (sz), and see whether all arguments are buffer-compatible.
+ */
+for (i = nitems = nbufs = 0; i < seqlen; i++) {
+Py_ssize_t itemlen;
+item = PySequence_Fast_GET_ITEM(seq, i);
+if (PyBytes_CheckExact(item)) {
+Py_INCREF(item);
+itemlen = PyBytes_GET_SIZE(item);
+}
+else {
+if (_getbuffer(item, &buffers[nbufs]) < 0) {
+PyErr_Format(PyExc_TypeError,
+"sequence item %zd: expected bytes, bytearray, "
+"or an object with a buffer interface, %.80s 
found",
+i, Py_TYPE(item)->tp_name);
+goto error;
+}
+itemlen = buffers[nbufs].len;
+nbufs++;
+}
+nitems = i + 1;  /* for error cleanup */
+if (itemlen > PY_SSIZE_T_MAX - sz) {
+

[issue16212] mmap() dumps core upon resizing the underlying file

[Charles-François Natali]

Charles-François Natali added the comment:

> I know how to avoid the problem in my case, the bug does not really affect 
> me. I posted it because I thought that the possibility to crash the 
> interpreter is something to be avoided at all costs.

I fully agree with you.
However, that's a problem inherent to mmap(), and I don't think
there's a way to avoid this, but I could be wrong.

> I've found a few examples of handling non-restartable signals with longjmps, 
> but not that familiar with the codebase to estimate how reliably it can be 
> done on all supported platforms. I don't really know how this code would 
> behave, say, on Windows.

You can't use longjmp from signal handlers. Well, you can, but 99% of
the code that does it is broken, because you can only call async-safe
functions from within a signal handler, and certainly can't run the
intepreter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15958] bytes.join() should allow arbitrary buffer objects

[Antoine Pitrou]

Antoine Pitrou added the comment:

The problem with your approach is that the sequence could be mutated while 
another thread is running (_getbuffer() may release the GIL). Then the 
pre-computed size gets wrong.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14229] On KeyboardInterrupt, the exit code should mirror the signal number

[Nadeem Vawda]

Changes by Nadeem Vawda :


--
resolution:  -> rejected
stage:  -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16212] mmap() dumps core upon resizing the underlying file

[Antoine Pitrou]

Antoine Pitrou added the comment:

A pity Posix isn't smart enough to refuse truncate()ing when there's a mmap 
open on the affected pages. Python 3's buffer API is superior in that respect 
:-)

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15958] bytes.join() should allow arbitrary buffer objects

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> The problem with your approach is that the sequence could be mutated while
> another thread is running (_getbuffer() may release the GIL). Then the
> pre-computed size gets wrong.

Well, then I withdraw my patch.

But what if the sequence will be mutated and PySequence_Size(seq) will become 
less seqlen? Then using PySequence_Fast_GET_ITEM() will be incorrect.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15298] _sysconfigdata is generated in srcdir, not builddir

[Trent Nelson]

Changes by Trent Nelson :


Removed file: 
http://bugs.python.org/file27552/issue15298-alt_sysconfigdata2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15958] bytes.join() should allow arbitrary buffer objects

[Antoine Pitrou]

Antoine Pitrou added the comment:

> But what if the sequence will be mutated and PySequence_Size(seq) will become 
> less seqlen? Then using PySequence_Fast_GET_ITEM() will be incorrect.

Perhaps we should detect that case and raise, then.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15936] Add link from os.urandom to random.SystemRandom

[Chris Jerdonek]

Changes by Chris Jerdonek :


--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10117] Tools/scripts/reindent.py fails on non-UTF-8 encodings

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16201] socket.gethostbyname incorrectly parses ip

[Michele Orrù]

Michele Orrù added the comment:

Updated && tested on GNU/Linux (gcc).

--
Added file: http://bugs.python.org/file27558/issue16201.3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15958] bytes.join() should allow arbitrary buffer objects

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


Removed file: http://bugs.python.org/file27557/bytes_join_buffers_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16220] wsgiref does not call close() on iterable response

[Phillip J. Eby]

Phillip J. Eby added the comment:

FYI, this looks like a bug in wsgiref.handlers.BaseHandler.finish_response(), 
which should probably be using a try/finally to ensure .close() gets called.  
(Which would catch a failed write() to the client.)

I'm kind of amazed this has gone undetected this long, but I guess that either:

1. other servers are probably catching errors from .run() and closing,
2. they're not using BaseHandler in their implementation, or
3. most apps don't have anything that essential in close() and/or the clients 
don't disconnect that often.  ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16212] mmap() dumps core upon resizing the underlying file

[Vladimir Ushakov]

Vladimir Ushakov added the comment:

> You can't use longjmp from signal handlers. Well, you can, but 99% of the 
> code that does it is broken, because you can only call async-safe functions 
> from within a signal handler, and certainly can't run the intepreter.

I don't see the reason to run the interpreter. But a quick look at the source 
shows that the current implementation already uses longjmps to handle SIGFPE 
(see pyfpe.h and fpectlmodule.c). It seems to be in use on many platforms.

The only problem I see so far is the possibility that we have to protect too 
much. However, as I could guess from quickly browsing through the mmap() 
implementation, the address of the mmap()ed region never leaves the module, all 
accesses are done using exported methods. If it is really the case, we can wrap 
them into something similar to PyFPE_START_PROTECT() and PyFPE_END_PROTECT() 
(see comments in pyfpe.h).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10050] urllib.request still has old 2.x urllib primitives

[Nadeem Vawda]

Nadeem Vawda added the comment:

Are we still planning on removing URLopener and FancyURLopener in 3.4? The 
documentation for 3.3 does not list these classes as deprecated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16203] Proposal: add re.fullmatch() method

[Tim Peters]

Tim Peters added the comment:

Serhiy, I expect this is easy to implement _inside_ the regexp engine.  The 
complications come from trying to do it outside the engine.  But even there, 
wrapping the original regexp  in

(?:)\Z

is at worst very close.  The only insecurity with that I've thought of concerns 
the doc's warnings about what can appear before an inline re.VERBOSE flag.  It 
probably works fine even if  does begin with  (?...x).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16203] Proposal: add re.fullmatch() method

[Matthew Barnett]

Matthew Barnett added the comment:

It certainly appears to ignore the whitespace, even if the "(?x)" is at the end 
of the pattern or in the middle of a group.

Another point we need to consider is that the user might want to use a 
pre-compiled pattern.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10050] urllib.request still has old 2.x urllib primitives

[Senthil Kumaran]

Senthil Kumaran added the comment:

Only the classes which are marked as deprecated are allowed removed in
the next release.
So the ones which we marked as deprecated in 3.3 can safely go in 3.4.
URLopener and FancyURLopener AFAIR had some dependency/ usage,
deprecating those may require some refactoring of the existing code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12486] tokenize module should have a unicode API

[Eric Snow]

Changes by Eric Snow :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15936] Add link from os.urandom to random.SystemRandom

[Ramchandra Apte]

Ramchandra Apte added the comment:

It should also mention that os.urandom is a low-level interface.

--
nosy: +ramchandra.apte

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16221] tokenize.untokenize() "compat" mode misses the encoding when using an iterator

[Eric Snow]

New submission from Eric Snow:

While traversing the passed iterable, untokenize() will go to "compatibility 
mode" one it hits a 2-tuple instead of a 5-tuple (coming from the iterable).  
That token will not go through the normal steps that other tokens do.  Most 
critically, if that token is the ENCODING token, the source is not encoded to 
that encoding at the end (even though it should have been).  I'll have a patch 
up in a moment (for the tip of default).

There are a couple of other issues at hand that I will bring up separately.

--
assignee: eric.snow
messages: 172848
nosy: eric.snow
priority: normal
severity: normal
stage: test needed
status: open
title: tokenize.untokenize() "compat" mode misses the encoding when using an 
iterator
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16221] tokenize.untokenize() "compat" mode misses the encoding when using an iterator

[Eric Snow]

Changes by Eric Snow :


--
keywords: +patch
stage: test needed -> patch review
Added file: http://bugs.python.org/file27559/untokenize_compat.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16222] "server-side clone" not found by devguide's search box

[Chris Jerdonek]

New submission from Chris Jerdonek:

Neither searching for "server-side" nor "clone" in the devguide "Quick Search" 
box pulls up the mention of server-side clones:

http://docs.python.org/devguide/committing.html#long-term-development-of-features

--
components: Devguide
messages: 172849
nosy: chris.jerdonek, eric.araujo, ezio.melotti, ncoghlan
priority: normal
severity: normal
status: open
title: "server-side clone" not found by devguide's search box
type: enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16223] untokenize returns a string if no encoding token is recognized

[Eric Snow]

New submission from Eric Snow:

If you pass an iterable of tokens and none of them are an ENCODING token, 
tokenize.untokenize() returns a string.  This is contrary to what the docs say:

   It returns bytes, encoded using the ENCODING token, which is the
   first token sequence output by tokenize().

Either the docs should be clarified or untokenize() fixed.  My vote is to fix 
it.  It could check that the first token is an ENCODING token and raise an 
exception.  Alternately it could fall back to using 'utf-8' by default.

[1] http://docs.python.org/py3k/library/tokenize.html#tokenize.untokenize

--
messages: 172850
nosy: eric.snow
priority: normal
severity: normal
stage: test needed
status: open
title: untokenize returns a string if no encoding token is recognized
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16224] tokenize.untokenize() misbehaves when moved to "compatiblity mode"

[Eric Snow]

New submission from Eric Snow:

When tokenize.untokenize() encounters a 2-tuple, it moves to compatibility 
mode, where only the token type and string are used from that point forward.  
There are two closely related problems:

* when the iterable is a sequence, the portion of the sequence prior to the 
2-tuple is traversed a second time under compatibility mode.
* when the iterable is an iterator, the first 2-tuple encountered is 
essentially gobbled up (see issue16221).

Either an explicit "iterable = iter(iterable)" or "iterable = list(iterable)" 
should happen at the very beginning of Untokenizer.untokenize().  If the 
former, Untokenizer.compat() should be fixed to not treat that first token 
differently.  If the latter, self.tokens should be cleared at the beginning of 
Untokenizer.compat().

I'll put up a patch with the second option when I get a chance.

--
assignee: eric.snow
components: Library (Lib)
messages: 172851
nosy: eric.snow
priority: normal
severity: normal
stage: test needed
status: open
title: tokenize.untokenize() misbehaves when moved to "compatiblity mode"
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16221] tokenize.untokenize() "compat" mode misses the encoding when using an iterator

[Eric Snow]

Eric Snow added the comment:

issue16224 _may_ supercede this ticket.

--
components: +Library (Lib)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16224] tokenize.untokenize() misbehaves when moved to "compatiblity mode"

[Eric Snow]

Eric Snow added the comment:

Actually, here's a patch with the first option.  It preserves iterators as 
iterators, rather than dumping them into a list.  I've also rolled the tests 
from issue16221 into this patch.  Consequently, if the patch is suitable, that 
issue can be closed.

--
keywords: +patch
stage: test needed -> patch review
Added file: http://bugs.python.org/file27560/untokenize_compat_force_iter.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16221] tokenize.untokenize() "compat" mode misses the encoding when using an iterator

[Eric Snow]

Eric Snow added the comment:

The patch that I have in #16224 takes care of this issue.  If that issue goes 
in another direction however...

--
status: open -> pending
superseder:  -> tokenize.untokenize() misbehaves when moved to "compatiblity 
mode"

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com