[issue17899] os.listdir() leaks FDs if invoked on FD pointing to a non-directory

[Evgeny Kapun]

Evgeny Kapun added the comment:

Simple test:

while True:
try:
listdir(0)
except NotADirectoryError:
pass

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17899] os.listdir() leaks FDs if invoked on FD pointing to a non-directory

[Evgeny Kapun]

Evgeny Kapun added the comment:

To make fdopendir fail, just pass any valid FD which points to a non-directory, 
such as a file or a pipe.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17862] itertools.chunks(iterable, size, fill=None)

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee:  -> rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17899] os.listdir() leaks FDs if invoked on FD pointing to a non-directory

[Larry Hastings]

Larry Hastings added the comment:

Patch attached.  I don't know how to make fdopendir fail, so I had to do it by 
inspection.

While I was in there, I ifdef'd out the variable that should only be used if 
fdopendir is available.

--
keywords: +patch
stage: needs patch -> patch review
Added file: 
http://bugs.python.org/file30120/larry.listdir.fd.leakage.bug.1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17810] Implement PEP 3154 (pickle protocol 4)

[Alexandre Vassalotti]

Alexandre Vassalotti added the comment:

Antoine, can you share the code for your benchmarks which show performance 
improvements when framing is enabled? I am seeing the same 10-15% slowdown even 
when pickling stuff to pure Python objects:

### Without the patch
./python -m timeit -r 50 -s "import pickle, _pyio; f = _pyio.BytesIO(); x = 
list(range(1000))" "pickle.dump(x, f, protocol=4)"
1 loops, best of 50: 28.5 usec per loop

### With the patch
./python -m timeit -r 50 -s "import pickle, _pyio; f = _pyio.BytesIO(); x = 
list(range(1000))" "pickle.dump(x, f, protocol=4)"
1 loops, best of 50: 32.9 usec per loop

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17890] argparse: mutually exclusive groups full of suppressed args can cause AssertionErrors

[Terry J. Reedy]

Terry J. Reedy added the comment:

Garrett and Yogesh, please submit contributor license agreements
http://www.python.org/psf/contrib/contrib-form/
if you have not yet. When one is properly recorded, a * appears after your name.

--
nosy: +terry.reedy
versions: +Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17897] Optimize unpickle prefetching

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17887] docs: summary page - generator vs iterator vs iterable

[Terry J. Reedy]

Terry J. Reedy added the comment:

To make the subset relationships clear in the iterable entry, perhaps replace
  See also iterator, sequence, and generator. 
with
  Some iterables are iterators and some iterators are generators. Some 
iterables are sequences. For details, see iterator, sequence, and generator.

I see that 'generator' is currently used in the 3.3.1 glossary for 'generator 
function' and there is no entry for generator (object) proper. I thought that 
had been fixed after pydev discussion.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17897] Optimize unpickle prefetching

[Alexandre Vassalotti]

Alexandre Vassalotti added the comment:

Do you have benchmark results to show the code with the patch is faster?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17887] docs: summary page - generator vs iterator vs iterable

[Terry J. Reedy]

Terry J. Reedy added the comment:

Briefly,
An iterable is an object that can be iterated with an iterator.
An iterator is an iterable that responds to next() calls, including the 
implicit calls in a for statement.
A generator is an iterator created by a generator function, which is a function 
with a yield keyword in its body.
A sequence is an iterable with minimal sequence methods.

The four terms are all in the glossary. The entry for iterable already points 
to the other three. What do you think is missing?

A range (as documented) is an immutable sequence.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17902] Document that _elementtree C API cannot use custom TreeBuilder for iterparse or IncrementalParser

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +eli.bendersky
stage:  -> patch review
versions: +Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17862] itertools.chunks(iterable, size, fill=None)

[Terry J. Reedy]

Terry J. Reedy added the comment:

[Anatoly, 'Versions 3.5' is for changes that should *not* happen in 3.4, such 
as a planned removal for something deprecated in 3.3.]

--
nosy: +terry.reedy
versions:  -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17899] os.listdir() leaks FDs if invoked on FD pointing to a non-directory

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +larry
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17859] improve error message for saving ints to file

[Terry J. Reedy]

Terry J. Reedy added the comment:

I think the point is that the error message does not make much sense unless one 
knows a) that the file is open in binary mode (and that could have happened in 
a different file) and b) that 'does not support the buffer interface' means 'is 
not a bytes-like object'. A possible replacement that addresses both problems:

binary-mode write requires bytes-like object, not 'int'

--
nosy: +terry.reedy
stage:  -> needs patch
versions:  -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17810] Implement PEP 3154 (pickle protocol 4)

[Antoine Pitrou]

Antoine Pitrou added the comment:

> The framing patch seems to have a significant negative effect on
> performance.

I wouldn't call it significant. Any speedup or slowdown less than 50% is
unlikely to be noticeable in real-world applications.

Mitigating the regression is probably a matter of tweaking the
read/write fast paths (optimizing for the common case where a frame is
ongoing and the buffer is neither full nor empty).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17810] Implement PEP 3154 (pickle protocol 4)

[Alexandre Vassalotti]

Alexandre Vassalotti added the comment:

The framing patch seems to have a significant negative effect on performance.

Report on Linux avassalotti 3.2.5-gg1130 #1 SMP Mon Feb 4 02:25:47 PST 2013 
x86_64 x86_64
Total CPU cores: 12

### fastpickle ###
Min: 0.447194 -> 0.505841: 1.13x slower
Avg: 0.455517 -> 0.509537: 1.12x slower
Significant (t=-22.05)
Stddev: 0.01438 -> 0.00967: 1.4875x smaller

### fastunpickle ###
Min: 0.583922 -> 0.638744: 1.09x slower
Avg: 0.589183 -> 0.649506: 1.10x slower
Significant (t=-21.77)
Stddev: 0.00939 -> 0.01720: 1.8324x larger

Would it be possible to mitigate the regression?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17902] Document that _elementtree C API cannot use custom TreeBuilder for iterparse or IncrementalParser

[Aaron Oakley]

New submission from Aaron Oakley:

It would really help to document that the C API can only use the default 
xml.etree.ElementTree.TreeBuilder for targets with iterparse (and by extension, 
IncrementalParser).

I got a nice surprise about that when I went from 3.2 to 3.3 and started 
getting "TypeError: event handling only supported for ElementTree.TreeBuilder 
targets".

I included a patch to add notes to iterparse and IncrementalParser, but I'm not 
sure what to refer to the C module as since xml.etree.cElementTree is 
deprecated.

--
assignee: docs@python
components: Documentation, XML
files: elementtree.rst-340a0.patch
keywords: patch
messages: 188329
nosy: Aaron.Oakley, docs@python
priority: normal
severity: normal
status: open
title: Document that _elementtree C API cannot use custom TreeBuilder for 
iterparse or IncrementalParser
type: behavior
versions: Python 3.4
Added file: http://bugs.python.org/file30119/elementtree.rst-340a0.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17854] symmetric difference operation applicable to more than two sets

[Terry J. Reedy]

Terry J. Reedy added the comment:

Sets have methods that do not have operators (such as len, isdisjoint),
operators that do not have non-special methods (such as in, <), and 
method-operator pairs that do the same thing (such as (union, |), 
(symmetric_difference, ^)). For the pairs, it gives the method signature and 
the *equivalent* operator expression. Since .union takes multiple 'other' args, 
the equivalent operator expression does too. Since .symmetric_difference only 
takes one 'other' arg, so does the expression.

A coherent proposal would change the method code and doc to the following:

symmetric_difference(other, ...)
set ^ other ^ ...
Return a new set with elements in an odd number of the sets.

s={1,2, 5}
t={2,3, 5}
u={3,4, 5}
print(s^t^u)
>>> 
{1, 4, 5}

I believe the proposal was once considered, and rejected. An argument for is 
that the effect of chained symmetric differences is not obvious, as evidenced 
by Amit's mistaken characterization. I had to think a bit before I was sure of 
the answer. An argument against is that what one actually gets is seldom 
wanted, so that allowing more than two inputs to the method would have little 
benefit. 

What might be done is to document the symmetric different of multiple sets with 
a parenthetical comment such as

"(The symmetric difference of multiple sets, a ^ b ^ c ^ ..., is a new set with 
elements appearing in an odd number of input sets.)"

This would let people know what to expect from such expressions, in a situation 
where the effect is less obvious than usual.

--
nosy: +rhettinger, terry.reedy
stage:  -> patch review
versions: +Python 2.7, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17810] Implement PEP 3154 (pickle protocol 4)

[Antoine Pitrou]

Antoine Pitrou added the comment:

Here is an updated framing patch which fixes the issue reported by Alexandre. 
There are also a couple added tests.

--
Added file: http://bugs.python.org/file30118/framing3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17849] Missing size argument in readline() method for httplib's class LineAndFileWrapper

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +orsenthil, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17901] _elementtree.TreeBuilder raises IndexError on end if constructed with element_factory=None

[Aaron Oakley]

New submission from Aaron Oakley:

When the _elementtree module is in use, the TreeBuilder class will raise an 
IndexError in treebuilder_handle_end if __init__ was passed "None".

I discovered this while writing a subclass of TreeBuilder with a modified 
__init__ method that delegated to TreeBuilder:

class MyTreeBuilder(ET.TreeBuilder):
def __init__(self, element_factory=None):
super().__init__(element_factory)

Used as a target, this class (and also simply "TreeBuilder(None)") will cause 
the IndexError when "parser.feed(data)" is called.

>>> import xml.etree.ElementTree as ET
>>> parser = ET.XMLParser(target=ET.TreeBuilder(None))
>>> parser.feed('22')
Traceback (most recent call last):
  File "", line 1, in 
IndexError: pop from empty stack

The error is raised from treebuilder_handle_end, but the cause appears to be in 
treebuilder_handle_start.

if (self->element_factory) {
node = PyObject_CallFunction(self->element_factory, "OO", tag, attrib);
} else {
node = create_new_element(tag, attrib);
}

I included a patch adding a check against Py_None to the "if" test above which 
seems to fix the issue. I also included a simple test case for it.

--
components: XML
files: _elementtree.c-340a0.patch
keywords: patch
messages: 188326
nosy: Aaron.Oakley
priority: normal
severity: normal
status: open
title: _elementtree.TreeBuilder raises IndexError on end if constructed with 
element_factory=None
type: behavior
versions: Python 3.2, Python 3.3, Python 3.4
Added file: http://bugs.python.org/file30117/_elementtree.c-340a0.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17900] Recursive OrderedDict pickling

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

Here is a patch which allows pickling of recursive OrderedDicts. Actually it 
simplifies __reduce__() code.

--
components: Library (Lib)
files: OrderedDict_pickle_recursive.patch
keywords: patch
messages: 188325
nosy: alexandre.vassalotti, pitrou, rhettinger, serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: Recursive OrderedDict pickling
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file30116/OrderedDict_pickle_recursive.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17898] gettext bug while parsing plural-forms metadata

[Steve Strassmann]

Steve Strassmann added the comment:

There seem to be several bugs involving this particular inner loop in 
gettext._parse(), but I don't think they're equivalent.

The present bug (issue17898) is that parsing a plural header breaks the 
following header when it happens to be a comment.

issue1475523 seems to involve multi-line handling

issue12425 seems to involve breaking when the plural-forms value is empty.

Perhaps a useful design pattern to follow for code which executes this inner 
loop would be to have some initialization and loop invariants which are 
asserted true on each iteration. For example, properly initializing k and v on 
each iteration.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17898] gettext bug while parsing plural-forms metadata

[R. David Murray]

R. David Murray added the comment:

Does this bear any relationship to issue 1475523?  (And yes, I know it 
is...sad...that that issue hasn't been fixed yet.)

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17899] os.listdir() leaks FDs if invoked on FD pointing to a non-directory

[Evgeny Kapun]

New submission from Evgeny Kapun:

When called with a file descriptor as an argument, os.listdir() duplicates it 
to pass to fdopendir(3). If this call fails, the new file descriptor is not 
closed, which leads to file descriptor leak.

--
components: Library (Lib)
messages: 188322
nosy: abacabadabacaba
priority: normal
severity: normal
status: open
title: os.listdir() leaks FDs if invoked on FD pointing to a non-directory
type: resource usage
versions: Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11016] Add S_ISDOOR to the stat module

[Christian Heimes]

Christian Heimes added the comment:

I have done some research. The POSIX 1.b extensions aren't used on any 
supported system (neither Linux nor BSD, Solaris or AIX).


random...@fastmail.us has compiled a list of even more file types:

Heirloom toolchest "ls" supports:
http://heirloom.cvs.sourceforge.net/viewvc/heirloom/heirloom/ls/ls.c?revision=1.9&view=markup
http://heirloom.cvs.sourceforge.net/viewvc/heirloom/heirloom/ls/ls.1?revision=1.5&view=markup
S_IFNWK HP-UX network special file
S_IFNAM  XENIX special named file
S_INSEM XENIX semaphore subtype of IFNAM (looked up from s->rdev)
S_INSHD XENIX shared data subtype of IFNAM " " " "

Of these, GNU coreutils ls only supports doors and whiteouts.

Chasing after a random hunch (something about AIX), I found these:

http://cd.textfiles.com/transameritech2/EXTRAS/JOVE-4.6/ASK.C
S_ISHIDDEN Hidden Directory [aix]
S_ISCDF Context Dependent Files [hpux] 
S_ISNWK Network Special [hpux]

http://lists.gnu.org/archive/html/bug-gnulib/2012-12/msg00084.html
S_ISMPX AIX "MPX" file (multiplex device?)

https://github.com/gagern/gnulib/blob/master/tests/test-sys_stat.c
has a massive pile of macros with no comments
S_ISCTG S_ISMPB S_ISMPX S_ISNAM S_ISNWK S_ISOFD S_ISOFL S_ISPORT

http://lists.gnu.org/archive/html/bug-gnulib/2004-08/msg00017.html
S_ISOFD Cray DMF (data migration facility): off line, with data
S_ISOFL Cray DMF (data migration facility): off line, with no data
S_ISCTG Contiguous
(It's possible that these may not be file types)

http://doiso.googlecode.com/svn/trunk/Source/mkisofs-1.12b5/include/statdefs.h
S_ISMPC  UNUSED multiplexed c
S_ISNAM  Named file (XENIX)
S_ISMPB  UNUSED multiplexed b
S_ISCNT  Contiguous file
S_ISSHAD Solaris shadow inode

http://www.opensource.apple.com/source/gnutar/gnutar-450/gnutar/lib/sys_stat_.h
S_ISMPB /* V7 */
S_ISPORT /* Solaris 10 and up */
S_TYPEISSEM S_TYPEISSHM - macros to check the XENIX IFNAM types
mentioned above
S_TYPEISMQ S_TYPEISTMO

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Evgeny Kapun]

Changes by Evgeny Kapun :


--
nosy: +abacabadabacaba

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12077] Harmonizing descriptor protocol documentation

[Ned Batchelder]

Changes by Ned Batchelder :


--
nosy: +nedbat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17898] gettext bug while parsing plural-forms metadata

[Ned Batchelder]

Changes by Ned Batchelder :


--
nosy: +nedbat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17810] Implement PEP 3154 (pickle protocol 4)

[Alexandre Vassalotti]

Alexandre Vassalotti added the comment:

Those methods wouldn't be much more a maintenance burden than the special cases 
already present in the implementation of __reduce__. These methods would only 
need to be provided by classes that wishes to support efficient in-place 
pickling provided by protocol 4. As such, this approach better as it would rely 
on duck typing rather than concrete type checks, which IMHO do not belong in 
the default object implementation.

Plus, having this generic API would allow pickle to share the same pickling and 
unpickling code for lists, dicts, sets and other mutable collections.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17636] Modify IMPORT_FROM to fallback on sys.modules

[Brett Cannon]

Brett Cannon added the comment:

I have uploaded a patch with failing tests that should work after this is all 
said and done. Philip, please make sure I covered your tests as expected (I 
tweaked one because it already was working the way I did it). This way we at 
least know what we are aiming for in terms of results.

--
keywords: +patch
Added file: http://bugs.python.org/file30115/circular_import_tests.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17898] gettext bug while parsing plural-forms metadata

[Steve Strassmann]

New submission from Steve Strassmann:

The gettext.py parser used by django (lib/python2.7/gettext.py),
GNUTranslations._parse(), around line 313 does not use clean values for k,v on 
each
iteration ("for item in tmsg.splitlines():")

To reproduce the problem (see traceback, below), try parsing a .PO file 
containing two headers like this, with a comment header immediately following a 
plurals header. This example was created by calling msgcat to combine several 
.po files into a single .po file. Msgcat inserted the comment line.

"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"#-#-#-#-#  messages.po (EdX Studio)  #-#-#-#-#\n"

Parsing the first header binds the inner loop variables:
  k= plural-forms v= ['nplurals=2', ' plural=(n != 1)', '']

Parsing the second header leaves k,v untouched, which then causes an improper
attempt to parse (since it's a comment, no further parsing of k,v should occur)
 v = v.split(';')

Bug workaround: I use polib to read and immediately save the file. This 
reorders the metadata to avoid presenting the parser with something that will 
break it.

Recommended bug fix: on each iteration over tmsg.splitlines, reset the values 
of k,v = (None, None)


Traceback:
File 
"/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/core/handlers/base.py"
 in get_response
89. response = middleware_method(request)
File 
"/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/middleware/locale.py"
 in process_request
24. translation.activate(language)
File 
"/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/__init__.py"
 in activate
105. return _trans.activate(language)
File 
"/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py"
 in activate
201. _active.value = translation(language)
File 
"/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py"
 in translation
191. current_translation = _fetch(language, fallback=default_translation)
File 
"/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py"
 in _fetch
180. res = _merge(localepath)
File 
"/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py"
 in _merge
156. t = _translation(path)
File 
"/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py"
 in _translation
138. t = gettext_module.translation('django', path, [loc], 
DjangoTranslation)
File 
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py"
 in translation
480. t = _translations.setdefault(key, class_(fp))
File 
"/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py"
 in __init__
76. gettext_module.GNUTranslations.__init__(self, *args, **kw)
File 
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py"
 in __init__
180. self._parse(fp)
File 
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py"
 in _parse
315. v = v.split(';')

Exception Type: AttributeError at /
Exception Value: 'list' object has no attribute 'split'

--
components: Library (Lib)
messages: 188318
nosy: straz
priority: normal
severity: normal
status: open
title: gettext bug while parsing plural-forms metadata
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17897] Optimize unpickle prefetching

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

When C implementation of the unpickle reads a data, it also prefetches some 
data using peek() (if available) and then concatenates read and peeked chunks 
in the one input buffer. This causes an additional copying when a large data is 
read. The proposed patch gets rid of concatenating by moving a peeking before 
reading.

--
components: Extension Modules
files: pickle_peek.patch
keywords: patch
messages: 188317
nosy: alexandre.vassalotti, pitrou, serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: Optimize unpickle prefetching
type: performance
versions: Python 3.4
Added file: http://bugs.python.org/file30114/pickle_peek.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17895] TemporaryFile name returns an integer in python3

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

http://docs.python.org/3/library/io.html#io.FileIO.name

The file name. This is the file descriptor of the file when no name is 
given in the constructor.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17896] Move Windows external libs from \..\ to \externals

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +loewis

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17810] Implement PEP 3154 (pickle protocol 4)

[Antoine Pitrou]

Antoine Pitrou added the comment:

> I think it could be worthwhile to investigate a generic API for
> pickling collections in-place. For example, a such API would helpful
> for pickling set subclasses in-place.

Is the use case important enough? Otherwise, this is more
__special_method__ complication that we'll have to maintain for pickle's
only use.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17895] TemporaryFile name returns an integer in python3

[Antoine Pitrou]

Antoine Pitrou added the comment:

Given the following property of TemporaryFile:

"Under Unix, the directory entry for the file is removed immediately after the 
file is created. Other platforms do not support this; your code should not rely 
on a temporary file created using this function having or not having a visible 
name in the file system."
(from http://docs.python.org/dev/library/tempfile.html#tempfile.TemporaryFile)

I find it absurd to expect the name attribute to return an actual filename, 
since it won't exist anymore by the time you use it (under Unix, at least). So, 
we could return an invalid filename, but I don't see the point, and the current 
behaviour is as good as any other.

If you want a usable name, by definition you must use NamedTemporaryFile. 
Recommend closing.

--
nosy: +georg.brandl, ncoghlan
resolution:  -> rejected

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17810] Implement PEP 3154 (pickle protocol 4)

[Alexandre Vassalotti]

Alexandre Vassalotti added the comment:

I am currently fleshing out an improved implementation for the reduce protocol 
version 4. One thing I am curious about is whether we should keep the special 
cases we currently have there for dict and list subclasses.

I recall Raymond expressed disagreement in #msg83098 about this behavior. I 
agree that having __setitem__ called before __init__ make it harder for dict 
and list subclasses to support pickling. To take advantage of the special case, 
subclasses need to do their required initialization in the __new__ method.

On the other hand, it does decrease the memory requirements for unpickling such 
subclasses---i.e., we can build the object in-place instead of building an 
intermediary list or dict. Reading PEP 307 confirms indeed that was the 
original intention.

One possible solution, other than removing the special case completely, is to 
make sure we initialize the object (using the BUILD opcode) before we call 
__setitem__ or append on it. This would be a simple change that would solve the 
initialization issue. However, I would still feel uneasy about the default 
object.__reduce__ behavior depending on the object's subtype.

I think it could be worthwhile to investigate a generic API for pickling 
collections in-place. For example, a such API would helpful for pickling set 
subclasses in-place.

__items__() or   Return an iterator of the items in the collection. Would be
__getitems__()   equivalent to iter(dict.items()) on dicts and iter(list) on
 lists.

__additems__(items)  Add a batch of items to the collection. By default, it 
would
 be defined as:

 for item in items:
 self.__additem__(item)

 However, subclasses would be free to provide a more 
efficient
 implementation of the method. Would be equivalent to
 dict.update on dicts and list.extend on lists.

__additem__(item)Add a single item to the collection. Would be equivalent to
 dict[item[0]] = item[1] on dicts and list.append on lists.

The collections module's ABCs could then provide default implementations of 
this API, which would give its users efficient in-place pickling automatically.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1159051] Handle corrupted gzip files with unexpected EOF

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

In both cases broken applications use the undocumented implementation details. 
But such changes are extremely strong for bugfix release and they should not be 
done without a special need. I propose to revert these changes in 2.7, 3.2 and 
3.3 (possibly leaving in the default branch). Unfortunately, I was not online 
right before the release of the latest bugfix release and failed to do this. 
Fortunately, it is now possible to fix this in regression fix releases.

--
nosy: +benjamin.peterson, georg.brandl
priority: normal -> release blocker
type: enhancement -> behavior
versions: +Python 2.7, Python 3.2, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17895] TemporaryFile name returns an integer in python3

[R. David Murray]

R. David Murray added the comment:

The 'name' attribute of TemporaryFile is not part of the API.  It happens to 
exist only because the underlying file object has a 'name' attribute.  On posix 
platforms the value is not really useful for anything.  In other words, that 
numpy code was buggy to start with, the bug was just hidden by the fact that in 
python2 name happened to be a string (''), and nobody looked at the 
result.  What numpy was doing with it produced a nonsense value, but I guess 
nobody noticed.

Now, that said, I don't know why the value changed between Python2 and Python3, 
and that might conceivably be a bug of some sort.  I'm guessing it is a 
consequence of the IO system rewrite and is not a bug per-se, but it might also 
be that there are improvements that could be made here.

--
nosy: +pitrou, r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15984] Wrong documentation for PyUnicode_FromObject() and PyUnicode_FromEncodedObject()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

In other languages usually the world "coercion" is used for implicit 
conversion, i.e. int->long, int->float, float->complex. str->unicode in Python 
2 (that's what PyUnicode_FromObject() does). But the last conversion is not 
supported in Python 3. The term "coercion" has also been used in Python 2 in 
the narrow sense (see the __coerce__() method), and in this sense Python 3 does 
not support "coercion". Therefore, I believe that it is better to avoid the use 
of this term.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17896] Move Windows external libs from \..\ to \externals

[Zachary Ware]

New submission from Zachary Ware:

PCbuild\readme.txt has a comment from Trent Nelson dated 2-April-2008 
suggesting moving the external library location for Windows from "\..\" to "\external\" to make switching between 
versions easier.  I've gotten rather annoyed with having all of the external 
libs cluttering the folder holding my cpython checkout, so I've written a patch 
that moves them :).

The patch actually moves the external libs into "\external-34", 
because in most cases it's best to have separately built externals for each 
Python version.  The idea is to then have an "external-35" after 3.4 is 
released, and possibly "external-27" and "external-33" if it is permissible to 
backport this kind of change.  There are only four places that would need to be 
changed in other versions; PCbuild/pyproject.props:19, PCbuild/rt.bat:41, 
Tools/buildbot/external-common.bat:4 and 5, and Lib/tkinter/_fix.py:52.  All 
four are simple replacements which I believe could be automated by the release 
script.  PCbuild/build_ssl.py is patched to find the externals dir in 
PCbuild/pyproject.props while finding the OpenSSL version there.

A few of the benefits to this that have occurred to me are:
- the fact that the dir holding your cpython checkout doesn't get cluttered up 
with external libs
- it becomes very easy to rebuild all of the externals (just `rmdir /s /q 
external-34` and run external(-amd64).bat again)
- no possibility of problems arising from one version trying to use an external 
lib built by/for another version

The only real downsides I'm aware of are that it is a change from what everyone 
is accustomed to, and that it will use up a bit more disk space (which could be 
mitigated by using junctions, as Trent suggested in the comment in readme.txt). 
 Also, committing this would cause an extra long build time on the buildbots 
due to having to recompile Tcl/Tk and OpenSSL.

Everything builds and works for me using the buildbot scripts after the patch.  
All comments welcome :)

Thanks,

Zach

--
components: Build, Extension Modules, Windows
files: move_externals.diff
keywords: patch
messages: 188309
nosy: brian.curtin, tim.golden, trent, zach.ware
priority: normal
severity: normal
status: open
title: Move Windows external libs from \..\ to \externals
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file30113/move_externals.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15902] imp.load_module won't accept None for the file argument for a C extension

[Brett Cannon]

Changes by Brett Cannon :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15902] imp.load_module won't accept None for the file argument for a C extension

[Brett Cannon]

Brett Cannon added the comment:

Leaving it up to the release manager to decide if this is worth cherrypicking 
or just waiting for the next bugfix release.

--
nosy: +larry
priority: normal -> release blocker
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15834] 2to3 benchmark not working under Python 3

[Brett Cannon]

Changes by Brett Cannon :


--
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15834] 2to3 benchmark not working under Python 3

[Roundup Robot]

Roundup Robot added the comment:

New changeset 66f168b468c3 by Brett Cannon in branch 'default':
#15834: Make 2to3 so it can actually be proper translated.
http://hg.python.org/benchmarks/rev/66f168b468c3

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17890] argparse: mutually exclusive groups full of suppressed args can cause AssertionErrors

[paul j3]

paul j3 added the comment:

I see three solutions - 

1) gholms' patch which removes '() ' and [] '

2) Yogesh's patch which removes all duplicated spaces.

3) remove the 2 asserts. 

The first 2 do the same thing most of the time, but may differ if the user 
somehow inserts spaces into names.  The third leaves the extra blanks, but 
renders them innocuous.  I doubt if the asserts were written to catch this 
problem.  They probably were included to verify the usage line had been split 
up as expected prior to reassembling on multiple lines.

As best I can tell test_argparse.py does not test for these spaces.  Curiously 
though a port of argparse to javascript does have a test case with the extra 
space.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17895] TemporaryFile name returns an integer in python3

[Julian Taylor]

New submission from Julian Taylor:

sys.version_info(major=3, minor=3, micro=1, releaselevel='final', serial=0)
In [3]: type(tempfile.TemporaryFile().name)
Out[3]: builtins.int

in python2 it returned a string, this is a somewhat pointless api change which 
breaks some third party code, e.g. numpy 
(https://github.com/numpy/numpy/issues/3302)

--
components: Library (Lib)
messages: 188305
nosy: jtaylor
priority: normal
severity: normal
status: open
title: TemporaryFile name returns an integer in python3
type: behavior
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17888] docs: more information on documentation team

[Tshepang Lekhonkhobe]

Changes by Tshepang Lekhonkhobe :


--
nosy: +tshepang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17860] subprocess docs lack info how to use output result

[Tshepang Lekhonkhobe]

Changes by Tshepang Lekhonkhobe :


--
nosy: +tshepang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17859] improve error message for saving ints to file

[Éric Araujo]

Éric Araujo added the comment:

Ah, right, I missed the part about the file being opened in binary mode.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15984] Wrong documentation for PyUnicode_FromObject() and PyUnicode_FromEncodedObject()

[R. David Murray]

R. David Murray added the comment:

Well, while 'coercion' does refer to changing from one type to another, and 
technically we are doing that here, in OO we generally think of subclasses as 
more-or-less being of the same type as the superclass.  So I think it would be 
clearer to spell out that we are changing the object type to be that of the 
superclass.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15902] imp.load_module won't accept None for the file argument for a C extension

[Brett Cannon]

Changes by Brett Cannon :


--
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
versions: +Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15902] imp.load_module won't accept None for the file argument for a C extension

[Roundup Robot]

Roundup Robot added the comment:

New changeset c0a21617dbee by Brett Cannon in branch '3.3':
Issue #15902: Fix imp.load_module() to accept None as a file when
http://hg.python.org/cpython/rev/c0a21617dbee

New changeset 322c556260d5 by Brett Cannon in branch 'default':
#15902: merge w/ 3.3
http://hg.python.org/cpython/rev/322c556260d5

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17838] Can't assign a different value for sys.stdin in IDLE

[Roger Serwy]

Roger Serwy added the comment:

3.2 still has the problem fixed in issue17585 for 3.3 and 3.4. This only 
applies if issue17585 patch gets applied to 3.2.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11016] Add S_ISDOOR to the stat module

[Christian Heimes]

Christian Heimes added the comment:

The stat module (or whatever it is going to be in Python 3.4) is missing more 
checks than S_ISDOOR:

# Solaris
S_ISDOOR()
S_ISPORT()

# POSIX 1.b real-time extension
S_ISMSG()
S_ISSEM()
S_ISSHM()

# whiteout, translucent file systems
S_ISWHT

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Raymond Hettinger added the comment:

> why the different fix for 3.3 

I reverted the 2.7.4 addition of __dict__ rather than introduce more 
differences between point releases with possible unintended effects.

In 3.3, the __dict__ attribute was there from the outset and was advertised in 
the docs, so it made more sense to leave it in and just suppress its inclusion 
in pickling.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Ben Hoyt]

Ben Hoyt added the comment:

Thanks. I thought about that -- but I think I *want* to benchmark it when
they're cached, so that we're comparing apples with apples, cached system
calls with cached systems calls. The benchmark would almost certainly be a
lot "better" (BetterWalk would be even faster) if I was comparing the
non-cached results. I'll think about it some more though.

Thoughts?

-Ben

On Fri, May 3, 2013 at 7:03 PM, Charles-François Natali <
rep...@bugs.python.org> wrote:

>
> Charles-François Natali added the comment:
>
> > However, the reason I'm keen on iterdir_stat() is that I'm seeing it
> speed up os.walk() by a factor of 10 in my recent tests (note that I've
> made local mods, so these results aren't reproducible for others yet). This
> is doing a walk on a dir tree with 7800 files and 155 dirs:
> >
> > Using fast _betterwalk
> > Priming the system's cache...
> > Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 1/3...
> > Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 2/3...
> > Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 3/3...
> > os.walk took 0.178s, BetterWalk took 0.017s -- 10.5x as fast
> >
> > Sometimes Windows will go into this "I'm really caching stat results
> good" mode -- I don't know what heuristic determines this -- and then I'm
> seeing a 40x speed increase. And no, you didn't read that wrong. :-)
>
> I/O benchmarks shouldn't use timeit or repeated calls: after the first
> run, most of your data is in cache, so subsequent runs are
> meaningless.
>
> I don't know about Windows, but on Linux you should do something like:
> # echo 3 > /proc/sys/vm/drop_caches
>
> to start out clean.
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Ben Hoyt]

Ben Hoyt added the comment:

2.7 fix works for me, thanks! Just curious -- why the different fix for 3.3 
(addition of __getstate__ instead of removal of __dict__)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12077] Harmonizing descriptor protocol documentation

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee:  -> rhettinger
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17894] Edits to descriptor howto

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee: docs@python -> rhettinger
nosy: +rhettinger
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee: amaury.forgeotdarc -> rhettinger
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
Removed message: http://bugs.python.org/msg188064

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Roundup Robot]

Roundup Robot added the comment:

New changeset 65cd71abebc8 by Raymond Hettinger in branch '3.3':
Issue #15535:  Fix pickling of named tuples.
http://hg.python.org/cpython/rev/65cd71abebc8

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17890] argparse: mutually exclusive groups full of suppressed args can cause AssertionErrors

[Yogesh Chaudhari]

Yogesh Chaudhari added the comment:

A user generated line with multiple spaces may be essential if we are getting 
the usage data (particularly from) a csv file (that may have random spaces in 
certain fields) or some other form of stored or auto-generated data.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Roundup Robot]

Roundup Robot added the comment:

New changeset 18303391b981 by Raymond Hettinger in branch '2.7':
Issue #15535:  Fix regression in pickling of named tuples.
http://hg.python.org/cpython/rev/18303391b981

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Charles-François Natali]

Charles-François Natali added the comment:

> However, the reason I'm keen on iterdir_stat() is that I'm seeing it speed up 
> os.walk() by a factor of 10 in my recent tests (note that I've made local 
> mods, so these results aren't reproducible for others yet). This is doing a 
> walk on a dir tree with 7800 files and 155 dirs:
>
> Using fast _betterwalk
> Priming the system's cache...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 1/3...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 2/3...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 3/3...
> os.walk took 0.178s, BetterWalk took 0.017s -- 10.5x as fast
>
> Sometimes Windows will go into this "I'm really caching stat results good" 
> mode -- I don't know what heuristic determines this -- and then I'm seeing a 
> 40x speed increase. And no, you didn't read that wrong. :-)

I/O benchmarks shouldn't use timeit or repeated calls: after the first
run, most of your data is in cache, so subsequent runs are
meaningless.

I don't know about Windows, but on Linux you should do something like:
# echo 3 > /proc/sys/vm/drop_caches

to start out clean.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com