Roundup Robot added the comment:
New changeset d37f963394aa by Serhiy Storchaka in branch '2.7':
Correct and improve comments in test_fileinput (closes #20501).
http://hg.python.org/cpython/rev/d37f963394aa
New changeset 204ef3bca9c8 by Serhiy Storchaka in branch '3.3':
Correct comments and
Serhiy Storchaka added the comment:
Thank you Vajrasky and Zachary.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20501
___
___
Serhiy Storchaka added the comment:
My bad. I had cleaned patch before committing to 3.x, but forgot to revert and
correct it in 2.7. Thank you Vajrasky an Zachary.
Does it mean: UTF-7 is an unlikely locale encoding?
Sorry for my bad English. I meant that UTF-7 is used here because it is
Zachary Ware added the comment:
For the UTF-7 comment, I'd go with something like locale encoding is probably
not UTF-7 or UTF-7 is a convenient, seldom used encoding, or something to
that effect.
About writeTmp vs. context manager: as you wish. It would still be good to
convert writeTmp to
Zachary Ware added the comment:
Agreed with Vajrasky, the 2.7 commit wasn't as clean as it could be (sorry I
didn't get an actual review done before you committed, I posted my last test
results in a bit of a hurry). The commented out line should either be removed,
or used instead of the
Zachary Ware added the comment:
Output attached.
--
Added file: http://bugs.python.org/file34231/20501.output
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20501
___
Serhiy Storchaka added the comment:
Same as on Linux (and differs from 3.x). Thank you Zachary.
Here is corrected patch. Added also a test which tests that readline() doesn't
read whole file.
--
Added file: http://bugs.python.org/file34232/fileinput_hook_encoded_3.patch
Changes by Serhiy Storchaka storch...@gmail.com:
Removed file: http://bugs.python.org/file33934/fileinput_hook_encoded.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20501
___
Changes by Serhiy Storchaka storch...@gmail.com:
Removed file: http://bugs.python.org/file34222/fileinput_hook_encoded_2.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20501
___
Zachary Ware added the comment:
New patch passes on Windows. Without the patch to fileinput.py, the new
hook_encoded tests pass and the new test_readline test fails (as expected).
--
___
Python tracker rep...@bugs.python.org
Roundup Robot added the comment:
New changeset 1a1a9d6fb278 by Serhiy Storchaka in branch '2.7':
Issue #20501: fileinput module no longer reads whole file into memory when using
http://hg.python.org/cpython/rev/1a1a9d6fb278
New changeset b4a139713b3b by Serhiy Storchaka in branch '3.3':
Added
Changes by Serhiy Storchaka storch...@gmail.com:
--
assignee: docs@python - serhiy.storchaka
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20501
___
Serhiy Storchaka added the comment:
Thank you Zachary that ran tests for me.
Thank you Gunnar for your report.
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
Vajrasky Kok added the comment:
From http://hg.python.org/cpython/rev/1a1a9d6fb278
+t1 = TESTFN
+#t1 = writeTmp(1, ['A\nB\r\nC\rD+IKw-'], mode='wb')
+self.addCleanup(safe_unlink, TESTFN)
You left out the debugging statement.
And this English statement:
# Unlikely
Zachary Ware added the comment:
Can anyone please test the patch on Windows?
It seems to work; memory usage is much lower with the patch than without using
an 8 MB file. I don't notice any behavioral change with the patch; if there's
anything specific to look for on that front, give me a
Serhiy Storchaka added the comment:
Thank you Zachary.
Here is a patch with a test. I'm not sure that it is successful on Windows.
--
Added file: http://bugs.python.org/file34222/fileinput_hook_encoded_2.patch
___
Python tracker
Zachary Ware added the comment:
The new test passes on Windows with the whole patch applied, but fails without
the changes to fileinput.py. Is this change meant to fix behavior, or just the
memory usage issue?
Just for completeness, here's the failure output (with unpatched fileinput.py):
Serhiy Storchaka added the comment:
What would be the failure output when comment out first 1, 2 or three checks?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20501
___
Serhiy Storchaka added the comment:
Can anyone please test the patch on Windows?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20501
___
___
Gunnar Aastrand Grimnes added the comment:
Agreed that a doc-fix is the first step.
Shall I make another issue for the fix in fileinput?
I also wonder if the comment in codecs is not too negative - it is not easy to
get it 100% right, but the method just above readlines is readline, which
Serhiy Storchaka added the comment:
As far as the issue is in the codecs module, this is 2.7 only issue, because
codecs.open is used only in 2.7. And I suppose that replacing codecs.open by
io.open should fix this issue.
--
keywords: +patch
nosy: +benjamin.peterson, hynek, pitrou,
New submission from Gunnar Aastrand Grimnes:
When reading large files with fileinput, it will work as expected and only
process a line at a time when used normally, but if you add an hook_encoded
openhook it will read the whole file into memory before returning the first
line.
Verify by
Gunnar Aastrand Grimnes added the comment:
The problem lies in codecs.py here:
http://hg.python.org/cpython/file/ae7facd874ba/Lib/codecs.py#l581
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20501
R. David Murray added the comment:
This requires (1) a doc update to indicate the problem and (2) a way to tell
fileinput to *not* use readlines to optimize by calling readlines(bufsize),
since in the case of using codecs it can be a problem. Presumably buffer=None
would be the logical way
24 matches
Mail list logo
