[issue5758] fileinput.hook_compressed returning bytes from gz file

Inada Naoki added the comment: > I did consider and confirm that mode="rb" does also provide a uniform > solution, but it also regresses the behavior of uncompressed inputs, making > them bytes where they were text. Of course, I suggested to use "rb" when you want to read bytes. * When read

[issue5758] fileinput.hook_compressed returning bytes from gz file

Jason R. Coombs added the comment: I did consider and confirm that mode="rb" does also provide a uniform solution, but it also regresses the behavior of uncompressed inputs, making them bytes where they were text. This approach feels like the "Python 1" compatibility approach. At least in m

[issue5758] fileinput.hook_compressed returning bytes from gz file

Inada Naoki added the comment: > I'm struggling with how to adapt the code to provide a uniform interface on > Python 3.6+. It's easy. Just passing `mode="rb"`. ``` FileInput(filelist, mode="rb", openhook=fileinput.hook_compressed) ``` This is better than `backports.hook_compressed`, becaus

[issue5758] fileinput.hook_compressed returning bytes from gz file

Jason R. Coombs added the comment: A backport now exists (https://pypi.org/project/backports.hook_compressed) and addresses the issue (https://github.com/jaraco/cmdix/actions/runs/873404846). -- ___ Python tracker

[issue5758] fileinput.hook_compressed returning bytes from gz file

Jason R. Coombs added the comment: The patch for this change has broken code that relied on the old behavior. For example: ``` import fileinput import bz2 import pathlib target = pathlib.Path('data.bz2') target.write_bytes(bz2.compress(b'Foo\nBar\nBiz')) inp = fileinput.FileInput([str(targe

[issue5758] fileinput.hook_compressed returning bytes from gz file

Change by Inada Naoki : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___ Py

[issue5758] fileinput.hook_compressed returning bytes from gz file

Change by Inada Naoki : -- nosy: +methane nosy_count: 5.0 -> 6.0 pull_requests: +24008 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/25272 ___ Python tracker _

[issue5758] fileinput.hook_compressed returning bytes from gz file

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.py

[issue5758] fileinput.hook_compressed returning bytes from gz file

Change by Inada Naoki : -- versions: +Python 3.10 -Python 3.2, Python 3.3 ___ Python tracker ___ ___ Python-bugs-list mailing list Un

[issue5758] fileinput.hook_compressed returning bytes from gz file

STINNER Victor added the comment: This issue is not newcomer friendly, I remove the easy keyword. -- keywords: -easy ___ Python tracker ___ ___

[issue5758] fileinput.hook_compressed returning bytes from gz file

Éric Araujo added the comment: > For 3.2 could we use the same fix, but without exposing the ability to > *change* the encoding? > That is, we use TextIOWrapper but always with the default None for encoding. Yes! > It also occurs to me that this really exposes a weakness in the design. What

[issue5758] fileinput.hook_compressed returning bytes from gz file

R. David Murray added the comment: It also occurs to me that this fix makes the charset hook look rather odd. We could render it redundant by passing charset to open in the non-openhook case, and mark it deprecated. There is also a bug in the hook_encoding docs. It says the file is opened w

[issue5758] fileinput.hook_compressed returning bytes from gz file

R. David Murray added the comment: For 3.2 could we use the same fix, but without exposing the ability to *change* the encoding? That is, we use TextIOWrapper but always with the default None for encoding. It also occurs to me that this really exposes a weakness in the design. What if the

[issue5758] fileinput.hook_compressed returning bytes from gz file

Éric Araujo added the comment: Thanks. Unless another core dev wants to do a complementary review I will slightly tweak the patch and commit it. I need to finish waking up and eat some food before I do that :) Technically adding a new argument means that this is a new feature and cannot be

[issue5758] fileinput.hook_compressed returning bytes from gz file

Tatiana Al-Chueyr added the comment: Improved patch, according to eric.araujo's suggestions and mnewman's guidance. -- Added file: http://bugs.python.org/file24815/issue5758_fileinput_gzip_with_encoding_v2.patch ___ Python tracker

[issue5758] fileinput.hook_compressed returning bytes from gz file

Éric Araujo added the comment: Nobody can remember everything: I tried in a shell and looked at the docs to find it :) -- ___ Python tracker ___

[issue5758] fileinput.hook_compressed returning bytes from gz file

R. David Murray added the comment: Duh. I should have remembered that the io module used None to represent the default encoding. Thanks, Éric. -- ___ Python tracker ___ __

[issue5758] fileinput.hook_compressed returning bytes from gz file

Tatiana Al-Chueyr added the comment: Excellent! Thanks for all remarks, Éric! I'll follow your suggestions and will submit a new patch. -- ___ Python tracker ___ ___

[issue5758] fileinput.hook_compressed returning bytes from gz file

Éric Araujo added the comment: We can use locale.getpreferredencoding, or be smarter and just pass down encoding=encoding to TextIOWrapper and let *it* call getpreferredencoding if it’s given a None :) -- ___ Python tracker

[issue5758] fileinput.hook_compressed returning bytes from gz file

R. David Murray added the comment: I think it should use the same default as open, but frankly I couldn't remember the right way to achieve that. Suggestions welcome :) -- nosy: +r.david.murray ___ Python tracker

[issue5758] fileinput.hook_compressed returning bytes from gz file

Éric Araujo added the comment: Thanks for the patch. I commented on not-so-important things on the code review site (which should have sent an email to Tatiana) and wanted to raise an issue here. Is it best that the encoding used defaults to Python’s default encoding (UTF-8) or the I/O defa

[issue5758] fileinput.hook_compressed returning bytes from gz file

Tatiana Al-Chueyr added the comment: I had a lot of help of R. David Murray, could someone review our patch, please? We are specially concerned about backward compatibility. -- keywords: +patch Added file: http://bugs.python.org/file24807/issue5758_fileinput_gzip_with_encoding.patch

[issue5758] fileinput.hook_compressed returning bytes from gz file

Tatiana Al-Chueyr added the comment: Working on this -- nosy: +tati_alchueyr ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue5758] fileinput.hook_compressed returning bytes from gz file

Amaury Forgeot d'Arc added the comment: gzip.open() only implements the "rb" mode, and returns bytes. fileinput could certainly wrap it with a io.TextIOWrapper. Then the encoding issue arises. fileinput.FileInput should grow an "encoding" parameter instead of always relying on the default enco

[issue5758] fileinput.hook_compressed returning bytes from gz file

Changes by Mark Lawrence : -- stage: -> needs patch versions: +Python 2.7, Python 3.1, Python 3.2 -Python 3.0 ___ Python tracker ___ _

[issue5758] fileinput.hook_compressed returning bytes from gz file

New submission from Michael Newman : The attached ZIP file contains "test.bat" which runs "test.py" with Python 2.6 and Python 3.0. Python 2.6 behaves as expected (see "py26.out"), since it returns strings from both "mike.txt" and "mike.txt.gz". However, the same test with Python 3.0 returns byt