[issue11594] 2to3 does not preserve line endings

[Jason R. Coombs]

Change by Jason R. Coombs :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Jason R. Coombs]

Jason R. Coombs  added the comment:

Slawomir, you're probably looking for issue34108.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Slawomir Nowaczyk]

Slawomir Nowaczyk  added the comment:

I've just tried 2to3 in 3.7.0, and it still breaks line endings for me...

Seems like one also needs to fix line 517 in refactor.py
  fp = io.open(filename, "w", encoding=encoding, newline='')

--
nosy: +Slawomir Nowaczyk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[miss-islington]

miss-islington  added the comment:


New changeset 3b3be1fe10f6c15e57360cac9d9dbc660666e655 by Miss Islington (bot) 
in branch '3.7':
bpo-11594: Ensure line-endings are respected when using 2to3 (GH-6483)
https://github.com/python/cpython/commit/3b3be1fe10f6c15e57360cac9d9dbc660666e655


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[miss-islington]

Change by miss-islington :


--
pull_requests: +6208

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset c127a86e1862df88ec6f9d15b79c627fc616766e by Łukasz Langa (Aaron 
Ang) in branch 'master':
bpo-11594: Ensure line-endings are respected when using 2to3 (GH-6483)
https://github.com/python/cpython/commit/c127a86e1862df88ec6f9d15b79c627fc616766e


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Aaron Ang]

Aaron Ang  added the comment:

@Jason R. Coombs
You are right. I managed to reproduce the problem with a test. It only occurs 
when a fix is applied.

Also, I figured out that the refactoring reads in the file using `open(file, 
'r')`, which basically transforms all line-endings to LF regardless the used 
line-endings. I think I fixed the problem, looking forward to receiving 
feedback 

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Aaron Ang]

Change by Aaron Ang :


--
keywords: +patch
pull_requests: +6181
stage: test needed -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Jason R. Coombs]

Jason R. Coombs  added the comment:

I do still see the issue on Python 3.7b3:

$ python ~/Dropbox/bin/scripts/which-line-ending onefile.py
Line ending is '\n'
$ python ~/Dropbox/bin/scripts/which-line-ending otherfile.py
Line ending is '\r\n'
$ python -V
Python 3.7.0b3
$ python -m lib2to3 . -w
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored ./onefile.py
--- ./onefile.py(original)
+++ ./onefile.py(refactored)
@@ -1 +1 @@
-print 'hello world'
+print('hello world')
RefactoringTool: Refactored ./otherfile.py
--- ./otherfile.py  (original)
+++ ./otherfile.py  (refactored)
@@ -1 +1 @@
-print 'hello world'
+print('hello world')
RefactoringTool: Files that were modified:
RefactoringTool: ./onefile.py
RefactoringTool: ./otherfile.py
$ $ python ~/Dropbox/bin/scripts/which-line-ending onefile.py
Line ending is '\n'
$ python ~/Dropbox/bin/scripts/which-line-ending otherfile.py
Line ending is '\n'

--
versions: +Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Aaron Ang]

Aaron Ang  added the comment:

I couldn't reproduce this issue. I tried reproducing this problem by extending 
the TestRefactoringTool class and creating two files: one file with LF 
line-endings and one file with CRLF line-endings.

The changes that I made can be found here: 
https://github.com/aaronang/cpython/commit/55e8bd317f37923e6e23780e6ae41858493e98d8.

The output of the tests:

Before: b'print("hi")\n\nprint("Like bad Windows newlines?")\n'
After:  b'print("hi")\n\nprint("Like bad Windows newlines?")\n'

Before: b'print("hi")\r\n\r\nprint("Like bad Windows newlines?")\r\n'
After:  b'print("hi")\r\n\r\nprint("Like bad Windows newlines?")\r\n'

Maybe this problem has been resolved?

--
nosy: +Aaron Ang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Łukasz Langa]

Change by Łukasz Langa :


--
keywords: +easy -patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Jason R. Coombs]

Jason R. Coombs  added the comment:

This issue affected me today. I'm editing a codebase that has mixed line 
endings in different files. I'd like to patch for Python 3 support without 
changing line endings. Even invoking a single fixer (print), the line endings 
mutate. Since I'm running on macOS, the files with CRLF get LF line endings.

Answers [in this 
question](https://stackoverflow.com/questions/39028517/2to3-how-to-keep-newline-characters-from-input-file)
 suggest the mutation can be suppressed by altering the _to_system_newlines 
function, but the proposed fix has no effect on the Python 3.6 version of this 
routine.

I thought I'd encountered this issue before, but I realized after searching 
that I was thinking of issue10639, which may present a model for retaining the 
newlines when refactoring the code.

I found I was able to work around the issue by putting lib2to3-clean.py in my 
current directory:

import lib2to3.refactor
import runpy
import functools


if __name__ == '__main__':
lib2to3.refactor._open_with_encoding = functools.partial(open, 
newline='')
runpy.run_module('lib2to3')


And invoking `python -m lib2to3-clean` instead of `-m lib2to3`. The addition of 
newline='' follows the [guidance in the 
docs](https://docs.python.org/release/3.2/library/functions.html#open) on how 
to avoid mutating newlines.

I've released this functionality in [jaraco.develop 
4.0](https://pypi.org/project/jaraco.develop) so others can readily invoke it 
with `rwt jaraco.develop -- -m jaraco.develop.lib2to3`.

--
nosy: +jason.coombs
versions: +Python 3.6 -Python 2.7, Python 3.2, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

I was surprised to see that the crlf.py file was not using CRLF in the new 
Mercurial repo.  It is also not in the .hgeol file.  I changed it locally, but 
it doesn’t change anything, the tests pass before and after the change.

--
keywords: +patch
versions:  -Python 3.1
Added file: http://bugs.python.org/file22292/2to3-crlf-test.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11594
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

Thanks.  Would you like to work on a unit test or full patch?

--
stage:  - test needed
title: 2to3 tool does not preserve line-endings - 2to3 does not preserve line 
endings

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11594
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Alexander Belchenko]

Alexander Belchenko bia...@ukr.net added the comment:

Éric, thank you for the proposal, but I'm not familiar enough with the codebase 
to work on it. 

The short scan over the tests reveals that there is at least one test which 
tries to test CRLF behavior, in the file test_refactor.py, but I don't 
understand what it doing?

def test_crlf_newlines(self):
old_sep = os.linesep
os.linesep = \r\n
try:
fn = os.path.join(TEST_DATA_DIR, crlf.py)
fixes = refactor.get_fixers_from_package(lib2to3.fixes)
self.check_file_refactoring(fn, fixes)
finally:
os.linesep = old_sep

So, in theory I can modify that test to check what if the file has LF-only 
line-endings originally, but os.linesep is CRLF, but then I don't know what the 
content I should create and how to run fixer over that.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11594
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11594] 2to3 does not preserve line endings

[Benjamin Peterson]

Benjamin Peterson benja...@python.org added the comment:

I can fix it. I just need to find time. :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11594
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com