[issue20844] coding bug remains in 3.3.5rc2

[eryksun]

eryksun added the comment:

This fix for issue 20731 doesn't address this bug completely because it's 
possible for ftell to return -1 without an actual error, as test2.py 
demonstrates. 

In text mode, CRLF is translated to LF by the CRT's _read function (Win32 
ReadFile). So the buffer that's used by FILE streams is already translated. To 
get the stream position, ftell first calls _lseek (Win32 SetFilePointer) to get 
the file pointer. Then it adjusts the file pointer for the unwritten/unread 
bytes in the buffer. The problem for reading is how to tell whether or not LF 
in the buffer was translated from CRLF? The chosen 'solution' is to just assume 
CRLF.

The example file test2.py is 33 bytes. At the time fp_setreadl calls 
ftell(tok->fp), the file pointer is 33, and Py_UniversalNewlineFgets has read 
the stream up to '#coding:latin-1\n'. That leaves 17 newline characters 
buffered. As stated above, ftell assumes CRLF, so it calculates the stream 
position as 33 - (17 * 2) == -1. That happens to be the value returned for an 
error, but who's checking? In this case, errno is 0 instead of the documented 
errno constants EBADF or EINVAL.

Here's an example in 2.7.7, since it uses FILE streams:

>>> f = open('test2.py')
>>> f.read(16)
'#coding:latin-1\n'
>>> f.tell()
Traceback (most recent call last):
  File "", line 1, in 
IOError: [Errno 0] Error

Can the file be opened in binary mode in Modules/main.c? Currently it's using 
`_Py_wfopen(filename, L"r")`. But decoding_fgets calls 
Py_UniversalNewlineFgets, which expects binary mode anyway.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20844] coding bug remains in 3.3.5rc2

[Mark Lawrence]

Mark Lawrence added the comment:

I can reproduce this with 3.4.1 and 3.5.0.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20844] coding bug remains in 3.3.5rc2

[Marc Schlaich]

Marc Schlaich added the comment:

I can reproduce this one. There are a few conditions which needs to be met:

- Linux line endings 
- File needs to have at least x lines (empty lines are fine). I guess this is 
the point why no one could reproduce it. The attached file has 19 lines but 
probably no one copy/pasted the empty lines. Downloading the file reproduces 
this in my case. The length of the encoding declaration is relevant to the 
number of required newlines. `#coding:latin-1` fails at a file with 19 lines, 
`#coding: latin-1` (whitespace added) requires 20 lines.

More observations:

- Also reproducible if utf8 is used as alias for utf-8 (`#coding: utf8` + 17 
lines), but not reproducible with utf-8  
- Python 3.4 is affected, too
- No issues on Python 3.3.2

--
nosy: +schlamar
versions: +Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20844] coding bug remains in 3.3.5rc2

[Musashi Tamura]

Musashi Tamura added the comment:

When opening LF-newline file, ftell() may return zero when the position is not 
at the beginning of the file.

Maybe LF-newline file should open in binary-mode.
http://support.microsoft.com/kb/68337

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20844] coding bug remains in 3.3.5rc2

[Musashi Tamura]

Musashi Tamura added the comment:

Thanks Mark.

Perhaps, the problem is text-mode handling. When using Windows's text-mode 
stream, ftell() may return -1 even if no error occured.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20844] coding bug remains in 3.3.5rc2

[Mark Lawrence]

Mark Lawrence added the comment:

Works fine for me

--
nosy: +BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20844] coding bug remains in 3.3.5rc2

[Musashi Tamura]

Musashi Tamura added the comment:

It seems that this is not fixed in 3.3.5. Someone please reproduce it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20844] coding bug remains in 3.3.5rc2

[STINNER Victor]

STINNER Victor added the comment:

It's a duplicate of the issue #20731.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20844] coding bug remains in 3.3.5rc2

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20844] coding bug remains in 3.3.5rc2

[Musashi Tamura]

New submission from Musashi Tamura:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\bug>python
Python 3.3.5rc2 (v3.3.5rc2:ca5635efe090, Mar  2 2014, 18:18:29) [MSC v.1600 64 
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\bug>python test2.py
  File "test2.py", line 1
SyntaxError: encoding problem: iso-8859-1

--
components: Windows
files: test2.py
messages: 212637
nosy: miwa
priority: normal
severity: normal
status: open
title: coding bug remains in 3.3.5rc2
versions: Python 3.3
Added file: http://bugs.python.org/file34276/test2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com