[issue2384] [Py3k] line number is wrong after encoding declaration

2008-10-09 Thread Amaury Forgeot d'Arc 

Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:

Committed r66867. 

I had to considerably change the unit tests, because the subprocess 
output is not utf-8 encoded; it's not even the same as sys.stdout, 
because the spawned process uses a PIPE, not a terminal: on my winXP, 
the main interpreter uses cp437, but the subprocess says cp1252. So I 
first run a 'python -c print(sys.stdout.encoding)' in the same 
conditions just to retrieve the encoding. fun fun.
I hope this still works on Unixes, will watch the buildbots.

--
resolution:  - fixed
status: open - pending

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-10-07 Thread Amaury Forgeot d'Arc 

Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:

Your patch does the correct thing, however an explanation of the -1
value would be welcome. Something like:
/* The file has been reopened; parsing will restart from 
 * the beginning of the file, we have to reset the line number.
 * But this function has been called from inside tok_nextc() which 
 * will increment lineno before it returns. So we set it -1 so that
 * the next call to tok_nextc() will start with tok-lineno == 0.
 */

Or we could change the place of the tok-lineno++ in tok_nextc() so that
it is called before the call to decoding_fgets(); other changes will be
needed.

Then, I think that your test is not correct: What is the meaning of the
following line?
sys.exit(traceback.tb_lineno(sys.exc_info()[2]))
(the module traceback has no attribute tp_lineno)
I presume that you intended something like:
traceback.print_exc()
sys.exit(sys.exc_info()[2].tb_lineno)
and test at some point that process.returncode == lineno

--
nosy: +amaury.forgeotdarc

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-10-07 Thread STINNER Victor 

STINNER Victor [EMAIL PROTECTED] added the comment:

@amaury: Ok, I added your long comment in tokenizer.c. You're also 
right about the strange code in the test. I reused ocean-city's 
test. sys.exc_info()[2].tb_lineno raises an additional (useless) 
error. So I simplified the code to use only raise RuntimeError(...) 
with the try/except/else.

Since tokenizer.c is hard to understand, I don't wnat to change the 
code of tok_nextc().

Added file: http://bugs.python.org/file11737/traceback_unicode-4.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-10-07 Thread STINNER Victor 

Changes by STINNER Victor [EMAIL PROTECTED]:


Removed file: http://bugs.python.org/file11625/tokenizer-coding-2.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-10-07 Thread STINNER Victor 

Changes by STINNER Victor [EMAIL PROTECTED]:


Removed file: http://bugs.python.org/file11717/tokenizer-coding-3.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-10-07 Thread STINNER Victor 

Changes by STINNER Victor [EMAIL PROTECTED]:


Removed file: http://bugs.python.org/file11609/test_traceback-gbk.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-10-07 Thread STINNER Victor 

Changes by STINNER Victor [EMAIL PROTECTED]:


Removed file: http://bugs.python.org/file11737/traceback_unicode-4.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-10-07 Thread STINNER Victor 

Changes by STINNER Victor [EMAIL PROTECTED]:


Added file: http://bugs.python.org/file11738/tokenizer-coding-4.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-10-07 Thread Amaury Forgeot d'Arc 

Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:

This issue depends on #3975 to properly display tracebacks from python 
files with encoding.

--
dependencies: +PyTraceBack_Print() doesn't respect # coding: xxx header

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-10-06 Thread STINNER Victor 

STINNER Victor [EMAIL PROTECTED] added the comment:

benjamin was afraid by the comment /* dirty hack */ in my previous 
comment. After reading tokenizer.c again and again, I can say that the 
fix is correct: the file is closed and then re-opened by fp_setreadl() 
(using io.open()), and so the file cursor goes back to the file start.

Added file: http://bugs.python.org/file11717/tokenizer-coding-3.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-27 Thread STINNER Victor 

Changes by STINNER Victor [EMAIL PROTECTED]:


Removed file: http://bugs.python.org/file11610/tokenizer-coding.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-27 Thread STINNER Victor 

STINNER Victor [EMAIL PROTECTED] added the comment:

Issue 2832 is a duplicate.

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-26 Thread Antoine Pitrou 

Changes by Antoine Pitrou [EMAIL PROTECTED]:


--
nosy: +haypo

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-26 Thread Antoine Pitrou 

Antoine Pitrou [EMAIL PROTECTED] added the comment:

#3973 is a duplicate.

--
nosy: +pitrou

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-26 Thread STINNER Victor 

STINNER Victor [EMAIL PROTECTED] added the comment:

ocean-city testcase is invalid: it uses subprocess.call() which 
returns the exit code, not the Python error line number! Here is a 
better testcase using subprocess.Popen() checking the line number but 
also the display line. It tests ASCII, UTF-8 and GBK charsets. Using 
GBK charset, you get the bug described by ocean-city (problem with 
multibyte charset). My testcase takes also care of script with # 
coding at the second line.

Added file: http://bugs.python.org/file11609/test_traceback-gbk.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-26 Thread STINNER Victor 

STINNER Victor [EMAIL PROTECTED] added the comment:

Hum, about the empty line error using a multibyte charset, the issue 
is different. PyTraceBack_Print() calls _Py_DisplaySourceLine() which 
doesn't take care of the charset.

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-26 Thread STINNER Victor 

STINNER Victor [EMAIL PROTECTED] added the comment:

Here is a patch fixing this issue: it's quite the same that ocean-city 
patch, but I prefer to patch lineno only if set_readline() succeed.

About the truncated traceback for multibyte charset: see the new 
issue3975.

Added file: http://bugs.python.org/file11610/tokenizer-coding.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-26 Thread STINNER Victor 

STINNER Victor [EMAIL PROTECTED] added the comment:

Oh! My patch breaks python -m. The problem is maybe no in the token 
parser but... somewhere else?
--- test.py ---
# coding: ASCII
raise Exception(line 2)
# try again!
---

Python 3.0 trunk unpatched:
---
$ ./python test.py
Traceback (most recent call last):
  File test.py, line 3, in module

$ ./python -m test
Traceback (most recent call last):
  File /home/haypo/prog/py3k/Lib/runpy.py, line 121, in 
_run_module_as_main
__main__, fname, loader, pkg_name)
  File /home/haypo/prog/py3k/Lib/runpy.py, line 34, in _run_code
exec(code, run_globals)
  File /home/haypo/prog/py3k/test.py, line 2, in module
raise Exception(line 2)
Exception: line 2
---

Python 3.0 trunk + tokenizer-coding.patch:
---
marge$ ./python test.py
Traceback (most recent call last):
  File test.py, line 2, in module
raise Exception(line 2)
Exception: line 2

Traceback (most recent call last):
  File /home/haypo/prog/py3k/Lib/runpy.py, line 121, in 
_run_module_as_main
__main__, fname, loader, pkg_name)
  File /home/haypo/prog/py3k/Lib/runpy.py, line 34, in _run_code
exec(code, run_globals)
  File /home/haypo/prog/py3k/test.py, line 1, in module
# coding: ASCII
Exception: line 2
---

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-26 Thread Hirokazu Yamamoto 

Hirokazu Yamamoto [EMAIL PROTECTED] added the comment:

Victor, this is fp_setreadl's problem, so if put tok-lineno = -1
anywhere, it should be in fp_setreadl(), I think.

r = set_readline(tok, cs);
if (r) {
/* 1 */
tok-encoding = cs;
tok-decoding_state = STATE_NORMAL;

At /* 1 */, set_readline could be buf_setreadl(), and fp_setreadl is
called elsewhere.

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-26 Thread STINNER Victor 

STINNER Victor [EMAIL PROTECTED] added the comment:

@ocean-city: Oops, sorry. Using your patch (set lineno in
fp_setreadl()), it works on both cases (python test.py or python -m
test).

The new patch includes your fix for tokenizer.c and a new version of the
testcase.

Added file: http://bugs.python.org/file11625/tokenizer-coding-2.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-09-21 Thread Jean-Michel Fauth 

Jean-Michel Fauth [EMAIL PROTECTED] added the comment:

Python 3.0rc1

If the lines are now displayed correctly, I think there is still a
numbering issue, a +1 offset.

Python 2.5.2

# -*- coding: cp1252 -*-   line 1, first line

s = 'abc'
import dummy
s = 'def'

---

pythonw -u testpy2.py
Traceback (most recent call last):
  File testpy2.py, line 4, in module
import dummy
ImportError: No module named dummy
Exit code: 1


Python 3.0rc1

# -*- coding: cp1252 -*-

s = 'abc'
import dummy
s = 'def'

---

c:\python30\pythonw -u testpy3.py
Traceback (most recent call last):
  File testpy3.py, line 5, in module
s = 'def'
ImportError: No module named dummy
Exit code: 1

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-08-29 Thread Dwayne Litzenberger 

Dwayne Litzenberger [EMAIL PROTECTED] added the comment:

Could -*- coding: ascii -*- and other equivalent encodings be fixed,
at least, before the release?

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-08-26 Thread Dwayne Litzenberger 

Changes by Dwayne Litzenberger [EMAIL PROTECTED]:


--
nosy: +dlitz

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-08-18 Thread Jean-Michel Fauth 

Jean-Michel Fauth [EMAIL PROTECTED] added the comment:

Py3.0b2. This bug seems to be quite annoying. Especially when one works
with a main module importing modules which are importing modules and so
on, all modules having an encoding declaration. The Traceback (and the
user) is (are) a little bit lost.

-
# -*- coding: cp1252 -*-
# modb.py

def fb():
i = 1
j = 0
r =  i / j
---
# -*- coding: cp1252 -*-
# moda.py

import modb

def fa():
modb.fb()
---
# -*- coding: cp1252 -*-
# main.py

import moda

def main():
moda.fa()

if __name__ == '__main__':
main()
---

Running main.py leads to an

c:\python30\pythonw -u main.py
(Traceback (most recent call last):
  File main.py, line 11, in module

  File main.py, line 8, in main

  File C:\jm\jmpy3\moda.py, line 8, in fa

  File C:\jm\jmpy3\modb.py, line 8, in fb

ZeroDivisionError: int division or modulo by zero
Exit code: 1

--
nosy: +jmfauth

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-08-17 Thread Barry A. Warsaw 

Barry A. Warsaw [EMAIL PROTECTED] added the comment:

While this is a bug, it's not serious enough to hold up the release.

--
priority: release blocker - high

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-07-30 Thread Benjamin Peterson 

Changes by Benjamin Peterson [EMAIL PROTECTED]:


--
priority: critical - release blocker

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-06-11 Thread Barry A. Warsaw 

Barry A. Warsaw [EMAIL PROTECTED] added the comment:

This is a bug and not a new feature, so it could go in after beta.  I'm
knocking it down to a critical.

--
nosy: +barry
priority: release blocker - critical

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-05-18 Thread Georg Brandl 

Changes by Georg Brandl [EMAIL PROTECTED]:


--
priority:  - release blocker

__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-03-20 Thread Hirokazu Yamamoto 

Hirokazu Yamamoto [EMAIL PROTECTED] added the comment:

Following dirty hack workarounds this bug. Comment of this function
says not ascii compatible encoding is not supported yet, (ie: UTF-16)
so probably this works.

Index: Parser/tokenizer.c
===
--- Parser/tokenizer.c  (revision 61632)
+++ Parser/tokenizer.c  (working copy)
@@ -464,6 +464,7 @@
Py_XDECREF(tok-decoding_readline);
readline = PyObject_GetAttrString(stream, readline);
tok-decoding_readline = readline;
+   tok-lineno = -1; /* dirty hack */
 
   cleanup:
Py_XDECREF(stream);

But if multibyte character is in line like this, its line will not be
printed.

# coding: cp932
# 1
raise RuntimeError(あいうえお)
# 2

C:\Documents and Settings\WhiteRabbitpy3k cp932.py
Traceback (most recent call last):
  File cp932.py, line 3, in module
[22819 refs]

This is because Python/trackeback.c 's tb_displayline() assumes
input line is encoded with UTF-8. (simply using FILE structure +
Py_UniversalNewlineFgets)

# http://mail.python.org/pipermail/python-3000/2008-March/012546.html
# sounds nice, if we can replace all FILE structure to Python's own
# fast enough codeced Reader or something.

--
type:  - behavior

__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
__
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2384] [Py3k] line number is wrong after encoding declaration

2008-03-18 Thread Hirokazu Yamamoto 

New submission from Hirokazu Yamamoto [EMAIL PROTECTED]:

# This issue inherits from issue2301.

If there is # coding:  is in source code and
coding is neigher utf-8 nor iso-8859-1, line number (tok-lineno)
becomes wrong.

Please look into Parser/tokenizer.c. In this case, 
tok-decoding_state becomes STATE_NORMAL, so fp_setreadl
newly opens file but *doesn't* seek to current position.
(Or maybe can we reuse already opened file?)

So

# coding: ascii
# 1
# 2
# 3
raise RuntimeError(a)
# 4
# 5
# 6

outputs 

C:\Documents and Settings\WhiteRabbitpy3k ascii.py

Traceback (most recent call last):
  File ascii.py, line 6, in module
# 4
RuntimeError: a
[22821 refs]

One line shifted because line number wrongly +1

And

# dummy
# coding: ascii
# 1
# 2
# 3
raise RuntimeError(a)
# 4
# 5
# 6

outputs

C:\Documents and Settings\WhiteRabbitpy3k ascii.py

Traceback (most recent call last):
  File ascii.py, line 8, in module
# 5
RuntimeError: a
[22821 refs]

Two lines shifted because line number wrongly +2

--
components: None
messages: 63905
nosy: ocean-city
severity: normal
status: open
title: [Py3k] line number is wrong after encoding declaration
versions: Python 3.0

__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2384
__
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com