[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Petri Lehtinen]

Changes by Petri Lehtinen :


--
nosy:  -petri.lehtinen

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Christopher Hoadley]

Changes by Christopher Hoadley :


--
nosy: +hoadlck
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[R. David Murray]

R. David Murray added the comment:

I think I'll leave it up to whoever works on this whether they want to tackle 
making posix mode and non-posix mode return the same values or turn this into 
an enhancement ticket for the proposed wrapped_lineno.  Or, if no one is 
interested, we can just close this.  (Or, third option, turn it into a doc 
issue and document exactly what shlex actually does with lineno).

--
stage: commit review -> needs patch
versions: +Python 3.5 -Python 3.2, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[R. David Murray]

R. David Murray added the comment:

Is it necessarily a bug if the behavior is different with posix=True or False?  
If I understand correctly, non-posix mode is a backward compatibility mode, and 
really nobody "should" be using non-posix mode any more :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Birk Nilson]

Birk Nilson added the comment:

After investigating the issue I have a couple of proposals.

Although a bit vague, the documentation of shlex.lineo seems to suggest that it 
should be incremented immediately on finding a newline character. Changing this 
to allow wrapped lines within a token without incrementing the line number 
changes the existing shlex API. Something I believe should be avoided since 
netrc relies on the existing behavior and third-party modules might too.

Instead I recommend the following steps.
Step #1: Fix the immediate issue of getting different line numbers for the same 
input depending on whether posix=(True|False), but keep the current - greedy - 
behavior of shlex.lineo.
Step #2: A separate patch introduces shlex.wrapped_lineo which does not 
increment the lineno immediately, but prior to reading the next token - as 
introduced in my previous patch.

Step #2 should arguably be introduced in a separate issue - if at all - since 
it is a new feature to the shlex API.

I will provide a patch for #1 within the next day or two along with one for #2 
if you guys think it is a good idea.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Petri Lehtinen]

Petri Lehtinen added the comment:

> I'm investigating it now and will get back with a revised &
> fully tested patch.

Sounds good. FWIW, it only broke on 3.x and default branches, and you can 
probably reproduce it on my own machine, too, by applying the patch and then 
running test_netrc.

This is what shlex documentation says about lineno:

> shlex.lineno
> Source line number (count of newlines seen so far plus one).

This is a bit vague, as it doesn't really state whether newlines should be 
treated "greedily" (consumbed before the next token is read) or "lazily" 
(consumed only when the next token is read).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Birk Nilson]

Birk Nilson added the comment:

Sorry about that. Rookie mistake. I'm investigating it now and will get back 
with a revised & fully tested patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Petri Lehtinen]

Petri Lehtinen added the comment:

Reverted the patch because of the broken netrc tests. This has to be 
investigated further.

--
resolution: fixed -> 
stage: committed/rejected -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Roundup Robot]

Roundup Robot added the comment:

New changeset 34f759fa5484 by Petri Lehtinen in branch '2.7':
Revert "Issue #16121: Fix line number accounting in shlex"
http://hg.python.org/cpython/rev/34f759fa5484

New changeset cda4a9dc415a by Petri Lehtinen in branch '3.2':
Revert "Issue #16121: Fix line number accounting in shlex"
http://hg.python.org/cpython/rev/cda4a9dc415a

New changeset 15f3fd6070b7 by Petri Lehtinen in branch '3.3':
Revert "Issue #16121: Fix line number accounting in shlex"
http://hg.python.org/cpython/rev/15f3fd6070b7

New changeset 1b0a6c1f8a08 by Petri Lehtinen in branch 'default':
Revert "Issue #16121: Fix line number accounting in shlex"
http://hg.python.org/cpython/rev/1b0a6c1f8a08

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[R. David Murray]

R. David Murray added the comment:

Something in this change seems to have broken netrc:

http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.3/builds/520

--
nosy: +r.david.murray
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Petri Lehtinen]

Petri Lehtinen added the comment:

Applied, thanks!

--
nosy: +petri.lehtinen
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Roundup Robot]

Roundup Robot added the comment:

New changeset e54ee8d2c16b by Petri Lehtinen in branch '2.7':
Issue #16121: Fix line number accounting in shlex
http://hg.python.org/cpython/rev/e54ee8d2c16b

New changeset f1d19fdb254f by Petri Lehtinen in branch '3.2':
Issue #16121: Fix line number accounting in shlex
http://hg.python.org/cpython/rev/f1d19fdb254f

New changeset 560e53fcf2b0 by Petri Lehtinen in branch '3.3':
Issue #16121: Fix line number accounting in shlex
http://hg.python.org/cpython/rev/560e53fcf2b0

New changeset f48c3c7a3205 by Petri Lehtinen in branch 'default':
Issue #16121: Fix line number accounting in shlex
http://hg.python.org/cpython/rev/f48c3c7a3205

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Birk Nilson]

Birk Nilson added the comment:

The implementation incremented the line number immediately when a newline was 
detected, even before the token had been processed completely - causing the 
issue Arfrever posted.

This also caused the unexpected behavior of a tokens line number including the 
amount of lines within the token itself. In other words '"a \n b \n c" \n d' 
would result in the token "a \n b \n c" to have the line number #3, followed by 
"d" being on the expected line #4. In my patch, the expected behavior of seeing 
the first token on line #1 and the second on #4 is introduced.

I also added the testLineNumbers method in the test suite - included in the 
patch.

--
keywords: +patch
nosy: +birknilson
Added file: http://bugs.python.org/file29207/issue16121.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Ezio Melotti]

Changes by Ezio Melotti :


--
nosy: +ezio.melotti
stage:  -> needs patch
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16121] shlex.shlex.error_leader() reports incorrect line number

[Arfrever Frehtes Taifersar Arahesis]

New submission from Arfrever Frehtes Taifersar Arahesis:

shlex.shlex.error_leader() reports incorrect line number with posix=True or 
when last token is not quoted.
This bug occurs in all versions of Python.

The attached script shows this bug:
$ ./shlex_test.py
### text1 posix=False
('var1', '"None", line 1: ')
('=', '"None", line 1: ')
('"x"', '"None", line 1: ')
('var2', '"None", line 2: ')
('=', '"None", line 2: ')
('"y"', '"None", line 2: ')
('var3', '"None", line 3: ')
('=', '"None", line 3: ')
('"z"', '"None", line 3: ')
### text1 posix=True
('var1', '"None", line 1: ')
('=', '"None", line 1: ')
('x', '"None", line 2: ')
('var2', '"None", line 2: ')
('=', '"None", line 2: ')
('y', '"None", line 3: ')
('var3', '"None", line 3: ')
('=', '"None", line 3: ')
('z', '"None", line 3: ')
### text2 posix=False
('var1', '"None", line 1: ')
('=', '"None", line 1: ')
('x', '"None", line 2: ')
('var2', '"None", line 2: ')
('=', '"None", line 2: ')
('y', '"None", line 3: ')
('var3', '"None", line 3: ')
('=', '"None", line 3: ')
('z', '"None", line 3: ')
### text2 posix=True
('var1', '"None", line 1: ')
('=', '"None", line 1: ')
('x', '"None", line 2: ')
('var2', '"None", line 2: ')
('=', '"None", line 2: ')
('y', '"None", line 3: ')
('var3', '"None", line 3: ')
('=', '"None", line 3: ')
('z', '"None", line 3: ')
### text3 posix=False
('"x"', '"None", line 1: ')
('"y"', '"None", line 2: ')
('"z"', '"None", line 3: ')
### text3 posix=True
('x', '"None", line 2: ')
('y', '"None", line 3: ')
('z', '"None", line 3: ')
### text4 posix=False
('x', '"None", line 2: ')
('y', '"None", line 3: ')
('z', '"None", line 3: ')
### text4 posix=True
('x', '"None", line 2: ')
('y', '"None", line 3: ')
('z', '"None", line 3: ')

Only "text1 posix=False" and "text3 posix=False" have all correct line numbers.

--
components: Library (Lib)
files: shlex_test.py
messages: 171905
nosy: Arfrever, zmedico
priority: normal
severity: normal
status: open
title: shlex.shlex.error_leader() reports incorrect line number
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4
Added file: http://bugs.python.org/file27404/shlex_test.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com