[issue12855] linebreak sequences should be better documented

[Martin Panter]

Changes by Martin Panter :


--
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2e6fda267a20 by Martin Panter in branch '2.7':
Issue #12855: Document what exactly unicode.splitlines() splits on
https://hg.python.org/cpython/rev/2e6fda267a20

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Alexander Schrijver]

Alexander Schrijver added the comment:

Martin: Yes, it does, thank you. Sorry, I didn't know you where waiting for my 
approval.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Martin Panter]

Martin Panter added the comment:

Alexander: does my latest patch linebreakdoc.v5.py2.7.patch address your 
concerns about the 2.7 documentation? If so, I can push it to the repository.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Martin Panter]

Martin Panter added the comment:

Here is an updated patch for Python 2, based on Benjamin’s commit, Matthew’s 
earlier py27 patch, and Alexander’s backport of related changes from Python 3. 
Let me know what you think.

--
nosy: +Alexander Schrijver
stage: resolved -> patch review
Added file: http://bugs.python.org/file43080/linebreakdoc.v5.py2.7.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Martin Panter]

Martin Panter added the comment:

Reopening to change the Python 2 documentation. A starting point may be 
Matthew’s patch and/or Alexander’s patch in Issue 22232.

--
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Roundup Robot]

Roundup Robot added the comment:

New changeset 6244a5dbaf84 by Benjamin Peterson in branch '3.4':
document what exactly str.splitlines() splits on (closes #12855)
https://hg.python.org/cpython/rev/6244a5dbaf84

New changeset 87af6deb5d26 by Benjamin Peterson in branch 'default':
merge 3.4 (#12855)
https://hg.python.org/cpython/rev/87af6deb5d26

--
nosy: +python-dev
resolution:  - fixed
stage: patch review - resolved
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Martin Panter]

Martin Panter added the comment:

Patch v4 adds back the reference to “universal newlines”. I did not alter the 
doc string, because I don’t think doc strings need to be as detailed as the 
main documentation.

--
Added file: http://bugs.python.org/file38748/linebreakdoc.v4.py3.5.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[R. David Murray]

R. David Murray added the comment:

SMRUTI: \f is the python escape code for the ASCII formfeed character.  It is 
the handling of that ASCII character (among others) that this issue is 
discussing.

--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Martin Panter]

Martin Panter added the comment:

Note to self, or anyone else handling this patch: See 
https://bugs.python.org/issue22232#msg225769 for further improvement ideas:

* Might be good to bring back the reference to universal newlines, but say it 
accepts additional line boundaries
* Terry also suggested a doc string improvement

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[SMRUTI RANJAN SAHOO]

SMRUTI RANJAN SAHOO added the comment:

i think in this, line \fone\nline two\n ,the space after line taking some 
garbage value or you can say hex value of \.  so that's why that is showing 
some hex value. if you write \n  instead of\ then you can't find that hex 
value. i attached my idle image here.

--
nosy: +SMRUTI RANJAN SAHOO
Added file: http://bugs.python.org/file38523/python.JPG

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Martin Panter]

Martin Panter added the comment:

Posting linebreakdoc.v3.py3.5.patch:

* Rebased onto recent “default” (3.5) branch
* Add missing 1C–1E codes
* Dropped reference to “universal newlines”, since that only handles CRs and 
LFs as I understand it

The newlines are already tested by 
test_unicodedata.UnicodeMiscTest.test_linebreak_7643() when the VT and FF codes 
were added in Issue 7643.

--
Added file: http://bugs.python.org/file38179/linebreakdoc.v3.py3.5.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Martin Panter]

Martin Panter added the comment:

Any reason why characters 1C–1E are excluded?

--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Zachary Ware]

Changes by Zachary Ware zachary.w...@gmail.com:


--
stage:  - patch review
versions: +Python 3.4, Python 3.5 -Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[David Halter]

David Halter added the comment:

I would vote for the inclusion of that patch. I just stumbled over this.

--
nosy: +davidhalter

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

 Would it be better to put this note in a different place?

You may just say that StreamReader.readline() uses unicode.splitlines(), and so 
point to unicode.splitlines() doc (use :meth:`unicode.splitlines` syntax). 
unicode.splitlines() is now well documented: line boundaries are not listed, 
even in Python 3 documentation.

Unicode line boundaries used by Python 2.7 and 3.3:

U+000A: Line feed
U+000B: Line tabulation
U+000C: Form feed
U+000D: Carriage return
U+001C: File separator
U+001D: Group separator
U+001E: Record separator
U+0085: control
U+2028: Line separator
U+2029: Paragraph separator

 It looks like \x0b and \x0c (vertical tab and form feed) were first
 considered line breaks in Python 2.7

Correct: U+000B and U+000C were added to Python 2.7 and 3.2.

 It might be worth putting a changed in 2.7 note somewhere in the docs

We add the following syntax exactly for this:

   .. versionchanged:: 2.6
  Also unset environment variables when calling :meth:`os.environ.clear`
  and :meth:`os.environ.pop`.

If you downloaded Python source code, go into Doc/ directory and run make 
html to compile the doc to HTML.

http://docs.python.org/devguide/setup.html
http://docs.python.org/devguide/docquality.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[STINNER Victor]

Changes by STINNER Victor victor.stin...@haypocalc.com:


--
components: +Unicode
versions: +Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Matthew Boehm]

Matthew Boehm boehm.matt...@gmail.com added the comment:

I can fix the patch to list all the unicode line boundaries. The three places 
I've considered putting it are:

1. On the howto/unicode.html

2. Somewhere in the stdtypes.html#typesseq description (maybe with other notes 
at the bottom)

3. As a note to the stdtypes.html#str.splitlines method description (where it 
is in the previous patch.)

I can move it to any of these places if you think it's a better fit. I'll fix 
the list so that it's complete, add a note about \x0b and \x0c being added in 
2.7/3.2, and possibly reference it from StreamReader.readline.

After confirming that my documentation matches the style guide, I'll make the 
docs, test the output, and upload a patch. I can do this for 2.7, 3.2 and 3.3 
separately.

Let me know if that sounds good and if you have any further thoughts. I should 
be able to upload new patches in 10 hours (after work today).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

 1. On the howto/unicode.html
 2. Somewhere in the stdtypes.html#typesseq description (maybe with other 
 notes at the bottom)
 3. As a note to the stdtypes.html#str.splitlines method description (where it 
 is in the previous patch.)

(3) is the best place. For Python 2, you should add a new unicode.splitlines 
entry, whereas the str.splitlines should be updated in Python 3.

 I can do this for 2.7, 3.2 and 3.3 separately.

You don't have to do it for 3.3: 2.7 and 3.2 are enough (I will do the change 
in 3.3 using Mercurial).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Matthew Boehm]

Matthew Boehm boehm.matt...@gmail.com added the comment:

I've attached a patch for 2.7 and will attach one for 3.2 in a minute.

I built the docs for both 2.7 and 3.2 and verified that there were no warnings 
and that the resulting web pages looked okay.

Things to consider:

* Placement of unicode.splitlines() method: I placed it next to str.splitlines. 
I didn't want to place it with the unicode methods further down because docs 
say The following methods are present only on unicode objects

* The docs for codecs.readlines() already mentions Line-endings are 
implemented using the codec’s decoder method and are included in the list 
entries if keepends is true. 

* Feel free to make any wording/style suggestions.

--
Added file: http://bugs.python.org/file23076/linebreakdoc.v2.py27.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Matthew Boehm]

Changes by Matthew Boehm boehm.matt...@gmail.com:


Added file: http://bugs.python.org/file23077/linebreakdoc.v2.py32.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12855] linebreak sequences should be better documented

[Matthew Boehm]

Matthew Boehm boehm.matt...@gmail.com added the comment:

I've attached a patch for python2.7 that adds a small not to 
library/stdtypes.html#str.splitlines explaining which sequences are treated as 
line breaks:


Note: Python recognizes \r, \n, and \r\n as line boundaries for strings.

In addition to these, Unicode strings can have line boundaries of u\x0b, 
u\x0c, u\x85, u\u2028, and u\u2029


Additional thoughts:

* Would it be better to put this note in a different place?

* It looks like \x0b and \x0c (vertical tab and form feed) were first 
considered line breaks in Python 2.7, probably related to this note from 
What's New in 2.7: The Unicode database provided by the unicodedata module 
is now used internally to determine which characters are numeric, whitespace, 
or represent line breaks. It might be worth putting a changed in 2.7 note 
somewhere in the docs.

Please let me know of any thoughts you have and I'll be glad to make any 
desired changes and submit a new patch.

--
keywords: +patch
title: open() and codecs.open() treat form-feed differently - linebreak 
sequences should be better documented
Added file: http://bugs.python.org/file23069/linebreakdoc.py27.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12855
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com