[issue18406] unicodedata.itergraphemes / str.itergraphemes / str.graphemes

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Issue30717 has a patch.

--
resolution:  -> duplicate
stage: needs patch -> resolved
status: open -> closed
superseder:  -> Add unicode grapheme cluster break algorithm

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18406] unicodedata.itergraphemes / str.itergraphemes / str.graphemes

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +serhiy.storchaka
versions: +Python 3.7 -Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18406] unicodedata.itergraphemes / str.itergraphemes / str.graphemes

[Socob]

Changes by Socob <206a8...@opayq.com>:


--
nosy: +Socob

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18406] unicodedata.itergraphemes / str.itergraphemes / str.graphemes

[Matthew Barnett]

Matthew Barnett added the comment:

This is basically what the regex module does, written in Python:

def get_grapheme_cluster_break(codepoint):
"""Gets the "Grapheme Cluster Break" property of a codepoint.

The properties defined here:


http://www.unicode.org/Public/UNIDATA/auxiliary/GraphemeBreakProperty.txt
"""
# The return value is one of:
#
# "Other"
# "CR"
# "LF"
# "Control"
# "Extend"
# "Prepend"
#  "Regional_Indicator"
# "SpacingMark"
# "L"
# "V"
# "T"
# "LV"
# "LVT"
...

def at_grapheme_boundary(string, index):
"""Checks whether the codepoint at 'index' is on a grapheme boundary.

The rules are defined here:

http://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries
"""
# Break at the start and end of the text.
if index <= 0 or index >= len(string):
return True

prop = get_grapheme_cluster_break(string[index])
prop_m1 = get_grapheme_cluster_break(string[index - 1])

# Don't break within CRLF.
if prop_m1 == "CR" and prop == "LF":
return False

# Otherwise break before and after controls (including CR and LF).
if prop_m1 in ("Control", "CR", "LF") or prop in ("Control", "CR", 
"LF"):
return True

# Don't break Hangul syllable sequences.
if prop_m1 == "L" and prop in ("L", "V", "LV", "LVT"):
return False
if prop_m1 in ("LV", "V") and prop in ("V",  "T"):
return False
if prop_m1 in ("LVT", "T") and prop == "T":
return False

# Don't break between regional indicator symbols.
if (prop_m1 == "REGIONALINDICATOR" and prop ==
  "REGIONALINDICATOR"):
return False

# Don't break just before Extend characters.
if prop == "Extend":
return False

# Don't break before SpacingMarks, or after Prepend characters.
if prop == "SpacingMark":
return False

if prop_m1 == "Prepend":
return False

# Otherwise, break everywhere.
return True

--
nosy: +mrabarnett

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18406] unicodedata.itergraphemes / str.itergraphemes / str.graphemes

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

It may be useful to also add the start position of the grapheme to the iterator 
output.

Related to this, please also see this pre-PEP I once wrote for a Unicode 
indexing module:

http://mail.python.org/pipermail/python-dev/2001-July/015938.html

--
components: +Unicode
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18406] unicodedata.itergraphemes / str.itergraphemes / str.graphemes

[Chris Rebert]

Changes by Chris Rebert :


--
nosy: +cvrebert

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18406] unicodedata.itergraphemes / str.itergraphemes / str.graphemes

[Ezio Melotti]

Changes by Ezio Melotti :


--
nosy: +benjamin.peterson, ezio.melotti, loewis
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18406] unicodedata.itergraphemes / str.itergraphemes / str.graphemes

[David P. Kendal]

New submission from David P. Kendal:

On python-ideas I proposed the addition of a way to iterate over the graphemes 
of a string, either as part of the unicodedata library or as a method on the 
built-in str type. 


I provided a sample implementation, but "MRAB" pointed out that my definition 
of a grapheme is slightly wrong; it's a little more complex than just 
"character followed by combiners". 


M.-A. Lenburg asked me to open this issue. 


--
messages: 192684
nosy: dpk
priority: normal
severity: normal
status: open
title: unicodedata.itergraphemes / str.itergraphemes / str.graphemes
type: enhancement
versions: Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com