[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 4d840e428ab1a2712f219c5e4008658cbe15892e by Miss Islington (bot) 
in branch '3.8':
[3.8] bpo-42318: Fix support of non-BMP characters in Tkinter on macOS 
(GH-23281). (GH-23784) (GH-23787)
https://github.com/python/cpython/commit/4d840e428ab1a2712f219c5e4008658cbe15892e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 28bf6ab61f77c69b732a211c398ac882bf3f65f4 by Serhiy Storchaka in 
branch '3.9':
[3.9] bpo-42318: Fix support of non-BMP characters in Tkinter on macOS 
(GH-23281). (GH-23784)
https://github.com/python/cpython/commit/28bf6ab61f77c69b732a211c398ac882bf3f65f4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 5.0 -> 6.0
pull_requests: +22644
pull_request: https://github.com/python/cpython/pull/23787

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +22641
pull_request: https://github.com/python/cpython/pull/23784

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Oh, the fix is not backported yet.

Automatically backporting does not work because of renames in the supporting 
test library.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset a26215db11cfcf7b5f55cab9e91396761a0e0bcf by Serhiy Storchaka in 
branch 'master':
bpo-42318: Fix support of non-BMP characters in Tkinter on macOS (GH-23281)
https://github.com/python/cpython/commit/a26215db11cfcf7b5f55cab9e91396761a0e0bcf


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +22175
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/23281

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Yash, this is specifically a macOS issue.  Printing astral chars in 
tkinter/IDLE on Windows and Linux has 'worked' (details not important) for over 
a year.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Ronald Oussoren]

Ronald Oussoren  added the comment:

> And yet one question. What do you see if you print '\udcf0\udc9f\udc98\udc80' 
> in IDLE?

This prints a smiley emoji, likewise for printing chr(128516)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

And yet one question. What do you see if you print '\udcf0\udc9f\udc98\udc80' 
in IDLE?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

You can ignore msg380917. It was written before I read msg380908. Now I have 
the needed information. Thank you.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Ronald Oussoren]

Ronald Oussoren  added the comment:

When I assign root.clipboard_get() to "v" I get:

>>> print(ascii(v))
'\udced\udca0\udcbd\udced\udcb8\udc84'
>>> print(v)
??

This is with Tk 8.6.10.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Well, try copy  (or other text with color emoji) to clipboard and run the 
following code:

import tkinter
root = tkinter.Tk()
print(ascii(root.clipboard_get()))

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Ronald Oussoren]

Ronald Oussoren  added the comment:

BTW. The unicodeFromTclStringAndSize() basically undoes the special treatment 
of \0 in Modified UTF-8 [1]. That page says that all known implementation of 
MUTF-8 treat surrogate pairs the same as CESU-8 [2], which is UTF-8 with 
characters outside of the BMP encoded as surrogate pairs which are then 
converted to UTF-8.

Neither encoding is currently supported by Python.

[1] https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8
[2] https://en.wikipedia.org/wiki/CESU-8

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Ronald Oussoren]

Ronald Oussoren  added the comment:

@Pixmew: I get this error with Tk 8.6.10 on macOS 11. With Tk 8.6.8 on macOS 
10.15 (from the python.org installer) I get the behaviour described in 
msg380906.

8.6.10 is the version of Tk we'd like to switch to for the "universal2", it is 
the latest release in the 8.6.x branch and contains numerous bug fixes.

The "Intel" installers (the ones currently on Python.org) we'll continue to use 
Tk 8.6.8 due to build issues on macOS 10.9 with newer Tk versions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Ronald Oussoren]

Ronald Oussoren  added the comment:

With 8.6.10:

>>> print(ascii("")) raises the SyntaxError mentioned earlier
>>> print(ascii(input())) works and prints:
'\udced\udca0\udcbd\udced\udcb8\udc84'

In an editor window I don't get spurious text, but syntax colouring is a bit 
off: The text after the closing quote is coloured as if it is inside the string 
literal. That continues for the characters on the next line

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Yash Shete]

Yash Shete  added the comment:

Well for me in Python 3.9.0
print("")   prints   and 
 print(ascii(""))   prints '\U0001f600'

It does not Raises error "utf-8' codec can't encode characters in position 
7-12: surrogates not allowed." as you are suggesting

--
nosy: +Pixmew

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Ronald Oussoren]

Ronald Oussoren  added the comment:

With 8.6.8 both "hang", in that the Shell window no longer accepts input. I've 
checked that ``print(input())`` works when I don't use an emoji. 

Interestingly enough, pasting ``print(ascii(""))`` into an edit window does 
work, I can continue editing, but the display is messed up. It looks like:

   print(ascii(""))print(ascii("

But with the first two identifiers coloured and the two other identifiers 
black. Saving the file results in the expected file contents.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Well, it is likely the same syntax error. Then what will print

print(ascii(input()))

when you paste  and press Enter?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Just to be sure, what is the result of pasting and executing the following code 
on Tk 8.6.8 and 8.6.10?

print(ascii(""))

--
nosy: +serhiy.storchaka
versions: +Python 3.10, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42318] [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a text widget

[Ronald Oussoren]

New submission from Ronald Oussoren :

As mentioned in msg380552: I get an SyntaxError with message "utf-8' codec 
can't encode characters in position 7-12: surrogates not allowed." when I paste 
a smiley emoji in an IDLE interactive shell and try to execute that line, for 
example using:

>>> print("")

The error is likely due to a surrogate pair being present in the UTF-8 
representation of a Tcl/Tk string.

It should be possible to work around this in 
_tkinter.c:unicodeFromTclStringAndSize by merging surrogate pairs. 

This is with:
- Python 3.10
- macOS 11 (arm64)
- Tk 8.6.10

With Tk 8.6.8 (as included in the macOS installers on python.org) printing 
won't work at all, as mentioned in bpo-42225.

--
components: Tkinter, macOS
messages: 380715
nosy: ned.deily, ronaldoussoren
priority: normal
severity: normal
stage: needs patch
status: open
title: [tkinter] surrogate pairs in Tcl/Tk string when pasting an emoji in a 
text widget
type: behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com