Serhiy Storchaka <storch...@gmail.com> added the comment:
> So the answer to your last question is "yes". I hope that the answer to
> your other questions follows from that
Thank you, this is the answer to all my questions. I've prepared a patch
to treat U+FFFE in general mapping as “undefined mapping”.
> (strictly speaking, it's only
> U+FFFE, not 0xFFFE, that is documented as indicating an undefined
> mapping; a patch should probably fix that).
As both integer 0xXXXX and string '\uXXXX' denote U+XXXX, I do not think
it necessary fixes.
> (I also wonder where the support for LookupError comes from - that
> appears to be undocumented)
I believe, this is what is meant by the words "undefined mapping".
----------
keywords: +patch
Added file: http://bugs.python.org/file25934/decode_charmap_fffe.patch
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue14850>
_______________________________________
diff -r 743cf3319862 Objects/unicodeobject.c
--- a/Objects/unicodeobject.c Sat Jun 09 22:51:39 2012 -0700
+++ b/Objects/unicodeobject.c Sun Jun 10 21:49:34 2012 +0300
@@ -7512,15 +7512,18 @@
if (PyErr_ExceptionMatches(PyExc_LookupError)) {
/* No mapping found means: mapping is undefined. */
PyErr_Clear();
- x = Py_None;
- Py_INCREF(x);
+ goto Undefined;
} else
goto onError;
}
/* Apply mapping */
+ if (x == Py_None)
+ goto Undefined;
if (PyLong_Check(x)) {
long value = PyLong_AS_LONG(x);
+ if (value == 0xFFFE)
+ goto Undefined;
if (value < 0 || value > 65535) {
PyErr_SetString(PyExc_TypeError,
"character mapping must be in
range(65536)");
@@ -7530,21 +7533,6 @@
if (unicode_putchar(&v, &outpos, value) < 0)
goto onError;
}
- else if (x == Py_None) {
- /* undefined mapping */
- startinpos = s-starts;
- endinpos = startinpos+1;
- if (unicode_decode_call_errorhandler(
- errors, &errorHandler,
- "charmap", "character maps to <undefined>",
- &starts, &e, &startinpos, &endinpos, &exc, &s,
- &v, &outpos)) {
- Py_DECREF(x);
- goto onError;
- }
- Py_DECREF(x);
- continue;
- }
else if (PyUnicode_Check(x)) {
Py_ssize_t targetsize;
@@ -7554,8 +7542,10 @@
if (targetsize == 1) {
/* 1-1 mapping */
- if (unicode_putchar(&v, &outpos,
- PyUnicode_READ_CHAR(x, 0)) < 0)
+ Py_UCS4 value = PyUnicode_READ_CHAR(x, 0);
+ if (value == 0xFFFE)
+ goto Undefined;
+ if (unicode_putchar(&v, &outpos, value) < 0)
goto onError;
}
else if (targetsize > 1) {
@@ -7590,6 +7580,19 @@
}
Py_DECREF(x);
++s;
+ continue;
+Undefined:
+ /* undefined mapping */
+ Py_XDECREF(x);
+ startinpos = s-starts;
+ endinpos = startinpos+1;
+ if (unicode_decode_call_errorhandler(
+ errors, &errorHandler,
+ "charmap", "character maps to <undefined>",
+ &starts, &e, &startinpos, &endinpos, &exc, &s,
+ &v, &outpos)) {
+ goto onError;
+ }
}
}
if (unicode_resize(&v, outpos) < 0)
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
