I'm thinking that 0xFEFF shouldn't be in a UTF16BE string, except at the start right?
For other kinds of UTF, I'm not sure if it is allowed or not. I know it is allowed in UTF16LE, although discouraged.
Instead of "can't use ZWNBS", I think that char is discouraged. Where
is the rule that discourages it?
See http://anubis.dkuug.dk/JTC1/SC2/WG2/docs/n2235.htm for the proposal to replace the ZWNBS use of U+FEFF with a new character U+2060 WORD JOINER.
See http://www.unicode.org/charts/PDF/UFE70.pdf for current definition of U+FEFF stating:
• use as an indication of non-breaking is deprecated; see 2060 instead.
See http://www.unicode.org/charts/PDF/U2000.pdf for the definition of U+2060 WORD JOINER which states:
• a zero width non-breaking space (only) • intended for disambiguation of functions for byte order mark → FEFF zero width no-break space
U+20620 WORD JOINER should be used instead of U+FEFF if one's font and application supports it.
Jim Allan
