Do you use Xalan as XSLT Processor ?
If so, I remember which could be a cause of your issue. I resolved it on my side years ago by compiling my own patched version of Xalan.

For "markers", you may use labels on your sitemap steps associated with a cocoon view.


Le 29/03/2022 à 18:36, Christopher Schultz a écrit :

On 3/29/22 12:06, Cédric Damioli wrote:
Could you provide more details ?
How is your XML processed before outputting the wrong UTF-8 sequence ?

It's somewhat straightforward:

<map:match pattern="/foo">
  <map:generate src="https://source/"; />

  <map:transform src="stuff-to-cincludes.xsl" />

  <map:transform src="other-stuff-to-cincludes.xsl" />

  <map:transform type="cinclude" />

  <map:transform src="my-big-transformer-to-xhtml.xsl" />

  <map:transform type="cinclude" /><!-- Yes, another one -->

  <map:transform type="i18n" />

  <map:transform src="strip-namespaces.xsl" /><!-- This is mine, not Cocoons ->

  <map:serialize type="xhtml" />

The xhtml serializer is the default, with encoding set to UTF-8. The HTTP response has "Content-Type: text/html" and the document itself contains:

<?xml version="1.0" encoding="UTF-8"?>


<meta content="text/html; charset=utf-8" http-equiv="content-type" />

So I think everything is configured correctly; it's just that those characters are getting mangled by something. I can try to cut-out some of those steps and see where it's happening.

I seem to remember being able to give each pipeline step a "marker" or something where you can say "stop after step 3" or whatever instead of having to chop-out configuration. Can you remind me or what that is again?


Le 29/03/2022 à 17:48, Christopher Schultz a écrit :

I'm still struggling with this. I have upgraded to 2.1.13 which includes the fix for but I'm still getting that American flag converted into those 4 HTML entities:


I would expect there to be a single (multibyte) character in the output with no HTML entities.

I've double-checked, and the source XML contains the flag as a single multi-byte character, served as UTF-8.

Any ideas for how to get this working? I'm sure I could put together a trivial test-case.


On 10/30/18 12:18, Christopher Schultz wrote:

Some additional information at the end.

On 10/30/18 11:58, Christopher Schultz wrote:

I'm attempting to do everything with UTF-8 in Cocoon 2.1.11. I have
a servlet generating XML in UTF-8 encoding and I have a pipeline
with a few transforms in it, ultimately serializing to XHTML.

If I have a Unicode character in the XML which is outside of the
BMP, such as this one: 🇺🇸  (that's an American flag, in case your
mail reader doesn't render it correctly), then I end up getting a
series of bytes coming from Cocoon after the transform that look
like UTF-16.

Here's what's in the XML:


Just like that. The bytes in the message for the flag character

f0  9f  87  ba  f0  9f  87  b8

When rendering that into XHTML, I'm getting this in the output:


The American flag in Unicode reference can be found here:

  You can see it broken down a bit better here for "Regional U":

and "Regional S":

What's happening is that some component in Cocoon has decided to
generate HTML entities instead of just emitting the character.
That's okay IMO. But what it does doesn't make sense for a UTF-8
output encodin g.

The first two entities "&#55356;&#56826;" are the decimal numbers
that represent the UTF-16 character for that "Regional Indicator
Symbol Letter U" and they are correct... for UTF-16. If I change
the output encoding from UTF-8 to UTF0-16, then the browser will
render these correctly. Using UTF-8, they show as four of those
ugly [?] characters on the screen.

I had originally just decided to throw up my hands and use UTF-16
encoding even though it's dumb. But it seems that MSIE cannot be
convinced to use UTF-16 no matter what, and I must continue to
support MSIE. :(

So it's back to UTF-8 for me.

How can I get Cocoon to output that character (or "those
characters") correctly?

It needs to be one of the following:

&#127482;&#127480; (HTML decimal entities)
&#x1f1fa;&#x1f1f8;             (HTML hex entities) f0 9f  87  ba
f0  9f  87  b8 (raw UTF-8 bytes)

Does anyone know how/where this conversion is being performed ion
Cocoon? Probably in a XHTML serializer (I'm using
org.apache.cocoon.serialization.XMLSerializer). I'm using
mime-type "text/html" and <encoding>UTF-8</encoding> in my sitemap
for that serializer (the one named "xhtml"). I believe I've mads
very few changes from the default, if any.

I haven't yet figured out how to get from what Java sees (\uE50C
for the "S" for example) to &#x1f1f8;, but knowing where the code
is that is making that decision would be very helpful.

Any ideas?


I created a text file (UTF-8) containing only the flag and read it in
using Java and printed all of the code points. There should be 2
"characters" in the file. It's 4 bytes per UTF-8 character so I
assumed I'd end up with 2 'char' primitives in the file, but I ended
up with more.

Here's the loop and the output:

         try( in = new"file.txt"))
             char[] chars = new char[10];

             int count =;

             for(int i=0; i<count; ++i)
                 System.out.println("Code point at " + i + " is " +
Integer.toHexString(Character.codePointAt(chars, i)));

         } catch (Exception e) {

== output ==

Code point at 0 is 1f1fa
Code point at 1 is ddfa
Code point at 2 is 1f1f8
Code point at 3 is ddf8
Code point at 4 is a

So Java thinks there are 4 things there, not 2. That could be a part
of the confusion. The code points shown for indexes 0 and 2 are the
"correct" ones. Those at indexes 1 and 3 should actually be *skipped*.

So, to render this string as an HTML numeric entity, we'd do something
like this:

String str = // this is the input

for(int i=0; i<str.length(); ++i) {
   int cp = Character.codePointAt(chars, i);


   // Skip any trailing "characters" that are actually a part of this one
   if(1 < Character.charCount(cp))
     i += Character.charCount(cp) - 1;

Using the above code is completely encoding-agnostic, because it's
describing the Unicode code point and not some set of bytes in a
particular flavor of UTF-x.


To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

Cédric Damioli
CMS - Java - Open Source

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to