BodyContentHandler and encoding

[Kaspar Fischer]

Dear list,

I am trying to convert documents to plaintext using Tika. For this, I do:

  StringWriter writer = new StringWriter();
  ContentHandler handler = new BodyContentHandler(writer);
  Parser parser = new AutoDetectParser();
  Metadata metadata = new Metadata();
  parser.parse(dataStream, handler, metadata, new ParseContext());
  String plaintext = writer.toString();

This works like a charm and I also get some metadata (like 
metadata.get("Content-Type")) for free.

The only issue I have run into is the encoding of documents. If I pass in a 
file in for example cp437 encoding, the plaintext seems to be in this encoding, 
too -- it is not in UTF-8 encoding. I am therefore wondering where and how the 
encoding detection must be done?

The "how" in this question is probably answered like this (please correct me if 
I am wrong!):

  CharsetDetector detector = new CharsetDetector();
  detector.setText(new BufferedInputStream(stream));
  String encoding = detector.detect().getName();

but I don't know how to tell the parser/handler about the encoding.

I have tried adding the encoding to the input metadata via

  metadata.add(Metadata.CONTENT_ENCODING, encoding);

but without success.

Many thanks for any feedback,
Kaspar