I ran into the same issue and setting -Dfile.encoding=UTF-8 in the startup
script fixed it.
On Wed, Jun 6, 2012 at 12:18 AM, 刘明敏 <diveintotomor...@gmail.com> wrote:
> sorry for the late reply, it is my fault
>
> after set -Dfile.encoding=UTF-8 when start up producer
>
> problem solved.
>
> On Sat, Jun 2, 2012 at 6:07 PM, 刘明敏 <diveintotomor...@gmail.com> wrote:
>
> > we encountered an encoding issue when dealing with Chinese character
> >
> > the producer send characters in right encode(UTF-8),while after the
> > consumer get it ,it all turns into question marks:????
> >
> > when start up producer,kafka broker server and consumer, we tried
> > specified -Dfile.encoding=UTF-8,but it doesn't work
> >
> >
> > In producer,we use StringEncoder,below is the snippet of producer:
> >
> >
> >
> >
> > val props = new Properties();
> >
> >
> >
> > ...
> >
> > props.put("serializer.class", "kafka.serializer.StringEncoder");
> >
> >
> > props.put("compression.codec", "1") //gzip
> >
> >
> >
> > val producerConfig = new ProducerConfig(props);
> >
> >
> > val producer = new Producer[String, String](producerConfig);
> >
> >
> > val data = new ProducerData[String, String](topic, partitionKey,
> List("string_to_send_to_borker"));
> >
> >
> >
> > producer.send(data);
> >
> >
> >
> > and consumer:
> >
> >
> >
> >
> > val topicMessageStreams =
> consumerConnector.createMessageStreams(Predef.Map(topic -> consumers), new
> StringDecoder)
> >
> >
> >
> > for ((topic, streamList) <- topicMessageStreams) {
> >
> >
> > for (stream <- streamList) {
> >
> >
> > val processor = new StreamProcessor(stream)
> >
> >
> >
> > new Thread(processor).start();
> >
> >
> > }
> >
> >
> > }
> >
> >
> >
> > and the StreamProcessor just iterate each streams
> >
> >
> > val message = iterator.next.message//chinese characters in message
> turns into ?????
> >
> >
> >
> > Anyone any help?
> >
> >
> > --
> > Best Regards
> >
> > ----------------------
> > 刘明敏 | mmLiu
> >
> >
>
>
> --
> Best Regards
>
> ----------------------
> 刘明敏 | mmLiu
>
