H, @Ali , I tried base64 but it did not work.
My original case, I collect the tweets that is in json format. And tweet text includes turkish characters. I will try the key.serializer.encoding properties and I will inform you Thanks, -----Original Message----- From: Radoslaw Gruchalski [mailto:[email protected]] Sent: Wednesday, November 9, 2016 2:33 PM To: Ali Akhtar; [email protected] Subject: Re: Kafka UTF 8 encoding problem Yes, understandandable, however, the OP mentions the data in UTF-8. If it’s not UTF, it needs to be converted to UTF. Or consider using value.serializer.encoding https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java#L29 – Best regards, Radek Gruchalski [email protected] On November 9, 2016 at 12:27:53 PM, Ali Akhtar ([email protected]) wrote: Its probably not UTF-8 if it contains Turkish characters. That's why base64 encoding / decoding it might help. On Wed, Nov 9, 2016 at 4:22 PM, Radoslaw Gruchalski <[email protected]> wrote: > Are you sure your string is in utf-8 in the first place? > What if you pass your string via something like: > > System.out.println( new String( > args[0].getBytes(StandardCharsets.UTF8), > StandardCharsets.UTF8) ) > > – > Best regards, > Radek Gruchalski > [email protected] > > > On November 9, 2016 at 12:14:03 PM, Baris Akgun (Garanti Teknoloji) ( > [email protected]) wrote: > > Hi, > > > > Producer Side// > > > > Properties props = *new* Properties(); > > props.put("metadata.broker.list", brokerList); > > props.put("serializer.class", “kafka.serializer.StringEncoder”); > > props.put("request.required.acks", "1"); > > > > Consumer side// > > > > I am using Spark Streaming Kafka API, I also try with Kafka CLI and > Java kafka api but I always face with same issue. > > > > Thanks > > > > *From:* Radoslaw Gruchalski [mailto:[email protected]] > *Sent:* Wednesday, November 9, 2016 1:49 PM > *To:* Baris Akgun (Garanti Teknoloji); [email protected] > *Subject:* Re: Kafka UTF 8 encoding problem > > > > Baris, > > > > Kafka does not care about encoding, everything is transported as bytes. > > What’s the configueration of your producer / consumer? > > Are you using Java / JVM? > > – > Best regards, > Radek Gruchalski > [email protected] > > > > On November 9, 2016 at 11:42:02 AM, Baris Akgun (Garanti Teknoloji) ( > [email protected]) wrote: > > Hi All, > > We are using Kafka 0,9.0.0 and we want to send our messages to topic > in > UTF-8 format but when we consume the messages from topic we saw that kafka > does not keep the original utf-8 format and we did not see the > messages exactly. > > > For example our message that includes turkish characters is "Barış" > but when we consume it we saw Bar?? . How can we solve that problem? > Is there any way to set kafka topic encoding? > > Thanks > > Barış > Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere > ozeldir ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza > ulasmis olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik > yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. > Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve guncelligi > konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu > bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige > ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin > korunamamasindan, virus icermesinden ve bilgisayar sisteminize > verebilecegi herhangi bir zarardan sorumlu tutulamaz. > > This message and attachments are confidential and intended solely for > the > individual(s) stated in this message. If you received this message although > you are not the addressee, you are responsible to keep the message > confidential. The sender has no responsibility for the accuracy or > correctness of the information in the message and its attachments. Our > company shall have no liability for any changes or late receiving, > loss of > integrity and confidentiality, viruses and any damages caused in > anyway to > your computer system. > > Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere > ozeldir ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza > ulasmis olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik > yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. > Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve guncelligi > konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu > bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige > ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin > korunamamasindan, virus icermesinden ve bilgisayar sisteminize > verebilecegi herhangi bir zarardan sorumlu tutulamaz. > > This message and attachments are confidential and intended solely for > the > individual(s) stated in this message. If you received this message although > you are not the addressee, you are responsible to keep the message > confidential. The sender has no responsibility for the accuracy or > correctness of the information in the message and its attachments. Our > company shall have no liability for any changes or late receiving, > loss of > integrity and confidentiality, viruses and any damages caused in > anyway to > your computer system. > Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz. This message and attachments are confidential and intended solely for the individual(s) stated in this message. If you received this message although you are not the addressee, you are responsible to keep the message confidential. The sender has no responsibility for the accuracy or correctness of the information in the message and its attachments. Our company shall have no liability for any changes or late receiving, loss of integrity and confidentiality, viruses and any damages caused in anyway to your computer system.
