Personally, I like converting data before writing to Kafka, so I can easily support many consumers who don't know about EBCDIC.
A third option is to have a consumer that reads EBCDIC data from one Kafka topic and writes ASCII to another Kafka topic. This has the benefits of preserving the raw data in Kafka, in case you need it for troubleshooting, and also supporting non-EBCDIC consumers. The cost is a more complex architecture, but if you already have a stream processing system around (Storm, Samza, Spark), it can be an easy addition. On Mon, Aug 25, 2014 at 5:28 PM, <sonali.parthasara...@accenture.com> wrote: > Thanks Gwen! makes sense. So I'll have to weigh the pros and cons of doing an > EBCDIC to ASCII conversion before sending to Kafka Vs. using an ebcdic > library after in the consumer > > Thanks! > S > > -----Original Message----- > From: Gwen Shapira [mailto:gshap...@cloudera.com] > Sent: Monday, August 25, 2014 5:22 PM > To: users@kafka.apache.org > Subject: Re: EBCDIC support > > Hi Sonali, > > Kafka doesn't really care about EBCDIC or any other format - for Kafka bits > are just bits. So they are all supported. > > Kafka does not "read" data from a socket though. Well, it does, but the data > has to be sent by a Kafka producer. Most likely you'll need to implement a > producer that will get the data from the socket and send it as a message to > Kafka. The content of the message can be anything, including EBCDIC -. > > Then you'll need a consumer to read the data from Kafka and do something > with this - the consumer will need to know what to do with a message that > contains EBCDIC data. Perhaps you have EBCDIC libraries you can reuse there. > > Hope this helps. > > Gwen > > On Mon, Aug 25, 2014 at 5:14 PM, <sonali.parthasara...@accenture.com> wrote: >> Hey all, >> >> This might seem like a silly question, but does kafka have support for >> EBCDIC? Say I had to read data from an IBM mainframe via a TCP/IP socket >> where the data resides in EBCDIC format, can Kafka read that directly? >> >> Thanks, >> Sonali >> >> ________________________________ >> >> This message is for the designated recipient only and may contain >> privileged, proprietary, or otherwise confidential information. If you have >> received it in error, please notify the sender immediately and delete the >> original. Any other use of the e-mail by you is prohibited. Where allowed by >> local law, electronic communications with Accenture and its affiliates, >> including e-mail and instant messaging (including content), may be scanned >> by our systems for the purposes of information security and assessment of >> internal compliance with Accenture policy. >> ______________________________________________________________________ >> ________________ >> >> www.accenture.com
