You can accomplish this with the console consumer -- it has a formatter flag that lets you plug in custom logic for formatting messages. The default does not do any formatting, but if you write your own implementation, you just need to set the flag to plug it in.
You can see an example of this in Confluent's Avro support: https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/formatter/AvroMessageFormatter.java This deserializes using Avro and writes using Avro's JSON encoding to get nice, human-readable output. There are also wrapper scripts included to set the appropriate flags so it is as easy to use as the normal console consumer scripts. If you'd like something similar, our next release will have JSON serializers (see https://github.com/confluentinc/schema-registry/tree/master/json-serializer) but does not yet include the corresponding formatter. However, the formatter implementation should look very similar to the deserializer. -Ewen -Ewen On Fri, Oct 16, 2015 at 7:10 PM, David Luu <manga...@gmail.com> wrote: > I was wondering, do the kafka consumer shell scripts (high and low level > ones) and kafkacat do any pre-processing of the topic messages before > outputting to stdout or does it just output "as is" in the format the > message originally came in through kafka from the producer? > > Meaning pretty printed JSON produced is consumed as pretty printed JSON, > line delimited JSON blobs (not pretty printed) is consumed the same way. > > I'm asking this as I notice some topics I'm consuming with the shell > scripts (and kafkacat) are pretty printed and some not. So just wanted to > confirm this while also checking with the developers of the topic producers > on whether they are pretty printing on their end or not. > > In general, I'm assuming as best practice, it's better not to pretty print > the JSON as a producer to save on message size for network transmission and > file storage by kafka, since those extra newlines and spaces/tabs add up > over time? > -- Thanks, Ewen
