Hi Baki,
It's enough for the producer to write the messages compressed. See here:
https://cwiki.apache.org/confluence/display/KAFKA/Compression
Thank you.
Daniel
> On 3 Nov 2016, at 21:27, Daniel Haviv <daniel.ha...@veracity-group.com> wrote:
>
> Hi,
> Kafka can compress/uncompress your messages for you seamlessly, adding
> compression on top of that will be redundant.
>
> Thank you.
> Daniel
>
>> On 3 Nov 2016, at 20:53, bhayat <baki...@gmail.com> wrote:
>>
>> Hello,
>>
>> I really wonder that whether i can stream compressed data with using
>> KafkaUtils.createDirectStream(...) or not.
>>
>> This is formal code that i use ;
>>
>> JavaPairInputDStream<String, String> messages =
>> KafkaUtils.createStream(javaStreamingContext, zookeeperConfiguration,
>> groupName, topicMap, StorageLevel.MEMORY_AND_DISK_SER());
>>
>> final JavaDStream<String> lines = messages.map(new
>> Function<Tuple2<String, String>, String>() {
>> @Override
>> public String call(Tuple2<String, String> tuple2) {
>> System.out.println("Stream Received: " + tuple2._2());
>> return tuple2._2();
>> }
>> });
>>
>> in this code i am consuming string message but in my case i need to consume
>> compressed data from stream then uncompress it and finally read string
>> message.
>>
>> Could you please asist me how i can go in right way ? ı am a bit confused
>> that whether spark streaming has this ability or not.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Stream-compressed-data-from-KafkaUtils-createDirectStream-tp28010.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
