Re: Kafka Direct Stream - Custom Serialization and Deserilization

[Benjamin Fradet]

There is one for the key of your Kafka message and one for its value.
On 26 Jun 2015 4:21 pm, "Ashish Soni"  wrote:

> my question is why there are similar two parameter String.Class and
> StringDecoder.class what is the difference each of them ?
>
> Ashish
>
> On Fri, Jun 26, 2015 at 8:53 AM, Akhil Das 
> wrote:
>
>> ​JavaPairInputDStream messages =
>> KafkaUtils.createDirectStream(
>> jssc,
>> String.class,
>> String.class,
>> StringDecoder.class,
>> StringDecoder.class,
>> kafkaParams,
>> topicsSet
>> );
>>
>> Here:
>>
>> jssc => JavaStreamingContext
>> String.class => Key , Value classes
>> StringDecoder => Key, Value decoder classes
>> KafkaParams => Map in which you specify all the kafka details (like
>> brokers, offset etc)
>> topicSet => Set of topics from which you want to consume data.​
>>
>> ​Here's a sample program
>> 
>> for you to start.​
>>
>>
>>
>> Thanks
>> Best Regards
>>
>> On Fri, Jun 26, 2015 at 6:09 PM, Ashish Soni 
>> wrote:
>>
>>> Hi ,
>>>
>>> If i have a below data format , how can i use kafka direct stream to
>>> de-serialize as i am not able to understand all the parameter i need to
>>> pass , Can some one explain what will be the arguments as i am not clear
>>> about this
>>>
>>> JavaPairInputDStream org.apache.spark.streaming.kafka.KafkaUtils
>>> .createDirectStream(JavaStreamingContext arg0, Class arg1, Class
>>> arg2, Class arg3, Class arg4, Map arg5, Set<
>>> String> arg6)
>>>
>>> ID
>>> Name
>>> Unit
>>> Rate
>>> Duration
>>>
>>>
>>
>

Re: Kafka Direct Stream - Custom Serialization and Deserilization

[Ashish Soni]

my question is why there are similar two parameter String.Class and
StringDecoder.class what is the difference each of them ?

Ashish

On Fri, Jun 26, 2015 at 8:53 AM, Akhil Das 
wrote:

> ​JavaPairInputDStream messages =
> KafkaUtils.createDirectStream(
> jssc,
> String.class,
> String.class,
> StringDecoder.class,
> StringDecoder.class,
> kafkaParams,
> topicsSet
> );
>
> Here:
>
> jssc => JavaStreamingContext
> String.class => Key , Value classes
> StringDecoder => Key, Value decoder classes
> KafkaParams => Map in which you specify all the kafka details (like
> brokers, offset etc)
> topicSet => Set of topics from which you want to consume data.​
>
> ​Here's a sample program
> 
> for you to start.​
>
>
>
> Thanks
> Best Regards
>
> On Fri, Jun 26, 2015 at 6:09 PM, Ashish Soni 
> wrote:
>
>> Hi ,
>>
>> If i have a below data format , how can i use kafka direct stream to
>> de-serialize as i am not able to understand all the parameter i need to
>> pass , Can some one explain what will be the arguments as i am not clear
>> about this
>>
>> JavaPairInputDStream org.apache.spark.streaming.kafka.KafkaUtils
>> .createDirectStream(JavaStreamingContext arg0, Class arg1, Class
>> arg2, Class arg3, Class arg4, Map arg5, Set<
>> String> arg6)
>>
>> ID
>> Name
>> Unit
>> Rate
>> Duration
>>
>>
>

Re: Kafka Direct Stream - Custom Serialization and Deserilization

[Akhil Das]

​JavaPairInputDStream messages =
KafkaUtils.createDirectStream(
jssc,
String.class,
String.class,
StringDecoder.class,
StringDecoder.class,
kafkaParams,
topicsSet
);

Here:

jssc => JavaStreamingContext
String.class => Key , Value classes
StringDecoder => Key, Value decoder classes
KafkaParams => Map in which you specify all the kafka details (like
brokers, offset etc)
topicSet => Set of topics from which you want to consume data.​

​Here's a sample program

for you to start.​



Thanks
Best Regards

On Fri, Jun 26, 2015 at 6:09 PM, Ashish Soni  wrote:

> Hi ,
>
> If i have a below data format , how can i use kafka direct stream to
> de-serialize as i am not able to understand all the parameter i need to
> pass , Can some one explain what will be the arguments as i am not clear
> about this
>
> JavaPairInputDStream org.apache.spark.streaming.kafka.KafkaUtils
> .createDirectStream(JavaStreamingContext arg0, Class arg1, Class
> arg2, Class arg3, Class arg4, Map arg5, Set
> arg6)
>
> ID
> Name
> Unit
> Rate
> Duration
>
>

Kafka Direct Stream - Custom Serialization and Deserilization

[Ashish Soni]

Hi ,

If i have a below data format , how can i use kafka direct stream to
de-serialize as i am not able to understand all the parameter i need to
pass , Can some one explain what will be the arguments as i am not clear
about this

JavaPairInputDStream

,
V
>
org

.apache

.spark

.streaming

.kafka

.KafkaUtils

.createDirectStream(JavaStreamingContext

arg0, Class

>
arg1, Class

>
arg2, Class

>
arg3, Class

>
arg4, Map

,
String
>
arg5, Set

>
arg6)

ID
Name
Unit
Rate
Duration