Re: Converting from KafkaUtils.createDirectStream to KafkaUtils.createStream

thanks I sorted this out. Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 25 April 2016 at 15:20,

Re: Converting from KafkaUtils.createDirectStream to KafkaUtils.createStream

Show the full relevant code including imports. On Fri, Apr 22, 2016 at 4:46 PM, Mich Talebzadeh wrote: > Hi Cody, > > This is my first attempt on using offset ranges (this may not mean much in > my context at the moment) > > val ssc = new StreamingContext(conf,

Re: Converting from KafkaUtils.createDirectStream to KafkaUtils.createStream

Hi Cody, This is my first attempt on using offset ranges (this may not mean much in my context at the moment) val ssc = new StreamingContext(conf, Seconds(10)) ssc.checkpoint("checkpoint") val kafkaParams = Map[String, String]("bootstrap.servers" -> "rhes564:9092", "schema.registry.url" ->

Re: Converting from KafkaUtils.createDirectStream to KafkaUtils.createStream

Spark streaming as it exists today is always microbatch. You can certainly filter messages using spark streaming. On Fri, Apr 22, 2016 at 4:02 PM, Mich Talebzadeh wrote: > yep actually using createDirectStream sounds a better way of doing it. Am I > correct that

Re: Converting from KafkaUtils.createDirectStream to KafkaUtils.createStream

yep actually using createDirectStream sounds a better way of doing it. Am I correct that createDirectStream was introduced to overcome micro-batching limitations? In a nutshell I want to pickup all the messages and keep signal according to pre-built criteria (say indicating a* buy signal*) and

Re: Converting from KafkaUtils.createDirectStream to KafkaUtils.createStream

You can still do sliding windows with createDirectStream, just do your map / extraction of fields before the window. On Fri, Apr 22, 2016 at 3:53 PM, Mich Talebzadeh wrote: > Hi Cody, > > I want to use sliding windows for Complex Event Processing micro-batching > > Dr

Re: Converting from KafkaUtils.createDirectStream to KafkaUtils.createStream

Hi Cody, I want to use sliding windows for Complex Event Processing micro-batching Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Converting from KafkaUtils.createDirectStream to KafkaUtils.createStream

Why are you wanting to convert? As far as doing the conversion, createStream doesn't take the same arguments, look at the docs. On Fri, Apr 22, 2016 at 3:44 PM, Mich Talebzadeh wrote: > Hi, > > What is the best way of converting this program of that uses >

Converting from KafkaUtils.createDirectStream to KafkaUtils.createStream

Hi, What is the best way of converting this program of that uses KafkaUtils.createDirectStream to Sliding window using val dstream = *KafkaUtils.createDirectStream*[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topic) to val dstream = *KafkaUtils.createStream*[String, String,