Re: Spark Streaming - Kafka Direct Approach: re-compute from specific time
Ah right i see. Thank you very much. On May 25, 2016 11:11 AM, "Cody Koeninger"wrote: > There's an overloaded createDirectStream method that takes a map from > topicpartition to offset for the starting point of the stream. > > On Wed, May 25, 2016 at 9:59 AM, trung kien wrote: > > Thank Cody. > > > > I can build the mapping from time ->offset. However how can i pass this > > offset to Spark Streaming job using that offset? ( using Direct Approach) > > > > On May 25, 2016 9:42 AM, "Cody Koeninger" wrote: > >> > >> Kafka does not yet have meaningful time indexing, there's a kafka > >> improvement proposal for it but it has gotten pushed back to at least > >> 0.10.1 > >> > >> If you want to do this kind of thing, you will need to maintain your > >> own index from time to offset. > >> > >> On Wed, May 25, 2016 at 8:15 AM, trung kien wrote: > >> > Hi all, > >> > > >> > Is there any way to re-compute using Spark Streaming - Kafka Direct > >> > Approach > >> > from specific time? > >> > > >> > In some cases, I want to re-compute again from specific time (e.g > >> > beginning > >> > of day)? is that possible? > >> > > >> > > >> > > >> > -- > >> > Thanks > >> > Kien >
Re: Spark Streaming - Kafka Direct Approach: re-compute from specific time
There's an overloaded createDirectStream method that takes a map from topicpartition to offset for the starting point of the stream. On Wed, May 25, 2016 at 9:59 AM, trung kienwrote: > Thank Cody. > > I can build the mapping from time ->offset. However how can i pass this > offset to Spark Streaming job using that offset? ( using Direct Approach) > > On May 25, 2016 9:42 AM, "Cody Koeninger" wrote: >> >> Kafka does not yet have meaningful time indexing, there's a kafka >> improvement proposal for it but it has gotten pushed back to at least >> 0.10.1 >> >> If you want to do this kind of thing, you will need to maintain your >> own index from time to offset. >> >> On Wed, May 25, 2016 at 8:15 AM, trung kien wrote: >> > Hi all, >> > >> > Is there any way to re-compute using Spark Streaming - Kafka Direct >> > Approach >> > from specific time? >> > >> > In some cases, I want to re-compute again from specific time (e.g >> > beginning >> > of day)? is that possible? >> > >> > >> > >> > -- >> > Thanks >> > Kien - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark Streaming - Kafka Direct Approach: re-compute from specific time
Thank Cody. I can build the mapping from time ->offset. However how can i pass this offset to Spark Streaming job using that offset? ( using Direct Approach) On May 25, 2016 9:42 AM, "Cody Koeninger"wrote: > Kafka does not yet have meaningful time indexing, there's a kafka > improvement proposal for it but it has gotten pushed back to at least > 0.10.1 > > If you want to do this kind of thing, you will need to maintain your > own index from time to offset. > > On Wed, May 25, 2016 at 8:15 AM, trung kien wrote: > > Hi all, > > > > Is there any way to re-compute using Spark Streaming - Kafka Direct > Approach > > from specific time? > > > > In some cases, I want to re-compute again from specific time (e.g > beginning > > of day)? is that possible? > > > > > > > > -- > > Thanks > > Kien >
Re: Spark Streaming - Kafka Direct Approach: re-compute from specific time
Kafka does not yet have meaningful time indexing, there's a kafka improvement proposal for it but it has gotten pushed back to at least 0.10.1 If you want to do this kind of thing, you will need to maintain your own index from time to offset. On Wed, May 25, 2016 at 8:15 AM, trung kienwrote: > Hi all, > > Is there any way to re-compute using Spark Streaming - Kafka Direct Approach > from specific time? > > In some cases, I want to re-compute again from specific time (e.g beginning > of day)? is that possible? > > > > -- > Thanks > Kien - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org