How will i can to know that for how much time particular RDD had
remained in pipeline .
On Fri, Jun 19, 2015 at 7:59 AM, Tathagata Das t...@databricks.com wrote:
Why do you need to uniquely identify the message? All you need is the time
when the message was inserted by the receiver, and
Why do you need to uniquely identify the message? All you need is the time
when the message was inserted by the receiver, and when it is processed,
isnt it?
On Thu, Jun 18, 2015 at 2:28 PM, anshu shukla anshushuk...@gmail.com
wrote:
Thanks alot , But i have already tried the second way
Is there any fixed way to find among RDD in stream processing systems ,
in the Distributed set-up .
--
Thanks Regards,
Anshu Shukla
Its not clear what you are asking. Find what among RDD?
On Thu, Jun 18, 2015 at 11:24 AM, anshu shukla anshushuk...@gmail.com
wrote:
Is there any fixed way to find among RDD in stream processing systems ,
in the Distributed set-up .
--
Thanks Regards,
Anshu Shukla
Sorry , i missed the LATENCY word.. for a large streaming query .How to
find the time taken by the particular RDD to travel from initial
D-STREAM to final/last D-STREAM .
Help Please !!
On Fri, Jun 19, 2015 at 12:40 AM, Tathagata Das t...@databricks.com wrote:
Its not clear what you are
Thanks alot , But i have already tried the second way ,Problem with that
is that how to identify the particular RDD from source to sink (as we can
do by passing a msg id in storm) . For that i just updated RDD and added
a msgID (as static variable) . but while dumping them to file some of the
Couple of ways.
1. Easy but approx way: Find scheduling delay and processing time using
StreamingListener interface, and then calculate end-to-end delay = 0.5 *
batch interval + scheduling delay + processing time. The 0.5 * batch
inteval is the approx average batching delay across all the records
