Re: Replaying logs with microsecond delay

[Dhruv Kumar]

Yes, thanks!

--
Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

> On May 15, 2018, at 21:31, Xingcan Cui  wrote:
> 
> Yes, that makes sense and maybe you could also generate dynamic intervals 
> according to the time spans.
> 
> Thanks,
> Xingcan
> 
>> On May 16, 2018, at 9:41 AM, Dhruv Kumar > > wrote:
>> 
>> As a part of my PhD research, I have been working on few optimization 
>> algorithms which try to jointly optimize delay and traffic (WAN traffic) in 
>> a geo-distributed streaming analytics setting. So, to show that the 
>> optimization actually works in real life, I am trying to implement these 
>> optimization algorithms on top of Apache Flink. For emulating a real life 
>> example, I need to generate a stream of records with some realistic delay 
>> (order of microseconds for fast incoming stream) between any two records. 
>> This stream will then by ingested and processed by Flink. 
>> 
>> Using the timestamps as is, in the form of event timestamps, only proves the 
>> algorithms from a theoretical/simulation perspective. 
>> 
>> Hope this answers your question to some extent at least. Let me know. 
>> 
>> Thanks!
>> --
>> Dhruv Kumar
>> PhD Candidate
>> Department of Computer Science and Engineering
>> University of Minnesota
>> www.dhruvkumar.me 
>> 
>>> On May 15, 2018, at 20:29, Xingcan Cui >> > wrote:
>>> 
>>> Hi Dhruv,
>>> 
>>> since there are timestamps associated with each record, I was wondering why 
>>> you try to replay them with a fixed interval. Can you give a little 
>>> explanation about that?
>>> 
>>> Thanks,
>>> Xingcan
>>> 
 On May 16, 2018, at 2:11 AM, Ted Yu > wrote:
 
 Please see the following:
 
 http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html 
 
 
 https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds
  
 
 
 On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar > wrote:
 Hi
 
 I am trying to replay a log file in which each record has a timestamp 
 associated with it. The time difference between the records is of the 
 order of microseconds. I am trying to replay this log maintaining the same 
 delay between the records (using Thread.sleep()) and sending it to a 
 socket. And then the Flink program reads the incoming data from this 
 socket. Currently, replay of the entire log file takes much more time (3 
 times) then the expected time (last_timstamp - first_timstamp).
 
 I wanted to know what are the standard ways of replaying log files if one 
 wants to maintain the same arrival delay between the records.
 
 Let me know if I am not clear above.
 
 Thanks 
 --
 Dhruv Kumar
 PhD Candidate
 Department of Computer Science and Engineering
 University of Minnesota
 www.dhruvkumar.me 
 
>>> 
>> 
> 

Re: Replaying logs with microsecond delay

[Xingcan Cui]

Yes, that makes sense and maybe you could also generate dynamic intervals 
according to the time spans.

Thanks,
Xingcan

> On May 16, 2018, at 9:41 AM, Dhruv Kumar  wrote:
> 
> As a part of my PhD research, I have been working on few optimization 
> algorithms which try to jointly optimize delay and traffic (WAN traffic) in a 
> geo-distributed streaming analytics setting. So, to show that the 
> optimization actually works in real life, I am trying to implement these 
> optimization algorithms on top of Apache Flink. For emulating a real life 
> example, I need to generate a stream of records with some realistic delay 
> (order of microseconds for fast incoming stream) between any two records. 
> This stream will then by ingested and processed by Flink. 
> 
> Using the timestamps as is, in the form of event timestamps, only proves the 
> algorithms from a theoretical/simulation perspective. 
> 
> Hope this answers your question to some extent at least. Let me know. 
> 
> Thanks!
> --
> Dhruv Kumar
> PhD Candidate
> Department of Computer Science and Engineering
> University of Minnesota
> www.dhruvkumar.me 
> 
>> On May 15, 2018, at 20:29, Xingcan Cui > > wrote:
>> 
>> Hi Dhruv,
>> 
>> since there are timestamps associated with each record, I was wondering why 
>> you try to replay them with a fixed interval. Can you give a little 
>> explanation about that?
>> 
>> Thanks,
>> Xingcan
>> 
>>> On May 16, 2018, at 2:11 AM, Ted Yu >> > wrote:
>>> 
>>> Please see the following:
>>> 
>>> http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html 
>>> 
>>> 
>>> https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds
>>>  
>>> 
>>> 
>>> On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar >> > wrote:
>>> Hi
>>> 
>>> I am trying to replay a log file in which each record has a timestamp 
>>> associated with it. The time difference between the records is of the order 
>>> of microseconds. I am trying to replay this log maintaining the same delay 
>>> between the records (using Thread.sleep()) and sending it to a socket. And 
>>> then the Flink program reads the incoming data from this socket. Currently, 
>>> replay of the entire log file takes much more time (3 times) then the 
>>> expected time (last_timstamp - first_timstamp).
>>> 
>>> I wanted to know what are the standard ways of replaying log files if one 
>>> wants to maintain the same arrival delay between the records.
>>> 
>>> Let me know if I am not clear above.
>>> 
>>> Thanks 
>>> --
>>> Dhruv Kumar
>>> PhD Candidate
>>> Department of Computer Science and Engineering
>>> University of Minnesota
>>> www.dhruvkumar.me 
>>> 
>> 
> 

Re: Replaying logs with microsecond delay

[Dhruv Kumar]

As a part of my PhD research, I have been working on few optimization 
algorithms which try to jointly optimize delay and traffic (WAN traffic) in a 
geo-distributed streaming analytics setting. So, to show that the optimization 
actually works in real life, I am trying to implement these optimization 
algorithms on top of Apache Flink. For emulating a real life example, I need to 
generate a stream of records with some realistic delay (order of microseconds 
for fast incoming stream) between any two records. This stream will then by 
ingested and processed by Flink. 

Using the timestamps as is, in the form of event timestamps, only proves the 
algorithms from a theoretical/simulation perspective. 

Hope this answers your question to some extent at least. Let me know. 

Thanks!
--
Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

> On May 15, 2018, at 20:29, Xingcan Cui  wrote:
> 
> Hi Dhruv,
> 
> since there are timestamps associated with each record, I was wondering why 
> you try to replay them with a fixed interval. Can you give a little 
> explanation about that?
> 
> Thanks,
> Xingcan
> 
>> On May 16, 2018, at 2:11 AM, Ted Yu > > wrote:
>> 
>> Please see the following:
>> 
>> http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html 
>> 
>> 
>> https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds
>>  
>> 
>> 
>> On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar > > wrote:
>> Hi
>> 
>> I am trying to replay a log file in which each record has a timestamp 
>> associated with it. The time difference between the records is of the order 
>> of microseconds. I am trying to replay this log maintaining the same delay 
>> between the records (using Thread.sleep()) and sending it to a socket. And 
>> then the Flink program reads the incoming data from this socket. Currently, 
>> replay of the entire log file takes much more time (3 times) then the 
>> expected time (last_timstamp - first_timstamp).
>> 
>> I wanted to know what are the standard ways of replaying log files if one 
>> wants to maintain the same arrival delay between the records.
>> 
>> Let me know if I am not clear above.
>> 
>> Thanks 
>> --
>> Dhruv Kumar
>> PhD Candidate
>> Department of Computer Science and Engineering
>> University of Minnesota
>> www.dhruvkumar.me 
>> 
> 

Re: Replaying logs with microsecond delay

[Xingcan Cui]

Hi Dhruv,

since there are timestamps associated with each record, I was wondering why you 
try to replay them with a fixed interval. Can you give a little explanation 
about that?

Thanks,
Xingcan

> On May 16, 2018, at 2:11 AM, Ted Yu  wrote:
> 
> Please see the following:
> 
> http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html 
> 
> 
> https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds
>  
> 
> 
> On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar  > wrote:
> Hi
> 
> I am trying to replay a log file in which each record has a timestamp 
> associated with it. The time difference between the records is of the order 
> of microseconds. I am trying to replay this log maintaining the same delay 
> between the records (using Thread.sleep()) and sending it to a socket. And 
> then the Flink program reads the incoming data from this socket. Currently, 
> replay of the entire log file takes much more time (3 times) then the 
> expected time (last_timstamp - first_timstamp).
> 
> I wanted to know what are the standard ways of replaying log files if one 
> wants to maintain the same arrival delay between the records.
> 
> Let me know if I am not clear above.
> 
> Thanks 
> --
> Dhruv Kumar
> PhD Candidate
> Department of Computer Science and Engineering
> University of Minnesota
> www.dhruvkumar.me 
> 

Re: Replaying logs with microsecond delay

[Dhruv Kumar]

Thanks a lot, Ted. Appreciate your help!

The approaches specified in the below links, are giving a very good level of 
accuracy. Solves my problem for now.

Thanks
--
Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

> On May 15, 2018, at 13:11, Ted Yu  wrote:
> 
> Please see the following:
> 
> http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html 
> 
> 
> https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds
>  
> 
> 
> On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar  > wrote:
> Hi
> 
> I am trying to replay a log file in which each record has a timestamp 
> associated with it. The time difference between the records is of the order 
> of microseconds. I am trying to replay this log maintaining the same delay 
> between the records (using Thread.sleep()) and sending it to a socket. And 
> then the Flink program reads the incoming data from this socket. Currently, 
> replay of the entire log file takes much more time (3 times) then the 
> expected time (last_timstamp - first_timstamp).
> 
> I wanted to know what are the standard ways of replaying log files if one 
> wants to maintain the same arrival delay between the records.
> 
> Let me know if I am not clear above.
> 
> Thanks 
> --
> Dhruv Kumar
> PhD Candidate
> Department of Computer Science and Engineering
> University of Minnesota
> www.dhruvkumar.me 
> 

Re: Replaying logs with microsecond delay

[Ted Yu]

Please see the following:

http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html

https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds

On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar  wrote:

> Hi
>
> I am trying to replay a log file in which each record has a timestamp
> associated with it. The time difference between the records is of the order
> of microseconds. I am trying to replay this log maintaining the same delay
> between the records (using Thread.sleep()) and sending it to a socket. And
> then the Flink program reads the incoming data from this socket. Currently,
> replay of the entire log file takes much more time (3 times) then the
> expected time (last_timstamp - first_timstamp).
>
> I wanted to know what are the standard ways of replaying log files if one
> wants to maintain the same arrival delay between the records.
>
> Let me know if I am not clear above.
>
> Thanks
> --
> *Dhruv Kumar*
> PhD Candidate
> Department of Computer Science and Engineering
> University of Minnesota
> www.dhruvkumar.me
>
>

Replaying logs with microsecond delay

[Dhruv Kumar]

Hi

I am trying to replay a log file in which each record has a timestamp 
associated with it. The time difference between the records is of the order of 
microseconds. I am trying to replay this log maintaining the same delay between 
the records (using Thread.sleep()) and sending it to a socket. And then the 
Flink program reads the incoming data from this socket. Currently, replay of 
the entire log file takes much more time (3 times) then the expected time 
(last_timstamp - first_timstamp).

I wanted to know what are the standard ways of replaying log files if one wants 
to maintain the same arrival delay between the records.

Let me know if I am not clear above.

Thanks 
--
Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me