Lemarais commented on pull request #317: URL: https://github.com/apache/incubator-nemo/pull/317#issuecomment-899381853
I referred the flink to record latency. the mark is created only in source vertex .It has a timestamp when it was created, a task id where it created, and a task id where it delivered from. executor record the timestamp when each mark is reached. by this record, we can calculate the latency for each path. but what we need to be aware of is that it takes no time to process data and the time it stays in buffer when the data processed by aggregation transform. Only the time it stagnated is added. There was a discussion with wonwook that it would be more accurate to use watermark. But It would be better to have both style of latency. flink style latency and latency calculated with watermark. However, flink did not support recording every watermark. and when I thought about whether it would be good to record all the watermarks or not, i thought that it is not a good logic to merge into master branch. So I did not include recording watermark part. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
