Dear Venkat, First of all, thanks for the feedback. We have some information about performance and QoS at the end of the Media Streaming page. However, it looks like it is not sufficient/clear. So, we’ll definitely enhance it.
Regarding the performance metric (some of these are already explained on the web page), we have a throughput and a QoS metric for Media Streaming and both are displayed on the server side (if you run the server with the command specified on the web page). You can compare the server throughputs by comparing the RTSP-Conns or kBits/Sec fields (they are proportional) when the server is running at a steady state and while the QoS requirements are met (if you keep the dataset and request (video) mix constant across runs). To make sure that we ensure QoS, we make sure that AvgDelay output by the server is less than 0. We came up with this value by gradually saturating the server and making sure that an additional client can still stream a video without any interruptions. This is not a perfect QoS metric, but it is an estimate to avoid oversaturating the server. Hope this helps. Please let us know if you have more questions and/or suggestions. -- Cansu On 04 Nov 2013, at 21:10, Venkatanathan Varadarajan <[email protected]<mailto:[email protected]>> wrote: Hi all, I was wondering if anyone could explain the results of the media-streaming benchmark. The documentation on the website does not talk about this. 1. The response time are in seconds. Is this the total time taken to stream a complete video or time between sending request and receiving a response from the server, at the client-side? 2. Similarly, what does it mean by "ops/sec". What are operations here? Streaming a video? The characteristic of performance of a media-streaming server at the client-side, I think, are average latency of each frame (audio or video) or frame transmission rate and % of user-perceivable stream disruptions/violation, or something similar. Is there an application-level metric in the result summary that is characteristic of the media-streaming benchmark? Or, what should be used as a metric for the media-streaming benchmark? Am I missing something? One general comment about the cloudsuite 2.0 benchmark suite, the documentation fails to explains the results/metrics for some of the benchmarks. A benchmark is incomplete without a defined/standardized metric that could be used to compare between different runs of the same benchmark. It would be great if there are sections for each benchmark in the website that also explains the metric that should be used for each. Thanks, Venkat -- Venkatanathan Varadarajan (Venkat), Graduate Student, University of Wisconsin-Madison.
