Stavros, Thank you for your response. Just one clarification: I was under the impression, as Cansu told in another email thread, that the websearch images currently, do not have magic breakpoints in place when a request is made or completed. Are you saying that the magic instructions to mark these events are there in, say the websearch image ( which is downloadable from EPFL) and I can measure QoS metrics by just focussing on those events ? In that case, that would be great. Or are you saying that I can measure QoS metrics by recreating the images, and adding magic instructions myself when the request is initiated/ended?
Thanks again. Hamza ----- Original Message ----- From: "Volos Stavros" <[email protected]> To: "Hamza Bin Sohail" <[email protected]> Cc: "<[email protected]>" <[email protected]> Sent: Tuesday, July 15, 2014 4:29:13 AM Subject: Re: Questions regarding cloudsuite request-oriented benchmarks - ping Hi Hamza, The statistics correspond to the emulated machine (as modelled by Simics). You cannot use those files to perform an QoS analysis for the Flexus-simulated CMP. In case you want to perform a QoS analysis, the benchmarks include magic breakpoints at various phases. Collecting the magic breakpoints of each individual request and the associated response, you can look at the QoS metrics. Regards, -Stavros. On Jul 5, 2014, at 6:46 AM, Hamza Bin Sohail <[email protected]> wrote: > > ping > > ----- Original Message ----- > From: "Hamza Bin Sohail" <[email protected]> > To: "Cansu Kaynak" <[email protected]> > Cc: [email protected] > Sent: Thursday, July 3, 2014 1:22:00 AM > Subject: Re: Questions regarding cloudsuite request-oriented benchmarks > > Cansu, > > Thanks alot for your reply. > > One more question before I stop bugging. > > For the websearch workload, http://parsa.epfl.ch/cloudsuite/search.html > says at the end: > > The statistics are kept in the outputDir/number_of_run/summary.xml. Among > other the following statistics are kept: <metric > unit=â€ops/secâ€>27.760</metric> > <users>55</users> > <responseTimes unit="seconds"> > <operation name="GET" r90th="0.500"> > <avg>0.101</avg> > <max>1.095</max> > <sd>0.116</sd> > <p90th>0.250</p90th> > <passed>true</passed> > <p99th>0.560</p99th> > </operation> > </responseTimes> > > Question: > 1) Is outputDir/number_of_run/summary.xml referring to statistics being > kept in the simulated machine or in simpflex ? Expanding on that, are these > values changeable in simflex or does the xml have to be loaded into the > simulated machine ? > > 2) Are these reported values (90th percentile, 99th percentile etc.) > meaningful only in a native setting where the benchmarks are entirely run on > native machines ? If not, do they accurately reflect the 90th and 99th > percentile values in a simflex timing simulation ? > > > Thanks > > Hamza > > ----- Original Message ----- > From: "Cansu Kaynak" <[email protected]> > To: "Hamza Bin Sohail" <[email protected]> > Cc: [email protected] > Sent: Sunday, June 29, 2014 1:28:21 PM > Subject: Re: Questions regarding cloudsuite request-oriented benchmarks - ping > > Dear Hamza, > > > 1) Technically, it is possible to insert magic instructions for each workload > to count the number of transactions completed during the simulation. > However, because Flexus uses statistical sampling to speed up cycle-accurate > simulations of representative execution windows in server workloads (which > can be on the order of tens of seconds), we use user IPC to measure > performance. > User IPC has been shown to accurately reflect overall system throughput. > For more information about statistical sampling and use of user IPC, you can > take a look at the IEEE Micro paper called SimFlex: Statistical Sampling of > Computer System Simulation ( > http://infoscience.epfl.ch/record/112669/files/simflex.pdf?version=1 ). > > > 2) Yes, there is a separate machine for each tier in multi-tier workload > setups. > > > 3) We collect the user IPC of each flexpoint (sampling unit) and then > calculate the average user IPC along with an average error for a given > confidence level as described in http://parsa.epfl.ch/cloudsuite/SMARTS.pdf . > > > Let us know if you have more questions. > > > > -- > Cansu > > > > On 29 Jun 2014, at 18:55, Hamza Bin Sohail < [email protected] > wrote: > > > ping > > ----- Original Message ----- > From: "Hamza Bin Sohail" < [email protected] > > To: [email protected] > Sent: Thursday, June 26, 2014 9:10:06 PM > Subject: Questions regarding cloudsuite request-oriented benchmarks > > > Hi all, > > I wanted to ask a few questions regarding request-oriented scaleout workloads > in cloudsuite e.g memcached, websearch etc. I'm interested in downloading the > simics images but before that, I had a few questions. > > 1) In the Wisconsin GEMS simulation infrastructure, there is a clear notion > of transactions for commercial workloads and performance > is measured in terms of throughput i.e cycles per transaction , and not in > terms of instructions executed. The reason obviously being that it is > perfectly possible for multi-threaded workloads to be executing instructions > while not doing any useful work e.g busy-waiting etc. Whenever a transaction > is initiated or ended, the drivers inform the simulator (simics) through > magic instructions. The magic instructions are event notification mechanisms. > So the question is: Do the cloudsuite simics images for memcached,websearch > and other request-oriented workloads have that mechanism in place to inform > simflex about the completion of events ? For example, when the Faban driver > gets a query response in websearch, or initiates one, does it tell Simflex > about it through magic instructions ? If not, what mechanism does simflex use > to report the number of requests completed in the final reported out ? > > 2) I am guessing these workloads setup as a distributed simics simulation, > correct ? For example, for websearch, is the frontend server on a separate > simulated machine, and the driver on a separate simulated machine, and the > index nodes on a separate simulated machine ? If yes, and if the answer to my > question in 1) is yes as well, then for websearch, are the magic instructions > placed in the faban driver to notify whether a request has ended or started ? > > 3) This question is only relevant if simflex does not know when a request was > generated or completed. If the answer to 1) is no, how is performance > measured ? Is the benchmark run for fixed number of instructions in each > sampling phase and then their execution times in each phase is aggregated / > extrapolated to what the most likely execution time would've been if the > entire benchmark were run to completion ? > > Thanks alot > > Hamza
