Dear Jakub, sorry for a slow reply. I asked because error bars are very high on your plots. Although it is possible to describe a trend for each configuration, the standard deviation of your results is very high. Are you measuring the speedup on your local machine? If each experiment does not last for very long, e.g. less than one second, the TurboBoost on CPU might introduce a non-deterministic factor which does not allow you to measure execution time accurately.
I'm definitely not an expert in statistics and maybe someone here has a much better knowledge about this problem, but I think that you might use a different equation to calculate an uncertainty for a division of two values with a given standard deviation: https://lben.epfl.ch/files/content/sites/lben/files/users/179705/Error%20Propagation_2012.pdf (section 5.3) http://lectureonline.cl.msu.edu/~mmp/labs/error/e2.htm (point 2) I think that your approach gives you a more pessimistic, wider boundary on the actual uncertainty. Best, Marcin sob., 7 lip 2018 o 19:08 użytkownik Jakub Golinowski < jakub.golinow...@gmail.com> napisał: > Dear Marcin, > > as for the error bars in the speed-up plot: speedup is a composite > measure, i. e. it is computed by division of 2 measured values with > uncertainty (sequential and parallel execution times), therefore I compute > the relative uncertainty of the speedup as the sum of relative standard > deviations for each of the measured values. Finally, absolute uncertainty > of the speedup is computed as follows in python code: > > std_speedup = ((std_seq/t_seq) + (std_par/t_par)) * speedup > > Thank you very much for asking about it because this is the part I am not > 100% sure about. The above formula is correct when computing an error > (uncertainty) of composite value created from combining measurements with > measurement errors. So my understanding is that if we accept the standard > deviation as a measure of measurement error then the above formula is > correct. For the full picture here is the link to function responsible for > speedup computation: link > <https://github.com/Jakub-Golinowski/opencv_hpx_backend/blob/5a0f6459b48bce63d931ae0089c1695059ac9a1b/python/mandelbrot_benchmark/mandelbrot_benchmark.py#L140> > > However, now I think that other approach might be to simply compute 3 > speedups for 3 experiments separately and then use the standard deviation > of these speedups as the value for error bars, what do you think? > > If there is other convention for capturing uncertainty in the performance > benchmark plots, please let me know. > > Best regards, > > Jakub Golinowski > > 2018-07-07 17:12 GMT+01:00 Jakub Golinowski <jakub.golinow...@gmail.com>: > >> Dear Hartmut, >> >> I did not look thoroughly into the tbb and openmp implementations and >> therefore do not have a satisfying understanding why HPX is slower than tbb >> and openmp in my benchmark. >> I can look into that and get back to you. >> >> Also, if you recall any materials or existing benchmarks that may be >> useful for me in this matter please let me know. >> >> Best regards, >> >> Jakub Golinowski >> >> 2018-07-07 16:41 GMT+01:00 Hartmut Kaiser <hartmut.kai...@gmail.com>: >> >>> Jakub, >>> >>> Thanks for your work on this! >>> Do you have an understanding why HPX is slower than tbb and/or openmp >>> yet? >>> >>> Regards Hartmut >>> --------------- >>> http://stellar.cct.lsu.edu >>> https://github.com/STEllAR-GROUP/hpx >>> >>> >>> > -----Original Message----- >>> > From: gsoc-boun...@stellar-group.org <gsoc-boun...@stellar-group.org> >>> On >>> > Behalf Of Jakub Golinowski >>> > Sent: Saturday, July 7, 2018 9:57 AM >>> > To: g...@stellar-group.org; hpx-users@stellar.cct.lsu.edu >>> > Subject: [STE||AR Gsoc] GSoC 2nd evaluation summary >>> > >>> > Hello, >>> > >>> > in this e-mail I am enclosing the link to the google docs with the >>> short >>> > summary of the progress I have made over the period of 2 moths of >>> Google >>> > Summer of Code. >>> > >>> > https://docs.google.com/document/d/1gAMrg9Zt0- >>> > s8QqGgZkEoF2CkdpulbYGP40JcJVfPpoc/edit?usp=sharing >>> > >>> > Best regards, >>> > >>> > Jakub Golinowski >>> >>> >>> >> >
_______________________________________________ hpx-users mailing list hpx-users@stellar.cct.lsu.edu https://mail.cct.lsu.edu/mailman/listinfo/hpx-users
