Hey, > (...) > > I'll skip the pause functionality then.
Ok :-) > Consider using the median rather than the average. In some cases one > can observe infrequent outliers, which would distort the average. > > Would the median value really be a good choice to represent all > benchmark sub-tests? I mean, surely those outliers should have some > impact on the final result. Performance bottlenecks can sometimes be > that thin line that separates high-end from low-end products. I'm not so > sure we should just ignore them. I hope we're talking about the same thing here. Imagine you're benchmarking the vector addition, then you may get the following individual performances: 0.1 sec 0.1 sec 0.1 sec 0.5 sec <-- some unexpected other load on the system here The median is 0.1 here, whereas the average is 0.2. I'd consider the median to provide a more reliable indicator for system performance here. > Rathre than a single score, what about providing multiple scores for > the main quantities of interest? One score for GFLOPs, one score for > memory bandwidth, one score for latency? Is this too technical already? > > Well, a multi-score is fine by me. However, memory bandwidth in terms of > GB/s is only measured by copy benchmark as far I've seen. We should > consider adding some more bandwidth benchmarks in that case. The classic benchmark for this is the STREAM benchmark (http://www.cs.virginia.edu/stream/) which covers the following four vector operations: x <- y (copy) x <- a * y (scale) x <- y + z (sum) x <- y + alpha * z (triad) All four of them are easy to reproduce. If you are amazed about the simplicity of the benchmark, you'll be surprised about how much it can tell you regarding performance for a huge number of applications. > As for latency, I don't think our average users would care that much > about it. Whenever I overclocked my computer, I would primarily focus on > achieving higher memory bandwidth instead of lower latency. That is why > I would rather see my memory bandwidth instead of latency. It would > certainly be a useful addition, but not *that* important to have > dedicated score. Valid point, latency and friends should go to the expert results section. > Live updates would certainly be cool. However, you need to make sure > that the plots are only drawn while in between different benchmark > tests, otherwise the plotting might induce a certain load which will > interfere with the benchmark. I think this can be accomplished > without too much effort. > > Well each benchmark is run in its own separate thread. I don't think the > main GUI thread can interfere with a benchmark's thread. Sure it can. Not in terms of data, but in terms of eating CPU cycles while the benchmark thread is running. Keep in mind that ultimately we're also running tests with OpenMP on the CPU. > But if you're > referring to data transfer between the CPU and GPU, then I can't say for > sure. To the best of my knowledge, Qt widgets utilize CPU and RAM, > there's no GPU and modern OpenGL involved. As for communication between > running benchmarks and GUI, all messages coming from benchmarks are > sent between sub-tests. All benchmarks' sub-tests are intact, so there > shouldn't be any problems regarding message emitting. Messages between CPU and GPU are less an issue. Still, the GUI should do as little work as possible while the benchmark is running. It's okay to update in between sub-benchmarks. > At this point we should certainly define a reasonable grouping of > the results. For example, our current LU factorization is fairly > slow because of the way it is implemented, not because the hardware > is poor. Are you available for an IRC session on this tomorrow, e.g. > 16:00 UTC? > > Sure, I'm available at 16:00 UTC tomorrow (I guess that's today now :D ) Yes, Friday, 16:00 UTC. > I quickly adjusted the one I designed some time ago, see attachment. > I can commit the GIMP-File if you like it. Of course we can alter it > further as needed and appropriate. > > Thanks, I implemented it right away and pushed to GitHub. Looks good. > Yeah I'd like the GIMP file. I'll play around with it and see if > anything can be improved. Ok, I'll push it right after sending the email. Don't forget to pull. ;-) Btw: Could you please consider rearranging the folder hierarchy such that src/ only contains actual code? Put things like the splash screen into a separate folder. > From what I can tell from the screenshots: This is awesome! :-) > > If there's a 'download and run/benchmark now' button on the right > somewhere, that's already all need. > > Well that's just the matrix market web page rendered in a C++ widget :) > I could alter the web page with JavaScript. Dynamically add a download > and run button that will get the matrix file in matrix market format. If it's just some alterations with JavaScript, it would be a nice and simple solution. However, I guess that a bit more parsing will be required. > The only complication is that matrix files in mtx format can only be > downloaded as compressed .tar.gz files. That mean I would *another* 3rd > party library to unpack it. At least there are libraries that can be > shipped together with the project, so no additional environment > variables need to be set. Speaking of which, can you recommend a good, > cross-platform, open-source C++ library for unpacking .tar.gz files? You'll need e.g. zlib and libtar. libarchive is able to deal with both the decompression and the .tar packing: https://github.com/libarchive/libarchive/ > The annoying thing with parsing webpages is that they change too > often. We can get in touch with the Florida Matrix Market people, > maybe they offer some more stable way of getting the data. > > They are hosting their collection as a public data set using Amazon Web > Services. Unfortunately, I couldn't find a way to get to it using C++. > I'll take a better look at it tomorrow. Isn't this just a matter of downloading the data from some http:// location? Best regards, Karli ------------------------------------------------------------------------------ Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft _______________________________________________ ViennaCL-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/viennacl-devel
