Hi. I created #benchmark channel in OpenCog slack. I believe it will be more convenient to discuss technical details there. -- Alexey
2018-03-15 10:25 GMT+03:00 'Nil Geisweiller' via opencog < [email protected]>: > Hi Vitaly, (Amen, see below), > > On 03/15/2018 02:21 AM, Vitaly Bogdanov wrote: > >> matcher contains an atomspace and specific query to match. To not keep >> large amount of atomspace data in repository the data can be generated >> using specific pattern to test particular execution flow of the matching >> algorithm. What do you think? >> > > Yes, that's ideal. But if one wants to include large files (say a few > MBs) because they happen to be difficult to generate, I personally see > no problem with that, since it will live in its dedicated repo. > > >> I see that atomspace repo already has perfomance benchmark: >> https://github.com/opencog/atomspace/tree/master/opencog/benchmark >> It mainly contains performance tests for particular methods but here is >> more complex scenario: https://github.com/opencog/ato >> mspace/blob/master/opencog/benchmark/profile_bindlink.cc >> > > Oh, true, I wasn't aware of that! (I'm in the list of contributors, > but after further inspection it appears that Emacs is the contributor > more than myself). > > At a first glance new tests can be just added to exising framework. Are >> > > Yes, but I still think it's much better to have all benchmarks to a > new repository. > > You could probably move profile_bindlink.cc to the new framework. > > Hmm, maybe the AtomSpace (non pattern matcher) benchmarks could be > moved to the new framework as well, that would make sense. We'd have a > repo "benchmark" for all OpenCog benchmarks, rather than multiple > repos, "pattern-matcher-benchmark", etc. I guess a one benchmark repo > is more elegant. > > Or we could have a benchmark repo associated to each project repo, like > > atomspace-benchmark > opencog-benchmark > ... > > What do you think? > > there any known issues with this benchmarks? >> > > I don't know. I haven't heard of Curtis, the main contributor of that > file, for a while. > > >> Definitely test environment seriously affects performance results. I >> don't see simple way to compare test results measured on different >> environments. Possible options I see: >> - keeping information about environment (CPU, OS, ...) with results; it >> will at least allow knowing which system was used to get the results >> > > Yes. > > - setting up continuos integration environment with hardware for >> performance tesing and running performance tests on each pull request to >> ensure change doesn't affect performance >> > > I suppose that would be ideal. I'm not the right person to manage > that, I believe Amen is. > > - using performance profile instead of timings to analyze introduced >> performance issues >> > > Sounds like a good idea, though I don't understand the details of it, > could you elaborate? > > Thanks, > Nil > > >> Best regards, >> Vitaly >> >> 2018-03-13 11:13 GMT+03:00 Nil Geisweiller <[email protected] >> <mailto:[email protected]>>: >> >> On 03/13/2018 07:22 AM, Linas Vepstas wrote: >> >> >> So, the first thing you should do is build a good benchmark >> tool, then we, >> >> you and the rest of opencog community, can supply it with a >> collection of >> >> critical tests. >> > >> > How do you see such tool? >> >> >> It could perhaps use cxxtest. On the other hand there probably no >> need to touch C++ at all. Maybe a combination of Scheme and bash >> scripts, or which ever way you're comfortable with. I suppose it >> could automatically append the results into a diary, that could >> perhaps be turned into a plot (which poses the problem of making >> sure the system is the same). >> >> Maybe it should have some functionality to repeat the benchmark a >> number of times and gather some stats (mean and std dev), as there >> tends to be a lot of variance. >> >> Relatedly I wrote this simplistic benchmark tool that merely reruns >> a number of times the unit tests and gather stats >> >> https://github.com/opencog/atomspace/blob/master/scripts/ben >> chmark_utests.sh >> <https://github.com/opencog/atomspace/blob/master/scripts/be >> nchmark_utests.sh> >> >> the problem is that unit tests are not meant to be representative >> benchmarks. Also benchmarks can be afforded to take more time to run >> as they don't need to run as often as unit tests. Personally I see >> no problem having a benchmark suite taking hours. They might also >> require larger data sets. For these reasons it makes sense to have >> the benchmark tool live in its own repository. I suppose this >> repository could be generally dedicated to OpenCog's benchmarks, or >> it could be for the pattern matcher alone. I don't know. >> >> Nil >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "opencog" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/opencog. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/opencog/488963bf-0b58-bef8-cdbb-50cee3c110fb%40gmail.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "opencog" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/opencog. To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CABpRrhz-o7Mr3kOxF1F_7EAY_nE7B2OnFaToVt8ZzM3D0pW-MQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
