Hi Vitaly, (Amen, see below),
On 03/15/2018 02:21 AM, Vitaly Bogdanov wrote:
matcher contains an atomspace and specific query to match. To not keep
large amount of atomspace data in repository the data can be generated
using specific pattern to test particular execution flow of the matching
algorithm. What do you think?
Yes, that's ideal. But if one wants to include large files (say a few
MBs) because they happen to be difficult to generate, I personally see
no problem with that, since it will live in its dedicated repo.
I see that atomspace repo already has perfomance benchmark:
https://github.com/opencog/atomspace/tree/master/opencog/benchmark
It mainly contains performance tests for particular methods but here is
more complex scenario:
https://github.com/opencog/atomspace/blob/master/opencog/benchmark/profile_bindlink.cc
Oh, true, I wasn't aware of that! (I'm in the list of contributors,
but after further inspection it appears that Emacs is the contributor
more than myself).
At a first glance new tests can be just added to exising framework. Are
Yes, but I still think it's much better to have all benchmarks to a
new repository.
You could probably move profile_bindlink.cc to the new framework.
Hmm, maybe the AtomSpace (non pattern matcher) benchmarks could be
moved to the new framework as well, that would make sense. We'd have a
repo "benchmark" for all OpenCog benchmarks, rather than multiple
repos, "pattern-matcher-benchmark", etc. I guess a one benchmark repo
is more elegant.
Or we could have a benchmark repo associated to each project repo, like
atomspace-benchmark
opencog-benchmark
...
What do you think?
there any known issues with this benchmarks?
I don't know. I haven't heard of Curtis, the main contributor of that
file, for a while.
Definitely test environment seriously affects performance results. I
don't see simple way to compare test results measured on different
environments. Possible options I see:
- keeping information about environment (CPU, OS, ...) with results; it
will at least allow knowing which system was used to get the results
Yes.
- setting up continuos integration environment with hardware for
performance tesing and running performance tests on each pull request to
ensure change doesn't affect performance
I suppose that would be ideal. I'm not the right person to manage
that, I believe Amen is.
- using performance profile instead of timings to analyze introduced
performance issues
Sounds like a good idea, though I don't understand the details of it,
could you elaborate?
Thanks,
Nil
Best regards,
Vitaly
2018-03-13 11:13 GMT+03:00 Nil Geisweiller <[email protected]
<mailto:[email protected]>>:
On 03/13/2018 07:22 AM, Linas Vepstas wrote:
>> So, the first thing you should do is build a good benchmark
tool, then we,
>> you and the rest of opencog community, can supply it with a
collection of
>> critical tests.
>
> How do you see such tool?
It could perhaps use cxxtest. On the other hand there probably no
need to touch C++ at all. Maybe a combination of Scheme and bash
scripts, or which ever way you're comfortable with. I suppose it
could automatically append the results into a diary, that could
perhaps be turned into a plot (which poses the problem of making
sure the system is the same).
Maybe it should have some functionality to repeat the benchmark a
number of times and gather some stats (mean and std dev), as there
tends to be a lot of variance.
Relatedly I wrote this simplistic benchmark tool that merely reruns
a number of times the unit tests and gather stats
https://github.com/opencog/atomspace/blob/master/scripts/benchmark_utests.sh
<https://github.com/opencog/atomspace/blob/master/scripts/benchmark_utests.sh>
the problem is that unit tests are not meant to be representative
benchmarks. Also benchmarks can be afforded to take more time to run
as they don't need to run as often as unit tests. Personally I see
no problem having a benchmark suite taking hours. They might also
require larger data sets. For these reasons it makes sense to have
the benchmark tool live in its own repository. I suppose this
repository could be generally dedicated to OpenCog's benchmarks, or
it could be for the pattern matcher alone. I don't know.
Nil
--
You received this message because you are subscribed to the Google Groups
"opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit
https://groups.google.com/d/msgid/opencog/488963bf-0b58-bef8-cdbb-50cee3c110fb%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
