--- Begin Message ---
Hi,that means we can't compare numenta results with other algorithms?
On Wednesday, 13 January 2016 11:58 AM, Matthew Taylor <[email protected]>
wrote:
This is a great question, and I'm not sure I can answer it. My litmus test has
always been whether the prediction is valuable or marketable. Basically, does
it solve your problem? Does it perhaps solve somebody else's problem? The value
is hard to measure, except in results. HTM is still very new.
This is one of the major miscommunications between us HTM folks and the rest of
the ML community. They expect measured success in the form of scientific proofs
or benchmarks, competing to solve ever more interesting problems. It is a noble
effort and well worth it. I have the utmost respect for our AI forefathers. But
what we and Numenta are doing is different. We started with the brain and a
cellular, synaptic level. At the core, this is an intelligence architecture
pulled straight from the neocortex. Our primary goal was to replicate this
intelligence in software, and it's going pretty well so far! :)
So anyway I don't know the answer to your question :P
Regards,
---------Matt TaylorOS Community Flag-BearerNumenta
On Tue, Jan 12, 2016 at 5:19 PM, Wakan Tanka <[email protected]> wrote:
Hello NuPIC,
How do you evaluate a correctness and accurancy of a prediction? Or if you have
multiple predictions for same data how do you compare which prediction was more
accurate? I've seen that there is NAB [1] but to be honest I did not get deep
into so I do not know if it might help or not. AFAIK when you want to do such
things the correlation should work fine, in this case correlation between
original and predicted data. But correlation works only when you have linear
data, it would not work e.g. on hotgym example where you have repeating cycles,
peaks, maybe random events in particular days etc. So my intuitive approach was
to calculate absolute difference [2] of original and predicted value and then
calculate mean of those values. The lower the mean is the better the prediction
is. Then I've realized that there is standard deviation [3] which can be
calculated from those absolute differences. Next step would be pick up all
values which have absolute differences of original and predicted value:
1. above mean + standard deviation
2. bellow mean - standard deviation
This should give me an overview of how many values falls in this interval and
how many is doesn't. The dataset where more values falls in the interval is
dataset with better prediction.
Does this make sense?
[1] http://numenta.com/blog/nab-a-benchmark-for-streaming-anomaly-detection.html
[2] https://en.wikipedia.org/wiki/Absolute_difference
[3] http://www.mathsisfun.com/data/standard-deviation.html
--- End Message ---
