Thank you Matt,
I agree that Numenta is doing things different and they are doing it
well. But don't you think that you will be forced one day to tackle this
problem, e.g. when you'll want to show that HTM does the better job than
the rest? What is your personal opinion?
On 01/13/2016 07:28 AM, Matthew Taylor wrote:
This is a great question, and I'm not sure I can answer it. My litmus
test has always been whether the prediction is valuable or marketable.
Basically, does it solve your problem? Does it perhaps solve somebody
else's problem? The value is hard to measure, except in results. HTM is
still very new.
This is one of the major miscommunications between us HTM folks and the
rest of the ML community. They expect measured success in the form of
scientific proofs or benchmarks, competing to solve ever more
interesting problems. It is a noble effort and well worth it. I have the
utmost respect for our AI forefathers. But what we and Numenta are doing
is different. We started with the brain and a cellular, synaptic level.
At the core, this is an intelligence architecture pulled straight from
the neocortex. Our primary goal was to replicate this intelligence in
software, and it's going pretty well so far! :)
So anyway I don't know the answer to your question :P
Regards,
---------
Matt Taylor
OS Community Flag-Bearer
Numenta
On Tue, Jan 12, 2016 at 5:19 PM, Wakan Tanka <[email protected]
<mailto:[email protected]>> wrote:
Hello NuPIC,
How do you evaluate a correctness and accurancy of a prediction? Or
if you have multiple predictions for same data how do you compare
which prediction was more accurate? I've seen that there is NAB [1]
but to be honest I did not get deep into so I do not know if it
might help or not. AFAIK when you want to do such things the
correlation should work fine, in this case correlation between
original and predicted data. But correlation works only when you
have linear data, it would not work e.g. on hotgym example where you
have repeating cycles, peaks, maybe random events in particular days
etc. So my intuitive approach was to calculate absolute difference
[2] of original and predicted value and then calculate mean of those
values. The lower the mean is the better the prediction is. Then
I've realized that there is standard deviation [3] which can be
calculated from those absolute differences. Next step would be pick
up all values which have absolute differences of original and
predicted value:
1. above mean + standard deviation
2. bellow mean - standard deviation
This should give me an overview of how many values falls in this
interval and how many is doesn't. The dataset where more values
falls in the interval is dataset with better prediction.
Does this make sense?
[1]
http://numenta.com/blog/nab-a-benchmark-for-streaming-anomaly-detection.html
[2] https://en.wikipedia.org/wiki/Absolute_difference
[3] http://www.mathsisfun.com/data/standard-deviation.html
