Hi Manuel, It's very hard to know how to interpret your findings. Could you put everything up on a github repo and post the address, so we can see what kind of data you're working on and exactly how you have your model set up?
Regards, Fergal Byrne On Tue, Jun 3, 2014 at 4:51 PM, Scheele, Manuel < [email protected]> wrote: > Hi All, > > I have observed some NuPIC behaviour that I can't explain and was hoping > someone on this list could help me with it. > > This is the observation: > - When training a region, it is not always beneficial to use all the > training data available. > - Sometimes, using more training data will deteriorate the inference result > - The graph describing the relationship between inference accuracy and > training data used has a valley at which using more or less training points > will deteriorate inference. > - The two attached plots show how increasing the column count (from 512 to > 1024) of a region moves the valley to regions allowing for more training > points to be used. > > How I came to this conclusion: > The procedure used is straight forward: > 1) Training input: A file containing X sequences is used as training set > 2) Testing input: the same file that was used in 1) is used as testing > input. The anomaly score is recorded for every point in the input file. The > mean anomaly score over all steps is calculated. > 3) Y additional sequences are added to the file in step 1) such that it > now contains X:=X+Y sequences. We go to step 1) unless we have run out of > datapoints to add. > > The plots attached show the results of this procedure for 512 and 1024 > columns (the remaining parameters are unchanged, cells per column are 32). > The x axis shows how many training/testing points were fed into the region > and the y axis shows the mean anomaly score recorded over all those points. > > My problem/question: > I don't understand the shape of the graph. Why, precisely, does it first > go down and then up again (in other words why the valley)? > > Additional comment: > I tested whether Early Stopping (only training until I hit the valley) > improves inference on previously unseen sequences and found that it > doesn't. On the other hand, it greatly reduces the time I spend on training. > > Any ideas from you guys? > > Regards, > Manuel > > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > > -- Fergal Byrne, Brenter IT Author, Real Machine Intelligence with Clortex and NuPIC https://leanpub.com/realsmartmachines Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014: http://euroclojure.com/2014/ and at LambdaJam Chicago, July 2014: http://www.lambdajam.com http://inbits.com - Better Living through Thoughtful Technology http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne e:[email protected] t:+353 83 4214179 Join the quest for Machine Intelligence at http://numenta.org Formerly of Adnet [email protected] http://www.adnet.ie
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
