Hi All, I have observed some NuPIC behaviour that I can't explain and was hoping someone on this list could help me with it.
This is the observation: - When training a region, it is not always beneficial to use all the training data available. - Sometimes, using more training data will deteriorate the inference result - The graph describing the relationship between inference accuracy and training data used has a valley at which using more or less training points will deteriorate inference. - The two attached plots show how increasing the column count (from 512 to 1024) of a region moves the valley to regions allowing for more training points to be used. How I came to this conclusion: The procedure used is straight forward: 1) Training input: A file containing X sequences is used as training set 2) Testing input: the same file that was used in 1) is used as testing input. The anomaly score is recorded for every point in the input file. The mean anomaly score over all steps is calculated. 3) Y additional sequences are added to the file in step 1) such that it now contains X:=X+Y sequences. We go to step 1) unless we have run out of datapoints to add. The plots attached show the results of this procedure for 512 and 1024 columns (the remaining parameters are unchanged, cells per column are 32). The x axis shows how many training/testing points were fed into the region and the y axis shows the mean anomaly score recorded over all those points. My problem/question: I don't understand the shape of the graph. Why, precisely, does it first go down and then up again (in other words why the valley)? Additional comment: I tested whether Early Stopping (only training until I hit the valley) improves inference on previously unseen sequences and found that it doesn't. On the other hand, it greatly reduces the time I spend on training. Any ideas from you guys? Regards, Manuel
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
