Hi Jim, Thanks for doing this study. I echo Pedro's comments and questions - we need more investigations like this.
In terms of future experiments, it would be great to see a result with a typical 80-20 split, i.e. 80% of the data for training and 20% for testing. Ideally it would be nice to see performance with random added noise or other distortions. Comparing against straight KNN for all of this is a good standard thing to do (one of my pet peeves in machine learning is that people don't do this enough. The KNN is provably almost optimal for large data sets - see the original Duda and Hart book!). My general expectation here is that the SP needs one or more thousand images to start doing a decent job. It probably wouldn't do hugely better or worse than KNN, though it should do better once you start adding noise, dropouts, etc. The SP won't learn too many invariances. In general I don't expect it do much better than standard spatial techniques on purely spatial tasks. The main job of the SP is to create a decent SDR for temporal memory and temporal pooling. Parameter tuning is admittedly difficult here. The specific numbers matter and you have to develop some intuitions. Unfortunately you can't directly apply our swarm - that is currently tied to the OPF and temporal datasets. It would be nice to improve that code to make it more general - it could help a lot in these kinds of investigations. Overall I am glad you started this project! Thank you for keeping us informed throughout, and taking it all the way to a good writeup. Is your code available somewhere in case others want to try additional experiments? --Subutai On Tue, Aug 19, 2014 at 6:52 PM, Jim Bridgewater <[email protected]> wrote: > Hi Pedro, > > Thank you for the feedback. > > 1. Previously I have run it using a randomly initialized SP with > learning disabled and got results comparable to those with learning > turned on which emphasizes that the spatial pooler as configured is > not generalizing particularly well. I never tried sending the bit > vectors directly to the classifier, but since you recommended it I > made a pass through version of the SP which simply copies the input to > the output and this produces better results than those using the real > SP (as I have it configured)! > > 2. I haven't and in terms of the images it's actually 100% or 0%, but > in terms of the characters the images represent (ground truth) it's > always 100% which was my rational for using the small training data > set since there are only 62 characters in both data sets (0-9, A-Z, > a-z). I have run a small case where I train on 62 images (normal > font) and test on 124 (normal and bold fonts) and I get around 80% > accuracy which seems a bit low for what amounts to a pretty simple > generalization task. > > 3. I am aware of MNIST, but I wanted to focus more on machine printed > characters for document recognition. That coupled with the fact that > when I was looking for data sets I did not find a place where MNIST > was freely available was enough to keep me from using it. > > 4. I started with the parameters in Ian's sp_viewer demo, ran a few > simple parameter searches to get a feel for how increment and > decrement values affected the SP, and got some advice from Subutai on > the mailing list. These parameters are probably not optimal. > > > How well do you guess an optimized SP can do on tasks like these? > > On Mon, Aug 18, 2014 at 8:45 PM, Pedro Tabacof <[email protected]> wrote: > > Hello Jim, > > > > Thank you for your work and report, we need more investigations like > yours. > > A few suggestions: > > > > Since you're using a KNN classifier, it'd be nice to use it directly on > the > > pixels as a baseline. It's an important benchmark to show that NuPIC > indeed > > is doing the heavy work. > > Have you tried a more balanced division between training and testing > sets? > > Using 100% or 1% of the data to train seems a bit to extreme to me. > > Did you look at the MNIST dataset? It's probably the most widely used > > benchmark for computer vision. It's gonna be computationally demanding > > (50-60K images), but we will have results that can be compared to other > > machine learning approaches. > > Did you use swarming or grid search to find out the best meta-parameters? > > > > A long time ago I used the previous NuPIC implementation for static > > classification (just the spatial pooler) and it was competitive with > SVMs. > > > > Pedro. > > > > > > On Tue, Aug 19, 2014 at 12:24 AM, Jim Bridgewater <[email protected]> > > wrote: > >> > >> Hi everyone, > >> > >> I've written up a summary of the work I did this summer as part of > >> Season of NuPIC that includes the most recent results. This summary > >> is attached along with a separate file that contains 8,928 images from > >> 144 fonts. These images were used to test the spatial pooler. The > >> gist of it is that the SP does very well (>97% accuracy) when you > >> train it on all of the images you test it on which is good, but very > >> time consuming and doesn't require any ability to generalize. When I > >> trained the SP on a much smaller data set of 186 images containing > >> normal, bold, and italic characters not included in the larger data > >> set the accuracy fell to about 32%. There are several ways to improve > >> this. One is reducing the potential radius so columns learn features > >> rather than entire characters. I tried this, but there appears to be > >> a bug in the SP's potential mapping that currently prevents this > >> technique from helping. Another way is to try different potential > >> mappings, like lines with different orientations, again in an effort > >> to get the SP's columns to learn features rather than entire > >> characters. I've written a mapping for this but have not tried it. > >> And yet another way to improve these results would be to add > >> additional SP regions in an effort to get more generalization. > >> > >> I look forward to hearing your comments! > >> > >> -- > >> James Bridgewater, PhD > >> Arizona State University > >> 480-227-9592 > >> > >> _______________________________________________ > >> nupic mailing list > >> [email protected] > >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >> > > > > > > > > -- > > Pedro Tabacof > > > > _______________________________________________ > > nupic mailing list > > [email protected] > > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > > > > > > -- > James Bridgewater, PhD > Arizona State University > 480-227-9592 > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
