In the previous results, potentialPct was 0.8 and potentialRadius was large so the potential pool for each column was all inputs. For these results, I increased potentialPct to 1.0 and reduced potentialRadius to 1 so each column has a receptive field that is a 3 x 3 square of pixels from the input image and the receptive fields of the columns are evenly distributed across the image. This increased the accuracy substantially as seen in the attached graph. Subjectively, I believe it also slowed down the simulations quite a bit and the KNN classifier still outperforms the SP + KNN combination. Need to start measuring how long each train / test cycle takes so I have a quantitative metric for speed.
On Wed, Sep 3, 2014 at 1:43 PM, Jim Bridgewater <[email protected]> wrote: > I understand what your saying about the SP being just one part of a > bigger system, but I am still dubious about the way columns in the SP > establish their receptive fields. If potentialRadius is large and > potentialPct is small then each column connects to some random looking > sample of pixels in the image. This seems less biologically faithful > (an less useful) than having receptive fields composed of regions of > contiguous pixels. If potentialRadius is small then we get smaller, > contiguous receptive fields which are evenly spaced out over the input > space. This is better but does result in many columns being committed > to regions of the image where there is no activity. The cones in the > retina are dense in the center and sparse at the edges which currently > we have no way to reproduce in a single SP. I suppose this could be > overcome by using a network that feeds the output from lots and lots > of little SPs into a higher level SP. Maybe the input splitter map > that David has been asking about could handle mapping an input image > to an array of little SPs like this. > > Not clear what you mean by pooling or training with sequences. By > training with sequences do you mean providing some context like the > sequence of letters in a word? > > On Wed, Sep 3, 2014 at 9:41 AM, Subutai Ahmad <[email protected]> wrote: >> Jim, >> >> Thanks for running these experiments. I agree with you that the SP + KNN >> combination should do better than it is doing now. I would expect it to get >> close to the pure KNN. I think tuning the parameter combinations might help >> - Scott's new hyper search library would be a big help here! Even with >> tuning, I don't expect much better results than the KNN on this particular >> graph. It should do a bit better than KNN once you add in noise. >> >> To do better than the KNN on the test set, the system will need to encode >> more invariances than the SP can handle. For HTM's, this is exactly where >> pooling and training with sequences comes in. The SP is only one part of an >> overall system. You can also hard code translation invariances (similar to >> what HMAX and convolutional nets do). >> >> --Subutai >> >> >> On Mon, Sep 1, 2014 at 7:34 PM, Jim Bridgewater <[email protected]> wrote: >>> >>> The attached graph shows the results for 80/20, 60/40, 40/60, and >>> 20/80 splits of the data set into a training data set and a testing >>> data set. I ran 40/60 and 20/80 because the KNN classifier showed the >>> same performance for 80/20 and 60/40 so I wanted to see when its >>> accuracy would start to fall off which it does, but just a little. >>> The KNN classifier by itself outperforms the SP + KNN combination on >>> all splits. My inclination is to work on improving performance on >>> this task rather than adding noise to the images because it seems like >>> a good algorithm should do this task perfectly. >>> >>> On Wed, Aug 27, 2014 at 1:27 AM, Jim Bridgewater <[email protected]> >>> wrote: >>> > The code is available at github.com/baroobob/nupic.vision >>> > >>> > Today I made data sets for 80/20 and 60/40 splits between training and >>> > testing, I'll post the results here when I have them. >>> > >>> > On Thu, Aug 21, 2014 at 8:47 AM, Subutai Ahmad <[email protected]> >>> > wrote: >>> >> >>> >> Hi Jim, >>> >> >>> >> Thanks for doing this study. I echo Pedro's comments and questions - we >>> >> need >>> >> more investigations like this. >>> >> >>> >> In terms of future experiments, it would be great to see a result with >>> >> a >>> >> typical 80-20 split, i.e. 80% of the data for training and 20% for >>> >> testing. >>> >> Ideally it would be nice to see performance with random added noise or >>> >> other >>> >> distortions. Comparing against straight KNN for all of this is a good >>> >> standard thing to do (one of my pet peeves in machine learning is that >>> >> people don't do this enough. The KNN is provably almost optimal for >>> >> large >>> >> data sets - see the original Duda and Hart book!). >>> >> >>> >> My general expectation here is that the SP needs one or more thousand >>> >> images >>> >> to start doing a decent job. It probably wouldn't do hugely better or >>> >> worse >>> >> than KNN, though it should do better once you start adding noise, >>> >> dropouts, >>> >> etc. The SP won't learn too many invariances. In general I don't >>> >> expect >>> >> it do much better than standard spatial techniques on purely spatial >>> >> tasks. >>> >> The main job of the SP is to create a decent SDR for temporal memory >>> >> and >>> >> temporal pooling. >>> >> >>> >> Parameter tuning is admittedly difficult here. The specific numbers >>> >> matter >>> >> and you have to develop some intuitions. Unfortunately you can't >>> >> directly >>> >> apply our swarm - that is currently tied to the OPF and temporal >>> >> datasets. >>> >> It would be nice to improve that code to make it more general - it >>> >> could >>> >> help a lot in these kinds of investigations. >>> >> >>> >> Overall I am glad you started this project! Thank you for keeping us >>> >> informed throughout, and taking it all the way to a good writeup. Is >>> >> your >>> >> code available somewhere in case others want to try additional >>> >> experiments? >>> >> >>> >> --Subutai >>> >> >>> >> >>> >> >>> >> On Tue, Aug 19, 2014 at 6:52 PM, Jim Bridgewater <[email protected]> >>> >> wrote: >>> >>> >>> >>> Hi Pedro, >>> >>> >>> >>> Thank you for the feedback. >>> >>> >>> >>> 1. Previously I have run it using a randomly initialized SP with >>> >>> learning disabled and got results comparable to those with learning >>> >>> turned on which emphasizes that the spatial pooler as configured is >>> >>> not generalizing particularly well. I never tried sending the bit >>> >>> vectors directly to the classifier, but since you recommended it I >>> >>> made a pass through version of the SP which simply copies the input to >>> >>> the output and this produces better results than those using the real >>> >>> SP (as I have it configured)! >>> >>> >>> >>> 2. I haven't and in terms of the images it's actually 100% or 0%, but >>> >>> in terms of the characters the images represent (ground truth) it's >>> >>> always 100% which was my rational for using the small training data >>> >>> set since there are only 62 characters in both data sets (0-9, A-Z, >>> >>> a-z). I have run a small case where I train on 62 images (normal >>> >>> font) and test on 124 (normal and bold fonts) and I get around 80% >>> >>> accuracy which seems a bit low for what amounts to a pretty simple >>> >>> generalization task. >>> >>> >>> >>> 3. I am aware of MNIST, but I wanted to focus more on machine printed >>> >>> characters for document recognition. That coupled with the fact that >>> >>> when I was looking for data sets I did not find a place where MNIST >>> >>> was freely available was enough to keep me from using it. >>> >>> >>> >>> 4. I started with the parameters in Ian's sp_viewer demo, ran a few >>> >>> simple parameter searches to get a feel for how increment and >>> >>> decrement values affected the SP, and got some advice from Subutai on >>> >>> the mailing list. These parameters are probably not optimal. >>> >>> >>> >>> >>> >>> How well do you guess an optimized SP can do on tasks like these? >>> >>> >>> >>> On Mon, Aug 18, 2014 at 8:45 PM, Pedro Tabacof <[email protected]> >>> >>> wrote: >>> >>> > Hello Jim, >>> >>> > >>> >>> > Thank you for your work and report, we need more investigations like >>> >>> > yours. >>> >>> > A few suggestions: >>> >>> > >>> >>> > Since you're using a KNN classifier, it'd be nice to use it directly >>> >>> > on >>> >>> > the >>> >>> > pixels as a baseline. It's an important benchmark to show that NuPIC >>> >>> > indeed >>> >>> > is doing the heavy work. >>> >>> > Have you tried a more balanced division between training and testing >>> >>> > sets? >>> >>> > Using 100% or 1% of the data to train seems a bit to extreme to me. >>> >>> > Did you look at the MNIST dataset? It's probably the most widely >>> >>> > used >>> >>> > benchmark for computer vision. It's gonna be computationally >>> >>> > demanding >>> >>> > (50-60K images), but we will have results that can be compared to >>> >>> > other >>> >>> > machine learning approaches. >>> >>> > Did you use swarming or grid search to find out the best >>> >>> > meta-parameters? >>> >>> > >>> >>> > A long time ago I used the previous NuPIC implementation for static >>> >>> > classification (just the spatial pooler) and it was competitive with >>> >>> > SVMs. >>> >>> > >>> >>> > Pedro. >>> >>> > >>> >>> > >>> >>> > On Tue, Aug 19, 2014 at 12:24 AM, Jim Bridgewater >>> >>> > <[email protected]> >>> >>> > wrote: >>> >>> >> >>> >>> >> Hi everyone, >>> >>> >> >>> >>> >> I've written up a summary of the work I did this summer as part of >>> >>> >> Season of NuPIC that includes the most recent results. This >>> >>> >> summary >>> >>> >> is attached along with a separate file that contains 8,928 images >>> >>> >> from >>> >>> >> 144 fonts. These images were used to test the spatial pooler. The >>> >>> >> gist of it is that the SP does very well (>97% accuracy) when you >>> >>> >> train it on all of the images you test it on which is good, but >>> >>> >> very >>> >>> >> time consuming and doesn't require any ability to generalize. When >>> >>> >> I >>> >>> >> trained the SP on a much smaller data set of 186 images containing >>> >>> >> normal, bold, and italic characters not included in the larger data >>> >>> >> set the accuracy fell to about 32%. There are several ways to >>> >>> >> improve >>> >>> >> this. One is reducing the potential radius so columns learn >>> >>> >> features >>> >>> >> rather than entire characters. I tried this, but there appears to >>> >>> >> be >>> >>> >> a bug in the SP's potential mapping that currently prevents this >>> >>> >> technique from helping. Another way is to try different potential >>> >>> >> mappings, like lines with different orientations, again in an >>> >>> >> effort >>> >>> >> to get the SP's columns to learn features rather than entire >>> >>> >> characters. I've written a mapping for this but have not tried it. >>> >>> >> And yet another way to improve these results would be to add >>> >>> >> additional SP regions in an effort to get more generalization. >>> >>> >> >>> >>> >> I look forward to hearing your comments! >>> >>> >> >>> >>> >> -- >>> >>> >> James Bridgewater, PhD >>> >>> >> Arizona State University >>> >>> >> 480-227-9592 >>> >>> >> >>> >>> >> _______________________________________________ >>> >>> >> nupic mailing list >>> >>> >> [email protected] >>> >>> >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>> >>> >> >>> >>> > >>> >>> > >>> >>> > >>> >>> > -- >>> >>> > Pedro Tabacof >>> >>> > >>> >>> > _______________________________________________ >>> >>> > nupic mailing list >>> >>> > [email protected] >>> >>> > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>> >>> > >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> James Bridgewater, PhD >>> >>> Arizona State University >>> >>> 480-227-9592 >>> >>> >>> >>> _______________________________________________ >>> >>> nupic mailing list >>> >>> [email protected] >>> >>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>> >> >>> >> >>> >> >>> >> _______________________________________________ >>> >> nupic mailing list >>> >> [email protected] >>> >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>> >> >>> > >>> > >>> > >>> > -- >>> > James Bridgewater, PhD >>> > Arizona State University >>> > 480-227-9592 >>> >>> >>> >>> -- >>> James Bridgewater, PhD >>> Arizona State University >>> 480-227-9592 >> >> > > > > -- > James Bridgewater, PhD > Arizona State University > 480-227-9592 -- James Bridgewater, PhD Arizona State University 480-227-9592
accuracy.pdf
Description: Adobe PDF document
