Thanks, Jim. That looks like good progress! The area under the curve is much better now. I'll be curious as to what you get with the KNN.
--Subutai On Wed, Jul 9, 2014 at 11:33 AM, Jim Bridgewater <[email protected]> wrote: > Subutai, > > Here's an updated document that shows results for maxBoost = 3 and > maxBoost = 1. The two cases are only different for data sets that > contain between 7 and 17 characters. Their accuracies are the same > for all other data sets. This document also shows results where I > increased the maximum number of training cycles to 100 which improved > the accuracy quite a bit. I'm running another test right now that > runs 100 cycles on all data sets rather than determining the # of > cycles by looking for SDR collisions. > > My classifier is even simpler / cruder than you suppose, it looks for > an exact match between training and testing SDRs. This likely reduces > the accuracy so I'm working on switching to the KNN classifier. > > On Wed, Jul 9, 2014 at 11:13 AM, Jim Bridgewater <[email protected]> > wrote: > > Hi Subutai, > > > > maxBoost was actually at 1, but I failed to update the code listing in > > the document properly. I ran it with maxBoost = 3 first, by mistake > > and those results were not as good. > > > > > > > > On Wed, Jul 9, 2014 at 8:08 AM, Subutai Ahmad <[email protected]> > wrote: > >> > >> Thanks for running the experiment! Without looking at the script in > detail > >> I'm not sure what is going on. (BTW, looks like maxBoost was still at > 3, not > >> 1. Not sure if that will affect anything though.) > >> > >> My hunch is still that the SP should be able to perform perfectly on > this > >> task. > >> > >> --Subutai > >> > >> > >> On Wed, Jul 9, 2014 at 12:20 AM, Jim Bridgewater <[email protected]> > wrote: > >>> > >>> Subutai, > >>> > >>> Here's the first go at it, the accuracies are still less than 100% for > >>> most of the data sets, but many of them hit the 30 cycle limit. I'll > >>> run it again with a higher limit. > >>> > >>> On Tue, Jul 8, 2014 at 9:13 PM, Subutai Ahmad <[email protected]> > wrote: > >>> > > >>> > Thanks, Jim. Yes, as long as the inputs are pretty distinct, I think > >>> > learning the training set should work perfectly for much larger data > >>> > sets. > >>> > In theory the limit here should be extremely high (well over a > billion > >>> > patterns). > >>> > > >>> > I'm also a bit unsure about how you are classifying each letter > during > >>> > testing. I am assuming you are essentially finding the letter whose > SP > >>> > output has the most overlap with the actual current SP output? > >>> > > >>> > As a control experiment, could you try the following parameter > changes > >>> > with > >>> > you 1024:64 SP and see if it has any effect? > >>> > > >>> > potentialPct = 0.8, > >>> > synPermInactiveDec = 0.001 > >>> > synPermActiveInc = 0.001 > >>> > maxBoost = 1.0 > >>> > > >>> > The above would have a potential pool less than 100%, slow learning > and > >>> > no > >>> > boosting. > >>> > > >>> > Thanks, > >>> > > >>> > --Subutai > >>> > > >>> > > >>> > > >>> > On Tue, Jul 8, 2014 at 8:34 PM, Jim Bridgewater <[email protected]> > >>> > wrote: > >>> >> > >>> >> Hi Subutai, > >>> >> > >>> >> The training and testing data sets are the same. I tested 4 > different > >>> >> configurations and all configurations achieve perfect results for > the > >>> >> data sets that contain between 1 and 13 characters. Two > >>> >> configurations get perfect results for the data set with 14 > characters > >>> >> and one achieves it for 15 and 16 characters. Are you saying you > >>> >> think it should be perfect for larger data sets? > >>> >> > >>> >> > >>> >> > >>> >> On Tue, Jul 8, 2014 at 9:52 AM, Subutai Ahmad <[email protected]> > >>> >> wrote: > >>> >> > Hi Jim, > >>> >> > > >>> >> > Thanks for doing this experiment. I have one quick question: are > the > >>> >> > training and test sets the same? If so, I think the SP should > >>> >> > perform > >>> >> > perfectly in this task. > >>> >> > > >>> >> > Thanks, > >>> >> > > >>> >> > --Subutai > >>> >> > > >>> >> > > >>> >> > On Sun, Jul 6, 2014 at 10:43 PM, Jim Bridgewater < > [email protected]> > >>> >> > wrote: > >>> >> >> > >>> >> >> Hi Yuwei, > >>> >> >> > >>> >> >> Thanks for the feedback, I'm glad you found it interesting. > >>> >> >> > >>> >> >> I don't know why the training for the 3 character data set takes > so > >>> >> >> long, but it just occurred to me that it may have something to do > >>> >> >> with > >>> >> >> the particular characters involved. The one character data set > is > >>> >> >> an > >>> >> >> image of a zero, 0. The two character data set contains images > of 0 > >>> >> >> and 1 which are pretty easy to distinguish from one another. The > >>> >> >> three character data set contains images of 0,1, and 2. The 0 > and > >>> >> >> the 2 are much harder to distinguish from one another, in fact > when > >>> >> >> I > >>> >> >> ran Ian's spviewer demo on a hexadecimal data set (0-9, A-F), the > >>> >> >> column that represents 2 is the last one to stabilize. > >>> >> >> > >>> >> >> Just tried a three character data set consisting of O, X, and I > and > >>> >> >> it > >>> >> >> only took 2 cycles to learn it so it does appear to be related to > >>> >> >> the > >>> >> >> characters involved, but I'm still not sure why the number of > >>> >> >> training > >>> >> >> cycles decreased for the data sets containing between 4 and 13 > >>> >> >> characters because the 2 and 0 are still included. > >>> >> >> > >>> >> >> By no SDR collisions I mean that two different characters cannot > >>> >> >> have > >>> >> >> the exact same SDR, but their SDRs can share bits as long as the > two > >>> >> >> SDRs are different by at least one bit. > >>> >> >> > >>> >> >> I put "plot accuracy as a function of training cycles" on my ToDo > >>> >> >> list. I'll let you know when I check that one off. > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> On Sun, Jul 6, 2014 at 2:29 PM, Yuwei Cui <[email protected]> > >>> >> >> wrote: > >>> >> >> > Hi Jim, > >>> >> >> > > >>> >> >> > This is an interesting task. I am confused about how the > amount of > >>> >> >> > required > >>> >> >> > training varies with the data set size (Fig. 3). Do you have > any > >>> >> >> > intuition > >>> >> >> > on this (why it first increase and then decreases). I am also > >>> >> >> > having > >>> >> >> > trouble > >>> >> >> > to understand the criterion for stop training sounds. You said > >>> >> >> > there > >>> >> >> > must be > >>> >> >> > "no collisions" for images of different characters. But for > SDR, > >>> >> >> > a > >>> >> >> > small > >>> >> >> > number of collisions should not affect the performance much > (since > >>> >> >> > the > >>> >> >> > representation is distributed). It would be interesting if you > >>> >> >> > could > >>> >> >> > plot > >>> >> >> > the "error rate" as a function of iteration number during > >>> >> >> > learning. > >>> >> >> > > >>> >> >> > Yuwei > >>> >> >> > > >>> >> >> > > >>> >> >> > On Sun, Jul 6, 2014 at 12:20 AM, Jim Bridgewater > >>> >> >> > <[email protected]> > >>> >> >> > wrote: > >>> >> >> >> > >>> >> >> >> Hi everyone, > >>> >> >> >> > >>> >> >> >> I wanted to make sure I know how to size the spatial pooler > >>> >> >> >> properly > >>> >> >> >> for a given task so I ran a few tests to measure its image > >>> >> >> >> recognition > >>> >> >> >> accuracy on data sets of different sizes. One of the > surprising > >>> >> >> >> results was how the amount of training required varies with > the > >>> >> >> >> size > >>> >> >> >> of the data set. The results are shown in the attached pdf. > >>> >> >> >> > >>> >> >> >> -- > >>> >> >> >> Jim Bridgewater, PhD > >>> >> >> >> Arizona State University > >>> >> >> >> 480-227-9592 > >>> >> >> >> > >>> >> >> >> _______________________________________________ > >>> >> >> >> nupic mailing list > >>> >> >> >> [email protected] > >>> >> >> >> > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >>> >> >> >> > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > -- > >>> >> >> > -- > >>> >> >> > Yuwei Cui > >>> >> >> > > >>> >> >> > Algorithm Internship, Numenta Inc. > >>> >> >> > > >>> >> >> > PhD Candidate, Neuroscience and Cognitive Science > >>> >> >> > > >>> >> >> > University of Maryland, College Park, MD, 20742 > >>> >> >> > > >>> >> >> > Homepage: http://terpconnect.umd.edu/~ywcui/ > >>> >> >> > > >>> >> >> > > >>> >> >> > _______________________________________________ > >>> >> >> > nupic mailing list > >>> >> >> > [email protected] > >>> >> >> > > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >>> >> >> > > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> -- > >>> >> >> Jim Bridgewater, PhD > >>> >> >> Arizona State University > >>> >> >> 480-227-9592 > >>> >> >> > >>> >> >> _______________________________________________ > >>> >> >> nupic mailing list > >>> >> >> [email protected] > >>> >> >> > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >>> >> > > >>> >> > > >>> >> > > >>> >> > _______________________________________________ > >>> >> > nupic mailing list > >>> >> > [email protected] > >>> >> > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >>> >> > > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> Jim Bridgewater, PhD > >>> >> Arizona State University > >>> >> 480-227-9592 > >>> >> > >>> >> _______________________________________________ > >>> >> nupic mailing list > >>> >> [email protected] > >>> >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >>> > > >>> > > >>> > > >>> > _______________________________________________ > >>> > nupic mailing list > >>> > [email protected] > >>> > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >>> > > >>> > >>> > >>> > >>> -- > >>> Jim Bridgewater, PhD > >>> Arizona State University > >>> 480-227-9592 > >>> > >>> _______________________________________________ > >>> nupic mailing list > >>> [email protected] > >>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >>> > >> > >> > >> _______________________________________________ > >> nupic mailing list > >> [email protected] > >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >> > > > > > > > > -- > > Jim Bridgewater, PhD > > Arizona State University > > 480-227-9592 > > > > -- > Jim Bridgewater, PhD > Arizona State University > 480-227-9592 > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
