Hi Ian,
Thanks for the detailed response. Unfortunately, I didn't see your message until this morning when I received the digest from the mailing list. Turned digests off so I'll get messages quicker from now on. I was thinking about this last night and realized that the combinatorial view point is not the right way to look at this since most of those output values will share active columns so they should not be used to represent unrelated input patterns. If each column can learn a "feature" from the images it's seen then in principle a 16 column SP can represent the 2^16 = 65,536 different images which combine these 16 features, but only if we allow it to have the number of active columns that corresponds to the number of features in the image rather than forcing it to have a specific number of active columns. Is there a way to do this? On Fri, May 30, 2014 at 1:51 AM, <nupic-requ...@lists.numenta.org> wrote: > Send nupic mailing list submissions to > nupic@lists.numenta.org > > To subscribe or unsubscribe, visit > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > or, via email, send a message with subject or body 'help' to > nupic-requ...@lists.numenta.org > > You can reach the person managing the list at > nupic-ow...@lists.numenta.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of nupic digest..." > > > Today's Topics: > > 1. Re: Some observations from Ian's Fall 2013 SP demo (Ian Danforth) > 2. Re: Some observations from Ian's Fall 2013 SP demo > (Jim Bridgewater) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 29 May 2014 20:27:51 -0700 > From: Ian Danforth <idanfo...@embodiedai.com> > To: "NuPIC general mailing list." <nupic@lists.numenta.org> > Subject: Re: [nupic-discuss] Some observations from Ian's Fall 2013 SP > demo > Message-ID: > <CAOajdBpF8XTt0=wkyj47q2b2b0a3hfzmv+3cjelacjc78mf...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Jim, > > Thanks for writing this up and sharing it with the list! This is both > expected and good behavior! Let me see if I can shift your intuitions a bit > so you'll agree with that :) > > To start out, what is the SP good for? It does two interesting things, it > each column memorizes things it's seen and together they perform > dimensionality reduction. > > Each column is a bit like a film negative, the more times it's exposed to a > pattern the more it will resemble that pattern. If you have a double > exposure, to multiple patterns, it might come to represent both, or a bit > of each. If you show one column the same pattern again and again it will > form a very strong representation of it. In my demo there are so few > patterns each column can come to perfectly represent the input it's shown. > If you force columns to become active in the presence of more than one of > those patterns it will become a representation of a blend of the two. If > you make a single column see all the patterns it will blend them all! Just > like a negative exposed again and again to several scenes. Each column is a > powerful feature learning system in it's own right, and when they are > allowed to compete, they naturally sort themselves into groups of good > representations. By forcing them to come on when they would normally be > out-competed, they blend into the all-purpose representations you saw. > > But why doesn't the SP naturally take advantage of the combinatorial nature > of having multiple columns on? Well that property really only comes into > play when you need dimensionality reduction. You need to start with a set > of inputs >> than the number of columns. Then when a column is exposed over > and over, it never sees exactly the same input, and only the common areas > to all the inputs (those bits that are most similar) will be retained in > the snapshot it retains. This is the pooling of similar, but not exactly > similar, features into the activity of each column. Once you have these set > of features, you *then* use the combinatorial property of having multiple > columns on so that you can assemble a complete representation from a set of > features. If four columns represent vertical and horizontal lines, you > might find the combination of all four representing a box. It is the > ability to compose complex representations out of a much smaller set of > simpler representations where the combinatorial numbers you're talking > about come into play. Instead of taking 10,000 columns to learn to > differentiate 10,000 things, you can get different representations from a > much much smaller set of columns. The unique combination of features in > each input providing the eventual unique SDR. > > I hope this helps in understanding the results! A good next question might > be, what is the minimum number of inputs you need before having 1 column / > input becomes unreasonable and you really *need* the property of > dimensionality reduction? > > Ian > > > On Thu, May 29, 2014 at 4:32 PM, Jim Bridgewater <jwbwa...@gmail.com> wrote: > >> Hi everyone, >> >> I'm working on an optical character recognition project for the season >> of nupic this summer. I've been playing around with Ian Danforth's >> Fall 2013 hackathon demo of the spatial pooler and put together the >> attached pdf document which shows some observations of how the spatial >> pooler behaves with different numbers of active columns. Maybe some >> people on this list have thought about this before and have some >> comments about why the SP displays this behavior. >> >> >> >> ---------- Forwarded message ---------- >> From: Scott Purdy <sc...@numenta.org> >> Date: Thu, May 29, 2014 at 4:02 PM >> Subject: Re: Update >> To: Jim Bridgewater <jwbwa...@gmail.com> >> >> >> The write up is quite nice, thanks for putting that together. You >> might send it to the discuss list in case anyone else is curious. I >> don't think the results are particularly suprising. In particular, the >> SP is known to not always perform well when the number of input bits >> or columns is small or there are a small number of input patterns. I >> will try to discuss with Subutai tomorrow to confirm that there is >> nothing wrong with your experimental method and see if I can get more >> details on the onion charts. >> >> I think the onion charts will make more sense on a more sophisticated >> data set as it would likely suffer from the same issues with small >> numbers of inputs. >> >> >> On Thu, May 29, 2014 at 1:50 PM, Jim Bridgewater <jwbwa...@gmail.com> >> wrote: >> > >> > Hi Scott, >> > >> > I've been playing with Ian's demo a bit to get a feel for how the >> > spatial pooler works. I suppose it was intended to be used with only >> > a small % of active columns, but I do find it interesting how poorly >> > it performs when 50% of the columns are active since it has the >> > greatest number of output values in this case. See attached pdf. >> > >> > I want to work on a way of showing the results that makes it easier to >> > see how the representation of each input image changes as the sp is >> > trained and whether the outputs are different for each unique input. >> > I was thinking about converting the sp output to a hexadecimal value >> > and plotting that vs time for each input image. >> > >> > Maybe you guys could send me some examples of the onion graphs Subutai >> > mentioned so I can see how they work. >> > >> > -- >> > Jim Bridgewater, PhD >> > Arizona State University >> > 480-227-9592 >> >> >> >> >> -- >> Jim Bridgewater, PhD >> Arizona State University >> 480-227-9592 >> >> _______________________________________________ >> nupic mailing list >> nupic@lists.numenta.org >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.numenta.org/pipermail/nupic_lists.numenta.org/attachments/20140529/030f10df/attachment-0001.html> > > ------------------------------ > > Message: 2 > Date: Fri, 30 May 2014 01:51:05 -0700 > From: Jim Bridgewater <jwbwa...@gmail.com> > To: nupic@lists.numenta.org > Subject: Re: [nupic-discuss] Some observations from Ian's Fall 2013 SP > demo > Message-ID: > <camma+tjpw4es9ovrvy0uf1+stxxtkktpuzqt7jmtzoxxsw8...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I realized that I was thinking about the number of possible output > values incorrectly. An SP that has 16 columns and is configured to > have 8 active columns technically has over 12,000 possible output > values, but only 2 of those output values have no active columns in > common so an SP configured this way can only distinguish between 2 > unrelated input patterns. In the tests I ran the two patterns are > blank and non-blank. I have updated the attached document to reflect > this realization. > > On Thu, May 29, 2014 at 4:32 PM, Jim Bridgewater <jwbwa...@gmail.com> wrote: >> Hi everyone, >> >> I'm working on an optical character recognition project for the season >> of nupic this summer. I've been playing around with Ian Danforth's >> Fall 2013 hackathon demo of the spatial pooler and put together the >> attached pdf document which shows some observations of how the spatial >> pooler behaves with different numbers of active columns. Maybe some >> people on this list have thought about this before and have some >> comments about why the SP displays this behavior. >> >> >> >> ---------- Forwarded message ---------- >> From: Scott Purdy <sc...@numenta.org> >> Date: Thu, May 29, 2014 at 4:02 PM >> Subject: Re: Update >> To: Jim Bridgewater <jwbwa...@gmail.com> >> >> >> The write up is quite nice, thanks for putting that together. You >> might send it to the discuss list in case anyone else is curious. I >> don't think the results are particularly suprising. In particular, the >> SP is known to not always perform well when the number of input bits >> or columns is small or there are a small number of input patterns. I >> will try to discuss with Subutai tomorrow to confirm that there is >> nothing wrong with your experimental method and see if I can get more >> details on the onion charts. >> >> I think the onion charts will make more sense on a more sophisticated >> data set as it would likely suffer from the same issues with small >> numbers of inputs. >> >> >> On Thu, May 29, 2014 at 1:50 PM, Jim Bridgewater <jwbwa...@gmail.com> wrote: >>> >>> Hi Scott, >>> >>> I've been playing with Ian's demo a bit to get a feel for how the >>> spatial pooler works. I suppose it was intended to be used with only >>> a small % of active columns, but I do find it interesting how poorly >>> it performs when 50% of the columns are active since it has the >>> greatest number of output values in this case. See attached pdf. >>> >>> I want to work on a way of showing the results that makes it easier to >>> see how the representation of each input image changes as the sp is >>> trained and whether the outputs are different for each unique input. >>> I was thinking about converting the sp output to a hexadecimal value >>> and plotting that vs time for each input image. >>> >>> Maybe you guys could send me some examples of the onion graphs Subutai >>> mentioned so I can see how they work. >>> >>> -- >>> Jim Bridgewater, PhD >>> Arizona State University >>> 480-227-9592 >> >> >> >> >> -- >> Jim Bridgewater, PhD >> Arizona State University >> 480-227-9592 > > > > -- > Jim Bridgewater, PhD > Arizona State University > 480-227-9592 > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: update.pdf > Type: application/pdf > Size: 274713 bytes > Desc: not available > URL: > <http://lists.numenta.org/pipermail/nupic_lists.numenta.org/attachments/20140530/84824396/attachment.pdf> > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > nupic mailing list > nupic@lists.numenta.org > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > > > ------------------------------ > > End of nupic Digest, Vol 13, Issue 45 > ************************************* -- Jim Bridgewater, PhD Arizona State University 480-227-9592 _______________________________________________ nupic mailing list nupic@lists.numenta.org http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
