Hello Marek. Thanks for taking the time to explain in such detail. It makes sense. And Matt, sorry I didn't answer you, for some reason I didn't see your message.
> ---------- Forwarded message ---------- > From: Marek Otahal <[email protected]> > To: "NuPIC general mailing list." <[email protected]> > Cc: > Date: Thu, 17 Dec 2015 22:46:29 +0100 > Subject: Re: Order of the training data matters? > Hey guys! On Fri, Dec 4, 2015 at 3:50 AM, Matthew Taylor <[email protected]> wrote: > I don't understand what is meant by "putting the like sequences > together". If there are true temporal sequences, wouldn't there only > be one order (chronological order)? > No. I think I understand the problem at hand: you have temporal sequences where some are alike (with overlapping sub-sequences), these are chronologically ordered within the sequence, but the sequences in the group/dataset are independent. Example be hourly GPS-positional data on a group of 20 people. So, for example, you have sequences { {1,3,a,b,c,3,9},{2,2,a,b,c,9,9}, {5,5,5,5},{5,5,1,a}}. You call resets after each sequence as the people move independently. Now Sebastian's observation is: HTM learns better { {1,3,a,b,c,3,9},{2,2,a,b,c,9,9}, {5,5,5,5},{5,5,1,a}} > than { {1,3,a,b,c,3,9}, {5,5,5,5},{5,5,1,a}, {2,2,a,b,c,9,9}} and that way better than >>> { {1,3,a,b,c,3,9}, {5,5,5,5},"1000-others",{5,5,1,a}, {2,2,a,b,c,9,9}}. Where sequences containing "a,b,c" are alike. This is sort of OK for HTM as a streaming model, and NOT OK for a neural network working with an independent dataset - this is complicated and has to be taken into account for benchmarking against other NNs on classical datasets; so a disadvantage of HTM - it cannot treat iid data. Why? (I'll go to extremes to show easily) SpatialPooler: - 1000x sequence with "abc" => synapses grow over inputs of 'a,b,c' and react to them 100%. - resets do not affect SP in any way! (maybe this should be changed?) - a new sequence with ABC will have less work on recognizing ABC (this ABC sequence could in real be arbitrarily long, eg 100 points). And will cause stronger feedback in upper TP, this is even some concept from psychology, "anchoring" I think. When you remember easier if you encounter some known/familiar facts. Take understanding a home address in English vs. Mandarin! TemporalPooler: - resets drop active contectual connections (ok) - but reset is an artificial concept in HTM, I'm not sure how well it's thought through. - similar to SP, some statistical history still affects the reseted state - eg where synapses have grown during the past process. In an extreme case, all synapses will move "to the left" and you may not at all detect a change "on the right" when sequences are ordered one way, or different result with different ordering. Matters? Why it "does not matter" for HTM as a sensory processing model - cortex? Because we live continuously, always building on past experiences. We do not encounter "resets" (maybe some clinical cases). For benchmarking on datasets, you could do both ways ;) A) train on randomly shuffeled sentences - this is the worst case scenario. B) in preprocessing, sort the seq. nicely (alike together) - this equals to a student skimming through the whole book first to have an overview and then studying the chapters in the best-for-him order. Cheers, Mark --------- > Matt Taylor > OS Community Flag-Bearer > Numenta > > > On Wed, Dec 2, 2015 at 10:09 AM, Jason Xie <[email protected]> > wrote: > > Conceptually I think it makes a lot of sense that nupic learns faster for > > specific inputs, if you group inputs individually. > > > > When many sequences are entered at once, the SP can better learn how to > > represent each sequence as it is entered and the TM's lookup table will > > learn a very strong association. > > > > It's like in life, it's easier to learn one thing at a time than > everything > > at once. > > > > On Wed, Dec 2, 2015 at 9:24 AM, Sebastián Narváez <[email protected]> > > wrote: > >> > >> I have a training data with various sequences, which I pass trough a > nupic > >> structure (made of SPs, TMs and a CLAClassifier on top). Some of the > >> sequences are alike (there are only one or two elements that change > between > >> them). I've noticed that if I put the alike sequences together, one > after > >> the other, nupic will learn the differences better than if I put them > >> appart. I reset() all the TMs when a sequence ends, so my guess is that > it > >> has to do with the classifier. Anyone knows why could this happen? > >> Note that I have not done any serious tests about this. I could, but it > >> would take some time. > > > > > > -- Marek Otahal :o)
