Neal, OK I read the paper ( http://infoscience.epfl.ch/record/175537/files/2069.pdf for anyone interested). Let me make sure I understand it:
- It looks like you get out a 512 bit string for each image, or are there many of these for each image? i.e. per patch - This descriptor is a compact representation of useful keypoints in the image - Those keypoints are constructed using a novel retina-inspired topology and then subsampled experimentally. - To see if a target object exists in an image you can begin comparing the 512 bit string of a training image with the input image. (To do this more efficiently you start by only using 16 bits which represent a coarse whole-image field) I'm not quite clear on if this is done with a sliding window across the new input, perhaps you can clarify that. If this is the case then generating the descriptor is the equivalent of what the SP is trying to do, feature extraction. The thesis of these kind of invariant feature systems is that they can craft features up front, and use them for object detection. The SP is much more in the ML camp of "the system should learn the features itself." It would be interesting to see what the SP makes of being fed a bunch of fairly dense representations like the 512 vector, but I don't have a strong intuition about what would happen. Try it! One thing you could try is to use the topology suggested by FREAK to set up the initial connection pools for the SP. You wouldn't end up using FREAK at all if you did it that way. The center neurons in the SP would have small dense receptive fields directly to the input image, and the peripheral neurons would have broad, less dense receptive fields. During training you could either have the SP fixed in place or you could slide it around the input image to simulate saccades. After feeding in the training videos multiple times (I suspect it will take multiple training runthroughs to stabilize) you can then feed in the test video. Ultimately you're trying to solve a 6 class labeling problem. What you're asking the system to do is to find transitions between feature vectors that are common within a class, but distinct from all the rest of the classes. Then your classifier will have to tease apart which TP states correspond to which class. Hopefully the TP will have learned which features are useful to make predictions and will quickly lock-on to the sequence of feature transitions unique to the class of video it's being presented! Another strategy that Subutai reminded me of that one of the participants in the last hackathon used, was to train a separate network on each class, and then use the anomaly score to see which of the networks was "least surprised" by the new input. The min of the anomaly scores is then your class. An interesting question is "how many frames do you need of a test video to reach high confidence?" Ian On Tue, Oct 29, 2013 at 6:28 AM, Ian Danforth <[email protected]>wrote: > Neal, > > I think this is a great project! I am going to have to dig into FREAK > before I can say how the SP will handle its feature set. But since the SP > was originally designed to handle topology, you might want to look into the > passthrough encoder and using the output of FREAK as a direct input to the > SP as long as it is a sparse vector. > > Also, right now, the 'pooling' aspect of the TP isn't enabled but I think > your project could definitely take advantage of it, so working with a few > others to get that back in shape might be a great contribution. > > A couple questions, what is your timeline and level of comfort with C++ > and/or Python? > > Ian > > > On Mon, Oct 28, 2013 at 9:31 PM, Neal Donnelly <[email protected]> wrote: > >> Hey everybody, >> >> I'm a senior at Princeton University looking to apply NuPIC as my senior >> thesis. Specifically, I'm intrigued to see how well it works for computer >> vision as I've been learning a lot about traditional approaches here at >> school. I'd like to run my plan by the community to get some feedback >> before I dive in. >> >> I would like to build a pipeline for video classification. I understand >> the online prediction framework isn't designed for classification, but I >> want to get a sense of how NuPIC can perform on traditional problems as the >> tool to build high level features. >> >> I would read in each video with OpenCV <http://opencv.org/>, and for >> each frame I would extract the keypoints with the >> FREAK<http://infoscience.epfl.ch/record/175537/files/2069.pdf>feature >> descriptor and the corresponding detector. FREAK appeals to me >> because its implemented in OpenCV, its fast, it produces state of the art >> results, and it draws direct inspiration from the retina. Each frame would >> be turned into a row in a CSV file with a timestamp and a list of features >> and their descriptors. >> >> The CSV file would then be fed into the swarm, and then I would run the >> resulting model. I would feed all the training videos to the model once >> through to get it to learn certain types of features, then for each >> training video feed it through again and take the top level representation >> as the representation of the video. Once I have a top level representation >> of each video, I'll feed those to a Bayesian classifier and see if I can >> train it to recognize the types of videos. My plan is to start with the six >> types of human actions in this >> dataset<http://www.nada.kth.se/cvap/actions/>from KTH. >> >> My questions are >> >> 1) Do I have to translate all my videos into CSVs? How do I separate the >> different video clips so that I can feed them all in - how can I put them >> all in the same file if they all need a time stamp? >> >> 2) Do I want to just build one model or do I need a separate one for each >> classification class? >> >> 3) What doesn't make sense about my plan? >> >> Thanks so much! >> Neal Donnelly >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
