Hey everybody, I'm a senior at Princeton University looking to apply NuPIC as my senior thesis. Specifically, I'm intrigued to see how well it works for computer vision as I've been learning a lot about traditional approaches here at school. I'd like to run my plan by the community to get some feedback before I dive in.
I would like to build a pipeline for video classification. I understand the online prediction framework isn't designed for classification, but I want to get a sense of how NuPIC can perform on traditional problems as the tool to build high level features. I would read in each video with OpenCV <http://opencv.org/>, and for each frame I would extract the keypoints with the FREAK<http://infoscience.epfl.ch/record/175537/files/2069.pdf>feature descriptor and the corresponding detector. FREAK appeals to me because its implemented in OpenCV, its fast, it produces state of the art results, and it draws direct inspiration from the retina. Each frame would be turned into a row in a CSV file with a timestamp and a list of features and their descriptors. The CSV file would then be fed into the swarm, and then I would run the resulting model. I would feed all the training videos to the model once through to get it to learn certain types of features, then for each training video feed it through again and take the top level representation as the representation of the video. Once I have a top level representation of each video, I'll feed those to a Bayesian classifier and see if I can train it to recognize the types of videos. My plan is to start with the six types of human actions in this dataset<http://www.nada.kth.se/cvap/actions/>from KTH. My questions are 1) Do I have to translate all my videos into CSVs? How do I separate the different video clips so that I can feed them all in - how can I put them all in the same file if they all need a time stamp? 2) Do I want to just build one model or do I need a separate one for each classification class? 3) What doesn't make sense about my plan? Thanks so much! Neal Donnelly
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
