Noah, OpenCV, as you know, is basically a toolkit, not an integrated system....
Right now, indeed, the vision processing we have hooked up to OpenCog is restricted to -- face detection, face tracking, face recognition -- visual saliency identification -- luminance-change detection which is all pretty simple... We have previously experimented with using kinfu to make a 3D map of a robot's surroundings.... OpenCog's TimeSpaceMap is well-suited to represent the output of kinfu (or similar) in a way that's integrated with the Atomspace... We have also played a bit with Region-CNNs, as a way of identifying what objects are where in a visual scene (initially from a closed class of objects) So if I were going to integrate additional external vision tools with OpenCog, I'd probably start with kinfu-or-similar, plus (Region-CNN-with-trained-models)-or-similar... Medium term, it's more interesting to integrate deep NN vision into OpenCog, which Yenat is working on in our Addis office, but that's a research project, whereas feeding output of kinfu and Region-CNN into OpenCog is "just" complex software integration and training/tuning, not really original research... Anyway I am curious what specific visual functions you are thinking of adding? -- Ben G On Sat, Sep 17, 2016 at 9:54 AM, Linas Vepstas <[email protected]> wrote: > > > On Fri, Sep 16, 2016 at 8:41 PM, Noah Bliss <[email protected]> wrote: >> >> Thank you for the info Linas, >> >> I'll look at the current code and see if I can get a more complete >> implementation of OpenCV started. You mentioned another dev's overly simple >> integration which, while better than nothing, hardly fulfills our goal or >> utilizes the full potential of OpenCV. >> >> With luck maybe I can get the visual end of opencog a bit more useful than >> a glorified motion detector. :P > > > I think the "saliency detector" code is buried somewhere in here: > https://github.com/hansonrobotics/HEAD -- building and running that is > probably the easiest way to get a working end-to-end demo. > >> Thanks again! I'll report back any major advances, otherwise check the >> pull requests and maybe my branch of you get curious. >> >> As a side, if I am not mistaken, atomspace does most of its storage in sql >> right? > > Only if you actually turn that on. Otherwise everything is in RAM. > >> >> Perhaps I could see about offloading visual processing to a dedicated >> machine along with whatever camera/sensor is being used, and get that set up >> with an "atomspace client" that could dump pre-formatted atoms straight into >> the db. > > netcat does that. The python snippet with netcat was an example. > > For everything else, we use ROS. There's a bit of a learning curve for ROS, > but its the ideal way for running multi-machine, distributed processing. > > --linas >> >> If there aren't any logistical restrictions to this method, it could >> provide a more modular design to opencog and also reduce unnecessary primary >> server strain. >> >> Noah B. >> >> >> On Fri, Sep 16, 2016, 12:25 Linas Vepstas <[email protected]> wrote: >>> >>> Hi Noah, >>> >>> Sounds like a good idea! We currently do not have any clear-cut plans, >>> but let me tell you a little about what has been done so far. Currently, >>> the main visual interface is in the repo >>> https://github.com/opencog/ros-behavior-scripting/ ... and its pretty >>> pathetic as vision goes. It does use OpenCV, but only as input into a >>> hacked version of pi_vision, and that is used to detect human faces, and map >>> them to 3D locations. Actually, I think that the pi_vision has been >>> replaced by the CMT tracker, recently, which seems to work a bit better, >>> maybe. The ID's of the faces are placed as atoms into the atomspace. Its >>> super-simple, and super-low-bandwidth: basically a handful of atoms that say >>> "I can see face 42 now".... and that's it. The 3D locations of the faces are >>> NOT kept in the atomspace -- they are kept off-line, mostly because of >>> bandwidth concerns. 30 frames a second of x,y,z points is not a lot, but is >>> pointless, because we currently can't do reasoning with that info, anyway. >>> >>> Re: new or moving objects: someone recently added support for "visual >>> saliency", and I flamed them a bit for how it was done: the information >>> pumped into the atomspace was a very simple message: "something is happening >>> in the visual field!" which is kind-of useless. Tell me, at least, is it >>> big, or is it small, near or far, moving fast or moving slowly? Is it >>> "windmilling" i.e. moving-without-moving, like clapping hands? or just >>> someone standing there, swaying side to side? >>> >>> With that kind of info, one can, at least, do some sort of scripted >>> reactions: the robot can say "Hey I think I see a fly" or "what's that going >>> on behind your left shoulder?" Anyway, that general kind of input is >>> handled by https://github.com/opencog/ros-behavior-scripting/ .. the >>> actual "state" of what is seen, what's going on is in src/self-model.scm >>> and so additional stuff can be added there, like "I see something small >>> moving"... scripted responses are in the file "behavior.scm", so if >>> something is seen, that is where you can script a response. >>> >>> All of the above is "short term". In the long term, it really has to be >>> learning. For that, it has to be something completely different. This email >>> is kind-of long already but ... the idea is to pattern-mine: "if 33% of the >>> screen is red and X happened at the same time, this is important, remember >>> and learn that!" Except this never happens. So instead, lets (randomly) >>> try "if 33% of the screen is blue and X happened at the same time..." well, >>> hey, that DOES happen, it means you went outside on a sunny day. So this >>> should be remembered and recorded as an important filter-event, that >>> converts visual stuff into knowledge. The tricky part here is that this is >>> ... CPU intensive, requires lots of training. Its a much much harder >>> problem. But.. enough. >>> >>> Anyway, the upshot is: "there are no rules" -- we've done very little >>> almost nothing with vision, so you can do anything you want. >>> >>> Re: python for opencog -- your best bet is to just poke atoms into the >>> atomspace with netcat, for example, like here: >>> >>> https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/face_atomic.py#L82-L87 >>> called from here: >>> >>> https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/face_atomic.py#L62-L66 >>> >>> and uses netcat here: >>> >>> https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/netcat.py >>> >>> Currently, this is probably the best way to use python to get data into >>> and out of the atomspace. >>> >>> --linas >>> >>> >>> On Fri, Sep 16, 2016 at 10:37 AM, Noah Bliss <[email protected]> wrote: >>>> >>>> I'm going to be showing a great deal of ignorance in this post, but who >>>> knows, it might help. >>>> >>>> I understand an issue recently discussed with embodiment concerns >>>> methods for processing visual input. It's well known that at this time >>>> sending raw video into atomspace is a bad idea and that humans have built >>>> in >>>> visual processors that assist our conscious minds in understanding what our >>>> eyes see. (Obvious simple example being that the image is preflipped). >>>> >>>> I understand opencog has (in some form) a python api which leads me to >>>> think using the visual processing engine OpenCV may not be a bad idea. It >>>> has a fantastic python api, allows for exporting specific data from raw >>>> video such as "33% of the screen is red", or there are 2 lines in the >>>> field >>>> of view." it also has a PHENOMINAL foreground/background separation engine >>>> that allows only a processing of new or moving objects in the field of >>>> view. >>>> >>>> While a more mature opencog engine may prefer a more "raw" processor, I >>>> see OpenCV as a great place to start for getting useful information into >>>> atomspace quickly. >>>> >>>> I have yet to start work on this, heck, I have yet to fully learn the >>>> ropes of the current opencog system, but I wanted to at least drop the info >>>> here in case anyone else had comments or wanted to get a head-start on me. >>>> >>>> Best regards my friends. >>>> Noah B. >>>> >>>> PS: My personal experience with OpenCV was specifically dealing with >>>> automated turrets. There are great YouTube examples of using OpenCV for >>>> face-tracking webcams attached to servos, and blob isolating security >>>> cameras if you wanted specific examples to look up. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "opencog" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>>> >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/opencog. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/opencog/1baaeade-567a-4456-aaa3-85e2b003fc7b%40googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "opencog" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/opencog. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/opencog/CAHrUA37v2zxE7nTbqrBtw65k539v_wW1JLX2%3D2jgC3bkDoyqsw%40mail.gmail.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "opencog" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/opencog. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/opencog/CABpkOB-4HYkmtoqnBNpWaqdRKwou-w9CPevYOtNDYxGiJL9N%3Dg%40mail.gmail.com. >> >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "opencog" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/opencog. > To view this discussion on the web visit > https://groups.google.com/d/msgid/opencog/CAHrUA34J1i2qe-KTOUEZ%2B8gXXWhW1jmUoDWQt2H%3DTY7copfXRw%40mail.gmail.com. > > For more options, visit https://groups.google.com/d/optout. -- Ben Goertzel, PhD http://goertzel.org Super-benevolent super-intelligence is the thought the Global Brain is currently struggling to form... -- You received this message because you are subscribed to the Google Groups "opencog" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/opencog. To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CACYTDBf1qsF21PyHU9V7t_nRPNtyiqn5FMjOtPeyFFrqMBzNhg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
