The concept of building a 3d map of the environment in atomspace certainly would be a better goal of there had to be a choice between the two. I'll definitely read up on kinfu before starting any sort of work.
A few simple examples of information gained through a more mature OpenCV implementation could consist of the following: A simple place to start that would have little overhead and export atoms easily used could be: Knowing environment. Consistent items not the focus. Blob statistics, delimited by motion Size of blob Color of blob Location on fov Speed and direction Acceleration Advanced sampling: Division of blob into sections, quarters horizontally, Shape/size/color/edge flat or rounded statistics of that quadrant Vertical division by the same standards. Obviously this would not be the end. We could divide a blob into more slices, account for variation in background, etc. We would need a more advanced way to get more information from a visual feed to get it anywhere near human-like. But we could at least get more useful data than we currently are. This kind of implementation could potentially augment other more "logical" representations of the environment by providing a more analog "eye-like" processing system. It also has the advantage of being potentially faster to implement and scale. I don't see this implementation ever being a replacement for any sort of 3d map formation, but rather a low-overhead way of quickly making sense of raw visual input without pumping raw video into atomspace. On Sat, Sep 17, 2016, 01:06 Ben Goertzel <b...@goertzel.org> wrote: > Noah, > > OpenCV, as you know, is basically a toolkit, not an integrated system.... > > Right now, indeed, the vision processing we have hooked up to OpenCog > is restricted to > > -- face detection, face tracking, face recognition > -- visual saliency identification > -- luminance-change detection > > which is all pretty simple... > > We have previously experimented with using kinfu to make a 3D map of a > robot's surroundings.... OpenCog's TimeSpaceMap is well-suited to > represent the output of kinfu (or similar) in a way that's integrated > with the Atomspace... > > We have also played a bit with Region-CNNs, as a way of identifying > what objects are where in a visual scene (initially from a closed > class of objects) > > So if I were going to integrate additional external vision tools with > OpenCog, I'd probably start with kinfu-or-similar, plus > (Region-CNN-with-trained-models)-or-similar... > > Medium term, it's more interesting to integrate deep NN vision into > OpenCog, which Yenat is working on in our Addis office, but that's a > research project, whereas feeding output of kinfu and Region-CNN into > OpenCog is "just" complex software integration and training/tuning, > not really original research... > > Anyway I am curious what specific visual functions you are thinking of > adding? > > -- Ben G > > > > On Sat, Sep 17, 2016 at 9:54 AM, Linas Vepstas <linasveps...@gmail.com> > wrote: > > > > > > On Fri, Sep 16, 2016 at 8:41 PM, Noah Bliss <l3viat...@gmail.com> wrote: > >> > >> Thank you for the info Linas, > >> > >> I'll look at the current code and see if I can get a more complete > >> implementation of OpenCV started. You mentioned another dev's overly > simple > >> integration which, while better than nothing, hardly fulfills our goal > or > >> utilizes the full potential of OpenCV. > >> > >> With luck maybe I can get the visual end of opencog a bit more useful > than > >> a glorified motion detector. :P > > > > > > I think the "saliency detector" code is buried somewhere in here: > > https://github.com/hansonrobotics/HEAD -- building and running that is > > probably the easiest way to get a working end-to-end demo. > > > >> Thanks again! I'll report back any major advances, otherwise check the > >> pull requests and maybe my branch of you get curious. > >> > >> As a side, if I am not mistaken, atomspace does most of its storage in > sql > >> right? > > > > Only if you actually turn that on. Otherwise everything is in RAM. > > > >> > >> Perhaps I could see about offloading visual processing to a dedicated > >> machine along with whatever camera/sensor is being used, and get that > set up > >> with an "atomspace client" that could dump pre-formatted atoms straight > into > >> the db. > > > > netcat does that. The python snippet with netcat was an example. > > > > For everything else, we use ROS. There's a bit of a learning curve for > ROS, > > but its the ideal way for running multi-machine, distributed processing. > > > > --linas > >> > >> If there aren't any logistical restrictions to this method, it could > >> provide a more modular design to opencog and also reduce unnecessary > primary > >> server strain. > >> > >> Noah B. > >> > >> > >> On Fri, Sep 16, 2016, 12:25 Linas Vepstas <linasveps...@gmail.com> > wrote: > >>> > >>> Hi Noah, > >>> > >>> Sounds like a good idea! We currently do not have any clear-cut plans, > >>> but let me tell you a little about what has been done so far. > Currently, > >>> the main visual interface is in the repo > >>> https://github.com/opencog/ros-behavior-scripting/ ... and its pretty > >>> pathetic as vision goes. It does use OpenCV, but only as input into a > >>> hacked version of pi_vision, and that is used to detect human faces, > and map > >>> them to 3D locations. Actually, I think that the pi_vision has been > >>> replaced by the CMT tracker, recently, which seems to work a bit > better, > >>> maybe. The ID's of the faces are placed as atoms into the atomspace. > Its > >>> super-simple, and super-low-bandwidth: basically a handful of atoms > that say > >>> "I can see face 42 now".... and that's it. The 3D locations of the > faces are > >>> NOT kept in the atomspace -- they are kept off-line, mostly because of > >>> bandwidth concerns. 30 frames a second of x,y,z points is not a lot, > but is > >>> pointless, because we currently can't do reasoning with that info, > anyway. > >>> > >>> Re: new or moving objects: someone recently added support for "visual > >>> saliency", and I flamed them a bit for how it was done: the information > >>> pumped into the atomspace was a very simple message: "something is > happening > >>> in the visual field!" which is kind-of useless. Tell me, at least, is > it > >>> big, or is it small, near or far, moving fast or moving slowly? Is it > >>> "windmilling" i.e. moving-without-moving, like clapping hands? or just > >>> someone standing there, swaying side to side? > >>> > >>> With that kind of info, one can, at least, do some sort of scripted > >>> reactions: the robot can say "Hey I think I see a fly" or "what's that > going > >>> on behind your left shoulder?" Anyway, that general kind of input is > >>> handled by https://github.com/opencog/ros-behavior-scripting/ .. > the > >>> actual "state" of what is seen, what's going on is in > src/self-model.scm > >>> and so additional stuff can be added there, like "I see something small > >>> moving"... scripted responses are in the file "behavior.scm", so if > >>> something is seen, that is where you can script a response. > >>> > >>> All of the above is "short term". In the long term, it really has to be > >>> learning. For that, it has to be something completely different. This > email > >>> is kind-of long already but ... the idea is to pattern-mine: "if 33% > of the > >>> screen is red and X happened at the same time, this is important, > remember > >>> and learn that!" Except this never happens. So instead, lets > (randomly) > >>> try "if 33% of the screen is blue and X happened at the same time..." > well, > >>> hey, that DOES happen, it means you went outside on a sunny day. So > this > >>> should be remembered and recorded as an important filter-event, that > >>> converts visual stuff into knowledge. The tricky part here is that > this is > >>> ... CPU intensive, requires lots of training. Its a much much harder > >>> problem. But.. enough. > >>> > >>> Anyway, the upshot is: "there are no rules" -- we've done very little > >>> almost nothing with vision, so you can do anything you want. > >>> > >>> Re: python for opencog -- your best bet is to just poke atoms into the > >>> atomspace with netcat, for example, like here: > >>> > >>> > https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/face_atomic.py#L82-L87 > >>> called from here: > >>> > >>> > https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/face_atomic.py#L62-L66 > >>> > >>> and uses netcat here: > >>> > >>> > https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/netcat.py > >>> > >>> Currently, this is probably the best way to use python to get data into > >>> and out of the atomspace. > >>> > >>> --linas > >>> > >>> > >>> On Fri, Sep 16, 2016 at 10:37 AM, Noah Bliss <l3viat...@gmail.com> > wrote: > >>>> > >>>> I'm going to be showing a great deal of ignorance in this post, but > who > >>>> knows, it might help. > >>>> > >>>> I understand an issue recently discussed with embodiment concerns > >>>> methods for processing visual input. It's well known that at this time > >>>> sending raw video into atomspace is a bad idea and that humans have > built in > >>>> visual processors that assist our conscious minds in understanding > what our > >>>> eyes see. (Obvious simple example being that the image is preflipped). > >>>> > >>>> I understand opencog has (in some form) a python api which leads me to > >>>> think using the visual processing engine OpenCV may not be a bad > idea. It > >>>> has a fantastic python api, allows for exporting specific data from > raw > >>>> video such as "33% of the screen is red", or there are 2 lines in > the field > >>>> of view." it also has a PHENOMINAL foreground/background separation > engine > >>>> that allows only a processing of new or moving objects in the field > of view. > >>>> > >>>> While a more mature opencog engine may prefer a more "raw" processor, > I > >>>> see OpenCV as a great place to start for getting useful information > into > >>>> atomspace quickly. > >>>> > >>>> I have yet to start work on this, heck, I have yet to fully learn the > >>>> ropes of the current opencog system, but I wanted to at least drop > the info > >>>> here in case anyone else had comments or wanted to get a head-start > on me. > >>>> > >>>> Best regards my friends. > >>>> Noah B. > >>>> > >>>> PS: My personal experience with OpenCV was specifically dealing with > >>>> automated turrets. There are great YouTube examples of using OpenCV > for > >>>> face-tracking webcams attached to servos, and blob isolating security > >>>> cameras if you wanted specific examples to look up. > >>>> > >>>> > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "opencog" group. > >>>> To unsubscribe from this group and stop receiving emails from it, send > >>>> an email to opencog+unsubscr...@googlegroups.com. > >>>> > >>>> > >>>> To post to this group, send email to opencog@googlegroups.com. > >>>> Visit this group at https://groups.google.com/group/opencog. > >>>> To view this discussion on the web visit > >>>> > https://groups.google.com/d/msgid/opencog/1baaeade-567a-4456-aaa3-85e2b003fc7b%40googlegroups.com > . > >>>> For more options, visit https://groups.google.com/d/optout. > >>> > >>> -- > >>> You received this message because you are subscribed to a topic in the > >>> Google Groups "opencog" group. > >>> To unsubscribe from this topic, visit > >>> https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe. > >>> To unsubscribe from this group and all its topics, send an email to > >>> opencog+unsubscr...@googlegroups.com. > >>> To post to this group, send email to opencog@googlegroups.com. > >>> Visit this group at https://groups.google.com/group/opencog. > >>> To view this discussion on the web visit > >>> > https://groups.google.com/d/msgid/opencog/CAHrUA37v2zxE7nTbqrBtw65k539v_wW1JLX2%3D2jgC3bkDoyqsw%40mail.gmail.com > . > >>> For more options, visit https://groups.google.com/d/optout. > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "opencog" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to opencog+unsubscr...@googlegroups.com. > >> To post to this group, send email to opencog@googlegroups.com. > >> Visit this group at https://groups.google.com/group/opencog. > >> To view this discussion on the web visit > >> > https://groups.google.com/d/msgid/opencog/CABpkOB-4HYkmtoqnBNpWaqdRKwou-w9CPevYOtNDYxGiJL9N%3Dg%40mail.gmail.com > . > >> > >> For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "opencog" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to opencog+unsubscr...@googlegroups.com. > > To post to this group, send email to opencog@googlegroups.com. > > Visit this group at https://groups.google.com/group/opencog. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/opencog/CAHrUA34J1i2qe-KTOUEZ%2B8gXXWhW1jmUoDWQt2H%3DTY7copfXRw%40mail.gmail.com > . > > > > For more options, visit https://groups.google.com/d/optout. > > > > -- > Ben Goertzel, PhD > http://goertzel.org > > Super-benevolent super-intelligence is the thought the Global Brain is > currently struggling to form... > > -- > You received this message because you are subscribed to a topic in the > Google Groups "opencog" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > opencog+unsubscr...@googlegroups.com. > To post to this group, send email to opencog@googlegroups.com. > Visit this group at https://groups.google.com/group/opencog. > To view this discussion on the web visit > https://groups.google.com/d/msgid/opencog/CACYTDBf1qsF21PyHU9V7t_nRPNtyiqn5FMjOtPeyFFrqMBzNhg%40mail.gmail.com > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "opencog" group. To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscr...@googlegroups.com. To post to this group, send email to opencog@googlegroups.com. Visit this group at https://groups.google.com/group/opencog. To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CABpkOB_mL24sT40JvcL%3DkE6BnqHHMpWGURJdWoWmVJd-p%3D2G0A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
