Ah, OK, I get it.... Yeah, having openCog know the size and color and direction of blobs would be nice... I'm not quite sure what to do with it immediately though...
I mean, for the "deep learning vision in OpenCog" direction we're gonna feed fine-grained visual data (pixels, voxels) into neural nets wrapped in Grounded SchemaNodes associated with RegionNodes in the TimeSpaceMap... OTOH, for hand-coding simple response rules for a robot or avatar controlled by OpenCog, you kinda need to know what you're seeing not just that you're seeing a red blob in a certain position, moving toward a blue blob... So one could argue that knowing where blobs of various colors are and where they're moving is -- too low-level for hand-coding response rules for an opencog agent -- not low-level enough to fuel deep visual learning in opencog ... but maybe I'm misunderstanding something? ben On Sat, Sep 17, 2016 at 11:18 PM, Noah Bliss <[email protected]> wrote: > The concept of building a 3d map of the environment in atomspace certainly > would be a better goal of there had to be a choice between the two. I'll > definitely read up on kinfu before starting any sort of work. > > A few simple examples of information gained through a more mature OpenCV > implementation could consist of the following: > > A simple place to start that would have little overhead and export atoms > easily used could be: > > Knowing environment. Consistent items not the focus. > > Blob statistics, delimited by motion > Size of blob > Color of blob > Location on fov > Speed and direction > Acceleration > > Advanced sampling: > Division of blob into sections, quarters horizontally, > Shape/size/color/edge flat or rounded statistics of that quadrant > Vertical division by the same standards. > > Obviously this would not be the end. We could divide a blob into more > slices, account for variation in background, etc. We would need a more > advanced way to get more information from a visual feed to get it anywhere > near human-like. But we could at least get more useful data than we > currently are. > > This kind of implementation could potentially augment other more "logical" > representations of the environment by providing a more analog "eye-like" > processing system. It also has the advantage of being potentially faster to > implement and scale. > > I don't see this implementation ever being a replacement for any sort of 3d > map formation, but rather a low-overhead way of quickly making sense of raw > visual input without pumping raw video into atomspace. > > > On Sat, Sep 17, 2016, 01:06 Ben Goertzel <[email protected]> wrote: >> >> Noah, >> >> OpenCV, as you know, is basically a toolkit, not an integrated system.... >> >> Right now, indeed, the vision processing we have hooked up to OpenCog >> is restricted to >> >> -- face detection, face tracking, face recognition >> -- visual saliency identification >> -- luminance-change detection >> >> which is all pretty simple... >> >> We have previously experimented with using kinfu to make a 3D map of a >> robot's surroundings.... OpenCog's TimeSpaceMap is well-suited to >> represent the output of kinfu (or similar) in a way that's integrated >> with the Atomspace... >> >> We have also played a bit with Region-CNNs, as a way of identifying >> what objects are where in a visual scene (initially from a closed >> class of objects) >> >> So if I were going to integrate additional external vision tools with >> OpenCog, I'd probably start with kinfu-or-similar, plus >> (Region-CNN-with-trained-models)-or-similar... >> >> Medium term, it's more interesting to integrate deep NN vision into >> OpenCog, which Yenat is working on in our Addis office, but that's a >> research project, whereas feeding output of kinfu and Region-CNN into >> OpenCog is "just" complex software integration and training/tuning, >> not really original research... >> >> Anyway I am curious what specific visual functions you are thinking of >> adding? >> >> -- Ben G >> >> >> >> On Sat, Sep 17, 2016 at 9:54 AM, Linas Vepstas <[email protected]> >> wrote: >> > >> > >> > On Fri, Sep 16, 2016 at 8:41 PM, Noah Bliss <[email protected]> wrote: >> >> >> >> Thank you for the info Linas, >> >> >> >> I'll look at the current code and see if I can get a more complete >> >> implementation of OpenCV started. You mentioned another dev's overly >> >> simple >> >> integration which, while better than nothing, hardly fulfills our goal >> >> or >> >> utilizes the full potential of OpenCV. >> >> >> >> With luck maybe I can get the visual end of opencog a bit more useful >> >> than >> >> a glorified motion detector. :P >> > >> > >> > I think the "saliency detector" code is buried somewhere in here: >> > https://github.com/hansonrobotics/HEAD -- building and running that is >> > probably the easiest way to get a working end-to-end demo. >> > >> >> Thanks again! I'll report back any major advances, otherwise check the >> >> pull requests and maybe my branch of you get curious. >> >> >> >> As a side, if I am not mistaken, atomspace does most of its storage in >> >> sql >> >> right? >> > >> > Only if you actually turn that on. Otherwise everything is in RAM. >> > >> >> >> >> Perhaps I could see about offloading visual processing to a dedicated >> >> machine along with whatever camera/sensor is being used, and get that >> >> set up >> >> with an "atomspace client" that could dump pre-formatted atoms straight >> >> into >> >> the db. >> > >> > netcat does that. The python snippet with netcat was an example. >> > >> > For everything else, we use ROS. There's a bit of a learning curve for >> > ROS, >> > but its the ideal way for running multi-machine, distributed processing. >> > >> > --linas >> >> >> >> If there aren't any logistical restrictions to this method, it could >> >> provide a more modular design to opencog and also reduce unnecessary >> >> primary >> >> server strain. >> >> >> >> Noah B. >> >> >> >> >> >> On Fri, Sep 16, 2016, 12:25 Linas Vepstas <[email protected]> >> >> wrote: >> >>> >> >>> Hi Noah, >> >>> >> >>> Sounds like a good idea! We currently do not have any clear-cut >> >>> plans, >> >>> but let me tell you a little about what has been done so far. >> >>> Currently, >> >>> the main visual interface is in the repo >> >>> https://github.com/opencog/ros-behavior-scripting/ ... and its pretty >> >>> pathetic as vision goes. It does use OpenCV, but only as input into >> >>> a >> >>> hacked version of pi_vision, and that is used to detect human faces, >> >>> and map >> >>> them to 3D locations. Actually, I think that the pi_vision has been >> >>> replaced by the CMT tracker, recently, which seems to work a bit >> >>> better, >> >>> maybe. The ID's of the faces are placed as atoms into the atomspace. >> >>> Its >> >>> super-simple, and super-low-bandwidth: basically a handful of atoms >> >>> that say >> >>> "I can see face 42 now".... and that's it. The 3D locations of the >> >>> faces are >> >>> NOT kept in the atomspace -- they are kept off-line, mostly because of >> >>> bandwidth concerns. 30 frames a second of x,y,z points is not a lot, >> >>> but is >> >>> pointless, because we currently can't do reasoning with that info, >> >>> anyway. >> >>> >> >>> Re: new or moving objects: someone recently added support for "visual >> >>> saliency", and I flamed them a bit for how it was done: the >> >>> information >> >>> pumped into the atomspace was a very simple message: "something is >> >>> happening >> >>> in the visual field!" which is kind-of useless. Tell me, at least, is >> >>> it >> >>> big, or is it small, near or far, moving fast or moving slowly? Is it >> >>> "windmilling" i.e. moving-without-moving, like clapping hands? or >> >>> just >> >>> someone standing there, swaying side to side? >> >>> >> >>> With that kind of info, one can, at least, do some sort of scripted >> >>> reactions: the robot can say "Hey I think I see a fly" or "what's that >> >>> going >> >>> on behind your left shoulder?" Anyway, that general kind of input is >> >>> handled by https://github.com/opencog/ros-behavior-scripting/ .. >> >>> the >> >>> actual "state" of what is seen, what's going on is in >> >>> src/self-model.scm >> >>> and so additional stuff can be added there, like "I see something >> >>> small >> >>> moving"... scripted responses are in the file "behavior.scm", so if >> >>> something is seen, that is where you can script a response. >> >>> >> >>> All of the above is "short term". In the long term, it really has to >> >>> be >> >>> learning. For that, it has to be something completely different. This >> >>> email >> >>> is kind-of long already but ... the idea is to pattern-mine: "if 33% >> >>> of the >> >>> screen is red and X happened at the same time, this is important, >> >>> remember >> >>> and learn that!" Except this never happens. So instead, lets >> >>> (randomly) >> >>> try "if 33% of the screen is blue and X happened at the same time..." >> >>> well, >> >>> hey, that DOES happen, it means you went outside on a sunny day. So >> >>> this >> >>> should be remembered and recorded as an important filter-event, that >> >>> converts visual stuff into knowledge. The tricky part here is that >> >>> this is >> >>> ... CPU intensive, requires lots of training. Its a much much harder >> >>> problem. But.. enough. >> >>> >> >>> Anyway, the upshot is: "there are no rules" -- we've done very little >> >>> almost nothing with vision, so you can do anything you want. >> >>> >> >>> Re: python for opencog -- your best bet is to just poke atoms into the >> >>> atomspace with netcat, for example, like here: >> >>> >> >>> >> >>> https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/face_atomic.py#L82-L87 >> >>> called from here: >> >>> >> >>> >> >>> https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/face_atomic.py#L62-L66 >> >>> >> >>> and uses netcat here: >> >>> >> >>> >> >>> https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/netcat.py >> >>> >> >>> Currently, this is probably the best way to use python to get data >> >>> into >> >>> and out of the atomspace. >> >>> >> >>> --linas >> >>> >> >>> >> >>> On Fri, Sep 16, 2016 at 10:37 AM, Noah Bliss <[email protected]> >> >>> wrote: >> >>>> >> >>>> I'm going to be showing a great deal of ignorance in this post, but >> >>>> who >> >>>> knows, it might help. >> >>>> >> >>>> I understand an issue recently discussed with embodiment concerns >> >>>> methods for processing visual input. It's well known that at this >> >>>> time >> >>>> sending raw video into atomspace is a bad idea and that humans have >> >>>> built in >> >>>> visual processors that assist our conscious minds in understanding >> >>>> what our >> >>>> eyes see. (Obvious simple example being that the image is >> >>>> preflipped). >> >>>> >> >>>> I understand opencog has (in some form) a python api which leads me >> >>>> to >> >>>> think using the visual processing engine OpenCV may not be a bad >> >>>> idea. It >> >>>> has a fantastic python api, allows for exporting specific data from >> >>>> raw >> >>>> video such as "33% of the screen is red", or there are 2 lines in >> >>>> the field >> >>>> of view." it also has a PHENOMINAL foreground/background separation >> >>>> engine >> >>>> that allows only a processing of new or moving objects in the field >> >>>> of view. >> >>>> >> >>>> While a more mature opencog engine may prefer a more "raw" processor, >> >>>> I >> >>>> see OpenCV as a great place to start for getting useful information >> >>>> into >> >>>> atomspace quickly. >> >>>> >> >>>> I have yet to start work on this, heck, I have yet to fully learn the >> >>>> ropes of the current opencog system, but I wanted to at least drop >> >>>> the info >> >>>> here in case anyone else had comments or wanted to get a head-start >> >>>> on me. >> >>>> >> >>>> Best regards my friends. >> >>>> Noah B. >> >>>> >> >>>> PS: My personal experience with OpenCV was specifically dealing with >> >>>> automated turrets. There are great YouTube examples of using OpenCV >> >>>> for >> >>>> face-tracking webcams attached to servos, and blob isolating security >> >>>> cameras if you wanted specific examples to look up. >> >>>> >> >>>> >> >>>> -- >> >>>> You received this message because you are subscribed to the Google >> >>>> Groups "opencog" group. >> >>>> To unsubscribe from this group and stop receiving emails from it, >> >>>> send >> >>>> an email to [email protected]. >> >>>> >> >>>> >> >>>> To post to this group, send email to [email protected]. >> >>>> Visit this group at https://groups.google.com/group/opencog. >> >>>> To view this discussion on the web visit >> >>>> >> >>>> https://groups.google.com/d/msgid/opencog/1baaeade-567a-4456-aaa3-85e2b003fc7b%40googlegroups.com. >> >>>> For more options, visit https://groups.google.com/d/optout. >> >>> >> >>> -- >> >>> You received this message because you are subscribed to a topic in the >> >>> Google Groups "opencog" group. >> >>> To unsubscribe from this topic, visit >> >>> https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe. >> >>> To unsubscribe from this group and all its topics, send an email to >> >>> [email protected]. >> >>> To post to this group, send email to [email protected]. >> >>> Visit this group at https://groups.google.com/group/opencog. >> >>> To view this discussion on the web visit >> >>> >> >>> https://groups.google.com/d/msgid/opencog/CAHrUA37v2zxE7nTbqrBtw65k539v_wW1JLX2%3D2jgC3bkDoyqsw%40mail.gmail.com. >> >>> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "opencog" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> >> an >> >> email to [email protected]. >> >> To post to this group, send email to [email protected]. >> >> Visit this group at https://groups.google.com/group/opencog. >> >> To view this discussion on the web visit >> >> >> >> https://groups.google.com/d/msgid/opencog/CABpkOB-4HYkmtoqnBNpWaqdRKwou-w9CPevYOtNDYxGiJL9N%3Dg%40mail.gmail.com. >> >> >> >> For more options, visit https://groups.google.com/d/optout. >> > >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "opencog" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to [email protected]. >> > To post to this group, send email to [email protected]. >> > Visit this group at https://groups.google.com/group/opencog. >> > To view this discussion on the web visit >> > >> > https://groups.google.com/d/msgid/opencog/CAHrUA34J1i2qe-KTOUEZ%2B8gXXWhW1jmUoDWQt2H%3DTY7copfXRw%40mail.gmail.com. >> > >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Ben Goertzel, PhD >> http://goertzel.org >> >> Super-benevolent super-intelligence is the thought the Global Brain is >> currently struggling to form... >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "opencog" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/opencog. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/opencog/CACYTDBf1qsF21PyHU9V7t_nRPNtyiqn5FMjOtPeyFFrqMBzNhg%40mail.gmail.com. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "opencog" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/opencog. > To view this discussion on the web visit > https://groups.google.com/d/msgid/opencog/CABpkOB_mL24sT40JvcL%3DkE6BnqHHMpWGURJdWoWmVJd-p%3D2G0A%40mail.gmail.com. > > For more options, visit https://groups.google.com/d/optout. -- Ben Goertzel, PhD http://goertzel.org Super-benevolent super-intelligence is the thought the Global Brain is currently struggling to form... -- You received this message because you are subscribed to the Google Groups "opencog" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/opencog. To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CACYTDBdJ8egnAdBYqzEjpKhfW78pU_sxBiii%3D19qT4%3D-19yLmA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
