Noah,

OpenCV, as you know, is basically a toolkit, not an integrated system....

Right now, indeed, the vision processing we have hooked up to OpenCog
is restricted to

-- face detection, face tracking, face recognition
-- visual saliency identification
-- luminance-change detection

which is all pretty simple...

We have previously experimented with using kinfu to make a 3D map of a
robot's surroundings.... OpenCog's TimeSpaceMap is well-suited to
represent the output of kinfu (or similar) in a way that's integrated
with the Atomspace...

We have also played a bit with Region-CNNs, as a way of identifying
what objects are where in a visual scene (initially from a closed
class of objects)

So if I were going to integrate additional external vision tools with
OpenCog, I'd probably start with kinfu-or-similar, plus
(Region-CNN-with-trained-models)-or-similar...

Medium term, it's more interesting to integrate deep NN vision into
OpenCog, which Yenat is working on in our Addis office, but that's a
research project, whereas feeding output of kinfu and Region-CNN into
OpenCog is "just" complex software integration and training/tuning,
not really original research...

Anyway I am curious what specific  visual functions you are thinking of adding?

-- Ben G



On Sat, Sep 17, 2016 at 9:54 AM, Linas Vepstas <linasveps...@gmail.com> wrote:
>
>
> On Fri, Sep 16, 2016 at 8:41 PM, Noah Bliss <l3viat...@gmail.com> wrote:
>>
>> Thank you for the info Linas,
>>
>> I'll look at the current code and see if I can get a more complete
>> implementation of OpenCV started. You mentioned another dev's overly simple
>> integration which, while better than nothing, hardly fulfills our goal or
>> utilizes the full potential of OpenCV.
>>
>> With luck maybe I can get the visual end of opencog a bit more useful than
>> a glorified motion detector. :P
>
>
> I think the "saliency detector" code is buried somewhere in here:
> https://github.com/hansonrobotics/HEAD -- building and running that is
> probably the easiest way to get a working end-to-end demo.
>
>> Thanks again! I'll report back any major advances, otherwise check the
>> pull requests and maybe my branch of you get curious.
>>
>> As a side, if I am not mistaken, atomspace does most of its storage in sql
>> right?
>
> Only if you actually turn that on. Otherwise everything is in RAM.
>
>>
>> Perhaps I could see about offloading visual processing to a dedicated
>> machine along with whatever camera/sensor is being used, and get that set up
>> with an "atomspace client" that could dump pre-formatted atoms straight into
>> the db.
>
> netcat does that.  The python snippet with netcat was an example.
>
> For everything else, we use ROS. There's a bit of a learning curve for ROS,
> but its the ideal way for running multi-machine, distributed processing.
>
> --linas
>>
>> If there aren't any logistical restrictions to this method, it could
>> provide a more modular design to opencog and also reduce unnecessary primary
>> server strain.
>>
>> Noah B.
>>
>>
>> On Fri, Sep 16, 2016, 12:25 Linas Vepstas <linasveps...@gmail.com> wrote:
>>>
>>> Hi Noah,
>>>
>>> Sounds like a good idea!  We currently do not have any clear-cut plans,
>>> but let me tell you a little about what has been done so far.   Currently,
>>> the main visual interface is in the repo
>>> https://github.com/opencog/ros-behavior-scripting/ ... and its pretty
>>> pathetic as vision goes.   It does use OpenCV, but only as input into a
>>> hacked version of pi_vision, and that is used to detect human faces, and map
>>> them to 3D locations.  Actually, I think that the pi_vision has been
>>> replaced by the CMT tracker, recently, which seems to work a bit better,
>>> maybe.  The ID's of the faces are placed as atoms into the atomspace.  Its
>>> super-simple, and super-low-bandwidth: basically a handful of atoms that say
>>> "I can see face 42 now".... and that's it. The 3D locations of the faces are
>>> NOT kept in the atomspace -- they are kept off-line, mostly because of
>>> bandwidth concerns.  30 frames a second of x,y,z points is not a lot, but is
>>> pointless, because we currently can't do reasoning with that info, anyway.
>>>
>>> Re: new or moving objects: someone recently added support for "visual
>>> saliency", and I flamed them a bit for how it was done: the information
>>> pumped into the atomspace was a very simple message: "something is happening
>>> in the visual field!" which is kind-of useless.  Tell me, at least, is it
>>> big, or is it small, near or far, moving fast or moving slowly?  Is it
>>> "windmilling" i.e. moving-without-moving, like clapping hands?  or just
>>> someone standing there, swaying side to side?
>>>
>>> With that kind of info, one can, at least, do some sort of scripted
>>> reactions: the robot can say "Hey I think I see a fly" or "what's that going
>>> on behind your left shoulder?"  Anyway, that general kind of input is
>>> handled by    https://github.com/opencog/ros-behavior-scripting/ .. the
>>> actual "state" of what is seen, what's going on is in src/self-model.scm
>>> and so additional stuff can be added there, like "I see something small
>>> moving"...  scripted responses are in the file "behavior.scm", so if
>>> something is seen, that is where you can script a response.
>>>
>>> All of the above is "short term". In the long term, it really has to be
>>> learning.  For that, it has to be something completely different. This email
>>> is kind-of long already but ... the idea is to pattern-mine: "if 33% of the
>>> screen is red and X happened at the same time, this is important, remember
>>> and learn that!"  Except this never happens.  So instead, lets (randomly)
>>> try "if 33% of the screen is blue and X happened at the same time..." well,
>>> hey, that DOES happen, it means you went outside on a sunny day. So this
>>> should be remembered and recorded as an important filter-event, that
>>> converts visual stuff into knowledge.  The tricky part here is that this is
>>> ... CPU intensive, requires lots of training. Its a much much harder
>>> problem.  But.. enough.
>>>
>>> Anyway, the upshot is: "there are no rules" -- we've done very little
>>> almost nothing with vision, so you can do anything you want.
>>>
>>> Re: python for opencog -- your best bet is to just poke atoms into the
>>> atomspace with netcat, for example, like here:
>>>
>>> https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/face_atomic.py#L82-L87
>>> called from here:
>>>
>>> https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/face_atomic.py#L62-L66
>>>
>>> and uses netcat here:
>>>
>>> https://github.com/opencog/ros-behavior-scripting/blob/master/face_track/netcat.py
>>>
>>> Currently, this is probably the best way to use python to get data into
>>> and out of the atomspace.
>>>
>>> --linas
>>>
>>>
>>> On Fri, Sep 16, 2016 at 10:37 AM, Noah Bliss <l3viat...@gmail.com> wrote:
>>>>
>>>> I'm going to be showing a great deal of ignorance in this post, but who
>>>> knows, it might help.
>>>>
>>>> I understand an issue recently discussed with embodiment concerns
>>>> methods for processing visual input. It's well known that at this time
>>>> sending raw video into atomspace is a bad idea and that humans have built 
>>>> in
>>>> visual processors that assist our conscious minds in understanding what our
>>>> eyes see. (Obvious simple example being that the image is preflipped).
>>>>
>>>> I understand opencog has (in some form) a python api which leads me to
>>>> think using the visual processing engine OpenCV may not be a bad idea. It
>>>> has a fantastic python api, allows for exporting specific data from raw
>>>> video such as "33% of the screen is red", or  there are 2 lines in the 
>>>> field
>>>> of view." it also has a PHENOMINAL foreground/background separation engine
>>>> that allows only a processing of new or moving objects in the field of 
>>>> view.
>>>>
>>>> While a more mature opencog engine may prefer a more "raw" processor, I
>>>> see OpenCV as a great place to start for getting useful information into
>>>> atomspace quickly.
>>>>
>>>> I have yet to start work on this, heck, I have yet to fully learn the
>>>> ropes of the current opencog system, but I wanted to at least drop the info
>>>> here in case anyone else had comments or wanted to get a head-start on me.
>>>>
>>>> Best regards my friends.
>>>> Noah B.
>>>>
>>>> PS: My personal experience with OpenCV was specifically dealing with
>>>> automated turrets. There are great YouTube examples of using OpenCV for
>>>> face-tracking webcams attached to servos, and blob isolating security
>>>> cameras if you wanted specific examples to look up.
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "opencog" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to opencog+unsubscr...@googlegroups.com.
>>>>
>>>>
>>>> To post to this group, send email to opencog@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/opencog.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/opencog/1baaeade-567a-4456-aaa3-85e2b003fc7b%40googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "opencog" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> opencog+unsubscr...@googlegroups.com.
>>> To post to this group, send email to opencog@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/opencog.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/opencog/CAHrUA37v2zxE7nTbqrBtw65k539v_wW1JLX2%3D2jgC3bkDoyqsw%40mail.gmail.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "opencog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to opencog+unsubscr...@googlegroups.com.
>> To post to this group, send email to opencog@googlegroups.com.
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/CABpkOB-4HYkmtoqnBNpWaqdRKwou-w9CPevYOtNDYxGiJL9N%3Dg%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opencog+unsubscr...@googlegroups.com.
> To post to this group, send email to opencog@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opencog/CAHrUA34J1i2qe-KTOUEZ%2B8gXXWhW1jmUoDWQt2H%3DTY7copfXRw%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.



-- 
Ben Goertzel, PhD
http://goertzel.org

Super-benevolent super-intelligence is the thought the Global Brain is
currently struggling to form...

-- 
You received this message because you are subscribed to the Google Groups 
"opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to opencog+unsubscr...@googlegroups.com.
To post to this group, send email to opencog@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/opencog/CACYTDBf1qsF21PyHU9V7t_nRPNtyiqn5FMjOtPeyFFrqMBzNhg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to