SDRs for Images

[Andrew Dillon]

Hey all,

I'm working through the NuPIC code (mostly htm.java since I understand 
Java best, but also the Python version when needed) and trying to 
understand how the different bits and pieces go about implementing HTM 
theory. I'm working on encoders right now and am having a bit of trouble 
understanding how images are encoded into an SDR.

My current understanding is that HTMs require input to be in the form of 
an SDR. These SDRs must contain a large number of bits, but the majority 
of them have to be off for any given encoding (e.g. n = 2048, w = 28).

I'm having trouble understanding how the *imageToVector() *function in 
the image_encoders.py file (in nupic.vision repo) goes about creating an 
SDR for an image. I think that what is happening is the image is 
converted into grayscale and a threshold is set (e.g. 127). All pixels 
whose value is below that threshold value are set to 0, with the rest 
being set to 1. The pixels are then thrown into an integer array 
starting at the top left pixel proceeding left to right, top to bottom. 
The resulting array is the SDR of that image?

This doesn't seem to be the way the brain works with visual data. We see 
all shades of gray, in addition to (although limited by the tiny visual 
spectrum) a rich range of other colors. Whereas an HTM that processes 
images in this manner would only see black and white. It seems to me 
that a massive amount of information is completely lost when an image is 
encoded this way. Is this just a temporary way of doing things until it 
is better understood how images are encoded in SDRs?

In addition, setting a threshold like this and throwing all the pixels 
into two different categories doesn't necessarily guarantee a sparse 
number of on bits, does it? Isn't it entirely reasonable to expect an 
image to have a lot more bits above the threshold than below? This would 
cause more 1s than 0s in the resulting SDR and the SDR would no longer 
be sparse, would it?

I've searched the mailing lists and haven't been able to come up with 
anything addressing my particular questions. Although, it wasn't very 
easy to search them and I may have easily overlooked things (I used this 
site <http://markmail.org/> to search the list, is there another way?). 
Any explanation or help that can be offered is greatly appreciated.

Thanks,
Andrew