On 10/27/2011 8:33 AM, Eugen Leitl wrote:
On Wed, Oct 26, 2011 at 12:53:24AM -0700, BGB wrote:
from what I read, IBM was using a digital crossbar.
It sounds like Kwabena Boahen (Carver Mead's school) is
on the right track
http://web.cecs.pdx.edu/~strom/onr_workshop/boahen.pdf
the group seems to be still publishing
http://www.stanford.edu/group/brainsinsilicon/pubs.html
yeah.
sadly I can't look at the PDF at the moment partly because of lack of
time and partly because Firefox+Acrobat is giving me crap (not working...).
I suspect something more "generic" would be needed.
I don't see how generic will do long-term any than for bootstrap
(above co-evolution) reasons.
more generic is more likely to be able to do something interesting.
More generic hardware (not optimized for a particular model)
also means it's less efficient. On the other hand, we don't
have a particular model to optimize for, so right now generic
is the way to go.
yeah.
I would rather see something which can do something "interesting",
rather than something which implements a simplistic model in HW, and
mostly promotes itself based on number of neurons and the number of
watts used.
one can then optimize once a basic working model is in place (with
demonstrable capabilities).
ARM or similar could work (as could a large number of 386-like cores).
I had considered something like GPGPU, but depending on the type of
neural-net, there could be issues with mapping it efficiently to
existing GPUs.
You could wire it up in a 3d grid, with offsets to mostly
local other grid sites and stream through memory. Refresh
rate could be 10 Hz or more, depending on how complex the
model, and how much random-access like memory accesses
you're producing.
a bit of a question though is how to handle the neural impulses. some
strategies would work a bit better on a GPU than others.
sadly, in some ways, a GPU is a bit more limited than a large pile of
ARM or 386 cores would be.
one such issue is that GPUs tend to execute threads in groups of 8 or 16
with the assumption that all have exactly the same control flow. any
divergent branching effectively "forks" the threads, resulting in a
(potentially significant) loss of efficiency.
this may likely result in the" most efficient" case being to always
calculate all weights for all neurons, regardless of whether or not they
have actually fired.
a model which allows fully independent threads, however, could likely
skip a lot of this (only updating neurons which fire or receive
impulses), and even if there are less total threads or execution is
slower, the average-case performance may still be better.
also, the strong-areas for GPUs are not necessarily the same as the
requirements for implementing neural nets. again, it could work, but it
If you can do plenty of parallel ops on short integers (8-16 bit)
then it seems to match well, provided you can feed the shaders.
Mapping things to memory is tricky, as otherwise you'll starve
the shaders and thus underutilize the hardware.
yes, however, it is worth noting that GPUs dedicate a lot of silicon to
lots of floating-point and vector calculations.
although the GPU can work, it is possible that a more specialized chip
(maybe lacking in floating-point or SIMD capabilities), could actually
be able to dedicate much more silicon to having processor cores, and
thus deliver higher performance.
granted, to work correctly there are a few real-valued calculations
which would need to be performed, either meaning:
special instructions (adding complexity and eating silicon, meaning less
cores);
LUTs, potentially eating up memory bandwidth (reducing overall
throughput, although cache can improve speed per-core and require less
bandwidth, it also costs a lot of silicon);
calculating in-band via fixed-point, which could be slower (but is
possibly still the best option in terms of maximizing number of cores
and raw processing power);
...
an example architecture would be to have a large number of simple
386-like cores (with some APIC-like features), probably removing a lot
of unnecessary features (segmentation, local MMUs, ...), and probably
without any local cache. each may have their own instruction
dispatchers, ALU, ... though.
potentially, a number of cores could then share a number of L1 caches,
with a shared L2 cache, a shared MMU, and is then connected to external
memory.
potentially, there would be a "special" core, with a more complete ISA,
and would serve more of an organizational role (and have special
abilities, such as being able to control the MMU, ...).
however, lacking fancy HW, falling back to the GPU should still be
fairly sane, as it still has a fair amount of performance.
is just less certain it is "ideal".
the second part of the question is:
assuming one can transition to a purely biology-like model, is this a
good tradeoff?...
if one gets rid of a few of the limitations of computers but gains some
of the limitations of biology, this may not be an ideal solution.
Biology had never had the issue to deal with high-performance numerics,
I'm sure if it had it wouldn't do too shabbily. You can always go hybrid
e.g. if you want to do proofs or cryptography.
biology also doesn't have software distribution, ability to make
backups, ...
Even hybrid analog systems can be converted into a large blob of
binary data and loaded back. The loss of precision due to digitization
is negligible, as the system will be noisy/low precision (<6 bit) to
start with.
I am not making an issue over analogue vs digital.
rather, the issue is whether or not one can save out the state of the
neural-net, and load in a new neural net, on demand...
if the neural-net is mapped to memory or similar, this can be done.
however, if the neural net is embedded into the HW and lacks direct
external access (say, ability to read/write the net), then one has a
problem.
likely, a system based purely on a neural net will not really be able to
save a copy of itself to external storage. instead, one will likely need
to retain aspects of traditional software and computation so as to be
able to do things like load and store neural-net images to things like
hard-drives or SSDs.
it is very possible that a primarily neural-net based "silicon brain"
would still retain a basic form of both an OS and traditional software,
which would likely run essentially "underneath" the neural-net image.
for AI systems, it is likely that it would be more of a mix, with parts
being based on neural-nets, and other parts as good-old executable code.
ideally, a silicon neural-net strategy would also readily allow the
installation of new software and creation of backups.
You can always digitize and serialize your data, put it through
a pipe and reinstantialize it in equivalent piece of hardware
on the other end. Consider crystalline computation (Margolus/Toffoli)
http://people.csail.mit.edu/nhm/cc.pdf which maps very well to
3d lattice of sites in future's molecular hardware. If you can
halt the execution, or clone a static shadow copy of the dynamic
process you can serialize and extract from the faces of the crystal
at leisure.
all of this works assuming there is access to the state.
this is thus the main requirement.
the most likely strategy here IMO is to leverage what existing OS's can
already do, essentially treating any neural-net processors as another
piece of hardware as far as the OS is concerned.
I'm not sure you need an OS. Many aspects of operation can be mapped
to hardware. Consider a 10 GBit/s short reach fiber, the length
of fiber through these keystrokes are passing is an optical FIFO
conveniently containing a standard MTU (1500 Bytes) frame/packet.
Same thing does vacuum to a sufficiently fast line of sight
laser link. With the right header layout, you can see how you
could make a purely photonic cut-through router or switch, with
zero software. Same thing with operating Toffoli's hypothetical
computronium crystal, you have stop and go, read and write, and
that's it. I/O could be mapped directly to crystal faces.
yes, but what is cheaper?...
in many cases, even many parts of "hardware" functionality is internally
implemented (in computers) via smaller specialized processors (such as
ARM cores) and program logic built into the chips via internal ROM.
if the OS is primarily managerial, then its raw performance may not
matter so much, and manifesting it in raw HW would be too expensive.
another issue is there may be many tasks which simply map more
efficiently to traditional computation than to the neural nets, and thus
leaving them as traditional software would make more sense (from a
performance perspective).
a simple example is doing arithmetic:
even a slow CPU can vastly outperform a human at performing arithmetic
calculations.
humans, being essentially a fairly high-end example of biological neural
nets, fail miserably at something computers take for granted.
so, the highest overall performance may be gained from combining the
powers of neural nets with those of CPUs and software.
this would probably mean using neural-net processors along-side
traditional CPU cores (rather than in place of them).
better would be to try for a strategy where the merits of both can be
gained, and as many limitations as possible can be avoided.
most likely, this would be via a hybrid model.
Absolutely. Hybrid at many scales, down to analog computation for neurons.
yeah.
analog is an idea I had not personally considered.
I guess a mystery here is how effectively semiconductor logic (in
integrated circuits) can work with analog signals.
the main alternative is, of course, 8 or 16-bit digital signals.
8 or 16 bit ALUs are ridiculously complicated, power-hungry and slow if compared
with analog computation in transistors or memristor. Of course the
precision is low, particularly if you're working at nanoscale. A few atoms
displaced shift the parameters of the device visibly, but then, you're
engaging a redundant mode of computation anyway. The brain does not just
tolerate noise, some aspects of it are noise-driven.
possibly, but the issue is if the "fidelity" is sufficient for the
device to work correctly.
it is also notable that IBM is apparently not using analog. there may be
some good reasons here (despite my personal disagreement with them
hard-wiring the things...).
also, it is worth noting that an ALU is not needed per-neuron, rather it
is very likely that a large number neurons would be spread over a
smaller number of ALUs, primarily only performing arithmetic when they
fire or are signaled.
in this case, memristors could probably be used as non-volatile
RAM-style storage.
I had more imagined as hybrids of neural-nets and traditional software.
granted, there is always a certain risk that it could mean, in some
distant-future setting, people sitting around with Windows running in
their heads (say, because they have incorporated processors and silicon
Security issues alone would be a killer. Consider buffer overruns in
the optical FIFO I mentioned, how do you cause them? You can't, obviously,
since the laws of the unverse instead of limits of human design are
asserting correct operation.
granted, one can avoid buffer overflows by bounds-checking their array
accesses.
neural-nets into their otherwise wetware brains, that or parts, or much,
or all, of their brain functionality has been migrated into silicon).
Right spintronics in graphenes looks good. No other good candidates for
computronium, apart from self-assembled macromolecular crystals (like
viral capsids, only containing CA cells, designed to link up and connect,
unlike real virus capids, who don't really want to, unless coaxed by
crystallographers).
near term though, probably most everything will be good old silicon and
transistor logic.
it is not clear at this point how long until alternative technologies
really get going, and very possibly humans may start integrating with
machinery well before the effective lifespan of silicon has ended.
I was more imagining that humans will be migrating parts of their mind
and consciousness into procedural logic and giant memory-mapped buffers
(probably well into the 64-bit range by this point, as representing
parts of a neural net in-memory will likely eat up a fairly massive
chunk of memory...).
or such...
_______________________________________________
fonc mailing list
[email protected]
http://vpri.org/mailman/listinfo/fonc
