On Wednesday 16 April 2008 04:15:40 am, Steve Richfield wrote: > The problem with every such chip that I have seen is that I need many > separate parallel banks of memory per ALU. However, the products out there > only offer a single, and sometimes two banks. This might be fun to play > with, but wouldn't be of any practical use that I can see.
How much memory are you thinking of, total? The current best is 2 Gbits on a chip, and that's pushing the density side of the equation big-time. Divide by 2 (room for those processors and busses) x 10000 x 32 and you get 3k words per processor. You can't even put a 10k x 10k square matrix on the chip. So you're bottlenecked by the off-chip pipe. > > ... architecture of an overall SIMD paradigm. > > Done right, the programmer would never see it. Remember, I plan to implement > coordination points, where everything stops until all sub-processors are to > the coordination point, whereupon everything continues. The compiler would > just drop one of these in wherever needed to keep things straight. Can't argue with that! Right now, I think there's more upside on the smart compiler side of the equation than the hardware, but don't let that stop you. > Queueing theory says that you are best with a minimum number of the fastest > possible "servers" (processors) to serve a queue of work. I think that my > 10K proposal produces the fastest processors, and putting several on a wafer > provides several of them for the most horrendous possible applications. It > appears (to me) that such a wafer, if well designed, would provide the > compute power to start working on AGI in earnest. WSI (wafer-scale integration) has been tried for decades -- we looked at it inthe 80s. There are some complex reasons, having to do with defect density and the like, that they still, e.g., cut them into chips and then turn around and rewire 8 of those chips onto a DIMM. > However, large neural networks are inherently parallel things, so Amdahl's > first law shouldn't be a factor. NNs have two properties you may stumble over. They involve lots of multiplication; and they involve lots of datacomm. Consider two architectures: a single ALU with a fast multiplier (for 32-bit words, 1024 full-adder circuits) versus 32 ALUs that each have a 32-bit adder (again for a total of 1024 FAs) and do a mult by a 32-cycle shift-&-add. Both architectures can do 32 mults in 32 cycles. But: the serial can do 5 mults in 5 cycles, but the parallel still needs 32. The serial can do 33 mults in 33 cycles, but the parallel needs 64. The amount of hardware isn't really the same. The serial needs one instruction decoder and one memory addresser -- the parallel needs 32 of each. So on the same real estate you can bulk up the drivers and make the serial faster. And finally, the serial suffers no slowdown at all when I interleave a shuffle-exchange step (to do an FFT) -- the parallel bets bogged down in datacomm. There's an interesting variant on the parallel version that we worked on specifically for matrix mult or neural nets (same basic operation). The overflow of each of the adders fed into the bottom of an adder tree, which was one bit wide at the leaves, two bits at the next level up, etc, with a full-word accumulator at the top. So we could do fully pipelined dot products for as long as we had the data to crunch. Which was all very cute but went the way of the Connection Machine for much the same reason. (but we went faster, heh heh) > ... which automatically happens when the rows of A just happen to match the > interleaving. Compilers could over-dimension arrays to make this so. Note > the use of "Multiple Tag Mode" on antique IBM-709/7090 computers, for which > you had to do the same to make it useful. This helps if you're multiplying NxN matrices with only N processors, but does you no good if you actually have enough processors to have one element per processor! > My design all fits on a single chip - or it will never work. See query about memory size above. > Observation: I am a front-runner type, looking to find the roads that lead > to here I want to go. This in preference to actually packing up the luggage > and actually draging it down that road. You sound like the sort that once > the things is sort of roughed out, likes to polish it up and make it as good > as possible. Further, you have a LOT more actual experience doing this sort > of thing with whizzbang chips than I do, and you actually understood what I > was proposing with a minimum of explanation. > > Question: Do you have any interest in helping transform my rather rough > concept to a sufficiently detailed road map that anyone with money and an > interest in AGI would absolutely HAVE to fund it? I simply don't see Intel > or anyone else currently running in a direction that will EVER produce an > AGI-capable processor, yet my approach looks like it has a good chance if > only I can smooth out the rough edges and eventually find someone to pay the > bills - things that you obviously have a lot of experience doing. Any > interest? Sorry -- I'm a front-runner type myself. I write books about robot ethics and nanotechnology. When I was at Rutgers doing this I had people working for me to do the details. And I'm currently doing AI, which is a lot more interesting than parallel architectures, to my mind. In the 80s I felt the biggest stumbling block to AI was processing power (and I agree with myself in retrospect). But as previously mentioned, the stuff becoming available now is on the verge of solving that problem, and we can spend our time figuring out what to do with it. Which reminds me that I've spent a lot more time on this than I should, and so I will have to let the whole subject, interesting as it is, go at that and get back to work. Have fun with your project and good luck! Josh ------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244&id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
