The wireheading problem has been with us for as long as opiates have been with us. Yet, somehow, the species didn't all go wirehead and die off.
There is really only one fundamental problem that makes wireheading an existential risk for humanity: Mutually consenting parents are not permitted to exclude from their "villages" those they perceive to be inimical to raising their children to reproductive viability. The problem is with humans who are supremacist theocrats about "inclusion". Not AI giving humans what they want. On Thu, May 4, 2023 at 3:08 PM Matt Mahoney <[email protected]> wrote: > On Thu, May 4, 2023 at 1:30 PM Mike Archbold <[email protected]> wrote: > > > > Finally, AI seems to have caught on, it's white hot! And such pessimism! > > I'm not as pessimistic as Eliezer Yudkowsky. Maybe you read his > warning in Time magazine. He is certain that we are doomed unless we > shut down AI now. He is calling for international treaties with > decreasing limits on GPU power, and calling for air strikes on > noncompliant data centers. > https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ > > I understand his reasoning. The future is uncertain. If there is a 1% > chance that AI will wipe out humanity and prevent 1 trillion future > humans, that is worse than killing most of the world's 8 billion > people now. AI is the solution to the $100 trillion per year problem > of having to pay people to do work that machines aren't smart enough > to do. So it is not going to be stopped. People want this. Even people > who are aware of the problem, like Sam Altman, believe the best way to > slow down AI is to get there first, win the race, and then set the > pace. Nope. > > Yudkowsky has spent 20 years on the alignment problem and is no closer > to a solution, so he assigns a very high probability of doom relative > to everyone else who has spent less time on it. People have proposed > to use intermediate level AI to communicate with ASI, kind of like > fleas trying to control humans by communicating through dogs. I don't > buy it, and neither does Goeffrey Hinton, who quit Google so he could > speak freely about the dangers of AI. He pointed out on CNN that there > are no cases of any species controlling a more intelligent species. > > Let's enumerate the ways that AI might kill us, which I rank from > least to most likely below. > > 1. Self improving AI goes FOOM! When humans create smarter than human > AI, then so can it, but faster. This leads to a singularity. The AI > might or might not have goals that align with human goals, but in > general we can't predict, and therefore control, agents that are more > intelligent than us. > > 2. Self replicating nanotechnology. Yudkowsky mentions bacteria sized > diamondoid robots extracting CHON from the air and solar power, which > is already more efficient than chlorophyll. These would have higher > reproductive fitness than plants, thus killing the food chain and all > DNA based life. They could be designed to infect people and kill > everyone on a timer. > > 3. AI controlled biology. Anyone with a cheap 3-D nanoscale printer > could engineer viruses as deadly contagious as the common cold and as > deadly as rabies. > > 4. We solve alignment, and AI kills us by giving us everything we want. > > First, I don't believe in singularities. There is no exact threshold > where machines exceed human intelligence. You can't compare them > because intelligence is a broad set of capabilities that machines are > achieving one by one. The Turing test this year was a big step, but it > really started in the 1950's when computers started beating humans in > arithmetic. Self improvement started centuries ago, when companies > built machines faster and stronger than humans and reinvested their > profits to grow exponentially. But we still have a long way to go in > robotics. The human body carries 10^23 bits in our DNA and processes > 10^16 transcription operations per second at 10^-14 J per operation. > We can't get there with transistors either. Clock speeds stalled > around 2-3 GHz in 2010 and are now at the size limit and still using > 10^5 too much power. It will take some time to switch to computation > by moving atoms instead of electrons. > > Second, self replicating nanotechnology is 60 years away at the > current rate of Moore's law doubling global computing capacity every > 1.5 years. The biosphere has 10^44 carbon atoms encoding 10^37 bits of > DNA, with 10^29 DNA copy and 10^31 RNA and protein transcription > operations per second, using 500 TW of solar power out of 90,000 TW > available at the Earth's surface. > > Third, self replicating agents go back to the 1987 Morris worm. But > they are really hard to test. It might be easy to design a pathogen > with a 99% fatality rate but not 100%. They would have to release > several, which is harder. We also cannot assume that hobbyists are > rational and don't want to infect or kill themselves, or understand > the risks. And even if they do, they will make mistakes, because you > only have one chance to get it right. > > Fourth was the scenario I described in my last post. Yudkowsky's early > attempt at human goal alignment was Coherent Extrapolated Volition, > what we would want for our future selves if we were smarter. That > won't work because AI is worth $1 quadrillion and will be funded by > people buying what they want right now. What we want is neural > activity in our nucleus accumbens. That is not a long term strategy > for survival. > > -- Matt Mahoney, [email protected] ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T7c0c95d86a178556-Me666bdecd0e308ec7a7be642 Delivery options: https://agi.topicbox.com/groups/agi/subscription
