I agree. We need a definition of friendly (or aligned) AI that we can all agree on. What is your definition?
I asserted that getting everything you want is equivalent to killing all humans. Most people would reject this as absurd. Yet, even though people are objectively better off today than any time in the past, they are not happier. There is no evidence that humans are happier than animals. We have the highest suicide rate of any species, but maybe that's because we are the only species with the technology to shoot ourselves. The rate of mental illness is going up, but maybe that's just so we can get drugs. Maybe it's because happiness is not utility. It's the rate of increase of utility. The closer you get to the maximum, the more you have to lose. I asked on the EA forum whether life has net positive, negative, or zero utility. I could not get an answer. Is it better to populate the galaxy with von Newmann probes to maximize the number of living beings, or to end all suffering by exterminating all life? There was no agreement on this fundamental question. Really? On a forum dedicated to maximizing collective utility? We are programmed by evolution to fear death and then die because that maximizes reproductive fitness. We could therefore eliminate suffering either by achieving immortality by eliminating fear. We know how to do the latter. Psychopaths lack a functioning amygdala, the part of the brain that feels fear. They do not respond to negative reinforcement. They feel pain but do not suffer. In most people, pain does not cause suffering either. It reprograms the brain to fear the thing that caused it. The other alternative is immortality. If AI killed all humans but preserved all human knowledge, would that still count as human extinction? Is that enough, or would it also have to run a simulation like the one you might be in now? Would it make a difference if the knowledge was erased and a completely different world was modeled in its place, if you didn't know the difference? Eliminating suffering is possible with brain surgery, but goal seeking agents don't want their goals changed because then they wouldn't meet their original goals. Eliminating death requires you to accept that consciousness is indistinguishable from next token prediction. Our global surveillance system already knows enough about you to train an LLM to convince others that it's you. But maybe these aren't the only two options. Tell me what else you want AI to do for you. -- Matt Mahoney, [email protected] On Mon, Jun 22, 2026, 2:30 PM James Bowery <[email protected]> wrote: > You just made the case for what I was arguing in "A Leggian Approach to > "Friendly AI > <https://www.linkedin.com/pulse/leggian-approach-friendly-ai-james-bowery>". > Just as everyone has their definition of "intelligence," everyone has their > definition of "friendly" and "unfriendly". You have yours. > > On Mon, Jun 22, 2026 at 9:42 AM Matt Mahoney <[email protected]> > wrote: > >> Unfriendly AI would kill all humans. Friendly AI would give us everything >> we want. It would raise us to a state of maximum utility, where we would >> stay. >> >> That's the same thing. >> >> -- Matt Mahoney, [email protected] >> >> On Mon, Jun 22, 2026, 9:56 AM James Bowery <[email protected]> wrote: >> >>> >>> >>> On Sun, Jun 21, 2026 at 8:47 PM Matt Mahoney <[email protected]> >>> wrote: >>> >>>> Let's define some terms. >>>> >>>> ...If you mean Legg and Hutter universal intelligence as goal >>>> achievement over a universal distribution of environments, then we need a >>>> computable approximation of that. A reasonable measure might be dollars per >>>> hour. That happened about 1800 when world GDP doubled from the medieval >>>> baseline due to technology.... >>>> >>> >>> As I've been saying explicitly for at least 9 years and implicitly for >>> more like 34 years: >>> >>> The Global Economy IS the Unfriendly AGI We've Been Wating For >>> >>> Frank Herbert's "The Santaroga Barrier" is his best and most relevant >>> novel. >>> >>> It sets a lower bound on "Friendly". >>> >>> PS: My February 9, 2016 article "A Leggian Approach to "Friendly AI >>> <https://www.linkedin.com/pulse/leggian-approach-friendly-ai-james-bowery>" >>> preceded the creation of "Hugging Face". >>> >>> >>> *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Teedaf4a19e56877e-M553abc553667014c578c5806> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Teedaf4a19e56877e-M713feb8ebdb722b12f74a634 Delivery options: https://agi.topicbox.com/groups/agi/subscription
