You don't have to make Jim Bowey and me co-authors of your paper just because you quote our work. If you do, I would rewrite the abstract to something more concise. "The sword is mightier than the pen". Sometimes that's true. Does that make ASI any safer?
My disagreement with MIRI is that the threat from AI is not about alignment. Automating human labor requires modeling 8 billion human brains, which we solve with a system of pervasive surveillance that is largely already built. The model knows what we want. We just can't verify that it got it right because we don't know what our own goals are as well as it does. We just have to trust that it does. That's the best we can do. Could these goals drift through cycles of recursive self improvement as MIRI claims? Does the question even make sense? To answer that, I tried to define precisely what it means for an agent to have a goal. https://mattmahoney.net/rsi.pdf Goals are a useful shortcut for describing human behavior but not for computation in general. Does a thermostat "want" to maintain a fixed temperature? Would a smarter thermostat build backup generators to help achieve its goal? Would it hack computers to make itself smarter? The actual threat of AI is that if you got everything you wanted, then you would be dead. It's our own brains that are misaligned. -- Matt Mahoney, [email protected] On Fri, Jun 19, 2026, 3:50 PM twenkid <[email protected]> wrote: > Title: The ban of Fable 5 and Mythos by US government is an illustration > of the paper "Power Overrides Intelligence" > <https://www.researchgate.net/publication/407308848_Power_Overrides_Intelligence_Answers_to_Matt_Mahoney's_summary_of_LessWrong's_The_Problem_regarding_the_so_called_existential_risks_of_Artificial_General_Intelligence_and_Superintelligence_in_August_2> > > https://www.researchgate.net/publication/407308848_Power_Overrides_Intelligence_Answers_to_Matt_Mahoney's_summary_of_LessWrong's_The_Problem_regarding_the_so_called_existential_risks_of_Artificial_General_Intelligence_and_Superintelligence_in_August_2 > > The paper archives a discussion on "existential risks" here, with a few > intro and outro notes. > > The title was my short summary addressing the concentration point of the > actual "risks", which I've discussed in the early 2000s as a child, but the > uncles and grandpas "AI-experts" keep blaming "the machines" who will "kill > us all", LOL. > (...) > *Intelligence *itself as cognition and cognitive capabilities is not the > "risk", except if it is paired with the *intelligence *services* *- > because they have *causal POWER* to enforce whatever they wish to the > others without correspondent counteraction; the inequality and > concentration of power is a major "risk" for many possible "catastrophes". > (...) > > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Teedaf4a19e56877e-M7f292d40953f8c6cf4239ca4> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Teedaf4a19e56877e-Mf8fcf7b885b414f2168239ef Delivery options: https://agi.topicbox.com/groups/agi/subscription
