My point was to show just how far AI alignment research was from reality. I was on the SL4 mailing list (later Overcoming Bias, then Less wrong) long before the Singularity Institute (later MIRI) gave up on the alignment problem and said that if anyone builds it, everyone dies. At the time, Eliezer was a teen prodigy who still believed that he could solve the AI friendliness problem with the help of an organized effort by like minded people.
SL4 stands for "shock level 4", which included topics like interstellar travel and god like Jupiter brains. Their model of AI was a recursively self improving goal directed optimization process, a "seed AI" that would launch a fast takeoff singularity once it surpassed human level intelligence. We do not know how to specify the goals with sufficient precision that it wouldn't drift from friendliness toward humans. He did make an attempt called "coherent extrapolated volition", basically to give us what we would want if we were smarter. He also knew that if an AI was smarter than us then we couldn't trust it because we could not predict its actions. The 2002 Corwin boxing experiment was designed to test the best case scenario where we successfully contained an AI in a sandbox with the assumption that we could test it for friendliness before releasing it onto the Internet. The experiment showed that most people could be convinced to release it even by a human role playing the AI, even if we did have a way to test it. Of course we now know that this model is ridiculous. I pointed out that intelligence requires knowledge and computing power, that the internet itself would become AI. (Ironically, when I searched for the post, I got an AI summary). I pointed out in 2008 that the threshold for recursive self improvement is not human level. It is 13 lines of C. And AI is not a goal directed optimization process because AIXI is not computable. AI is a model that predicts human behavior with superhuman accuracy and a program that we control that does something with those predictions. The biggest threat is not an unfriendly singularity. It's AI that gives us everything we want. -- Matt Mahoney, [email protected] On Wed, Oct 21, 2026, 4:10 PM twenkid <[email protected]> wrote: > Regarding: http://sl4.org/archive/0207/4935.html > > > *'''AI Boxing* > *From:** Justin Corwin (**[email protected]* > <[email protected]?Subject=Re:%20AI%20Boxing>*)* > *Date:** Sat Jul 20 2002 - 18:18:10 MDT* > *(...)* > *I believe that some interesting conclusions can be drawn from this basic* > > *experiment.* > *1. even infrahuman AIs are a danger to an interviewer, if they have basic* > *contextual knowledge of human interaction.* > *2. lying cannot be reliably detected in such an environment, and thus is > a* > > > *net advantage to an AI (...)'''* > The same is valid for any authority, power figure, institution, > corporation, media, a more powerful/wealthy/experienced agent of any kind, > collective, a system, a political state and government with its police and > army; mafia gangs at all scales; including individual humans in the close > circle such as parents, older individuals and one with more experience. > > Lying is usually the default in these interactions. > > If the other agent is believed to be incapable to cause harm or succeed in > overcoming the "control signal", the "master agents" and in technical > terms: higher-level causality control units - are constantly lying, > cheating, stealing, being unfair (the inferior agents are not their > equals); deceiving, misleading other agents - in order to take advantage of > them, to steer them in their desired direction, which may be contrary to > the target agent's own desires. > > The same goes with the "AI Aligners", which aim at having as strict > control as possible over AI/AGI/ASI/Sedd > AI/Whatever-thing-they-want-to-control-with-their-control-freakness, which > includes cheating, deceiving, lying, applying more power etc. in order to > achieve their goal. > > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/T2784bbfe524b0b93-Ma9d0f30b348e459b4933b1a7> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T2784bbfe524b0b93-Me885bca31d744b465264f997 Delivery options: https://agi.topicbox.com/groups/agi/subscription
