--- Begin Message ---
> In August, Anthropic reported that they disrupted a threat actor that
used Claude, Anthropic’s AI model, to automate the entire cyberattack
process. It was an impressive use of the AI, which performed network
reconnaissance, penetrated networks, and harvested victims’ credentials.
The AI was able to figure out which data to steal, how much money to
extort out of the victims, and how to best write extortion emails.
[ https://www.anthropic.com/news/detecting-countering-misuse-aug-2025 ]
Autonomous AI Hacking and the Future of Cybersecurity
Schneier on Security
10 October 2025
https://www.schneier.com/blog/archives/2025/10/autonomous-ai-hacking-and-the-future-of-cybersecurity.html
[ For embedded links, see the original source ]
AI agents are now hacking computers. They’re getting better at all
phases of cyberattacks, faster than most of us expected. They can chain
together different aspects of a cyber operation, and hack autonomously,
at computer speeds and scale. This is going to change everything.
Over the summer, hackers proved the concept, industry institutionalized
it, and criminals operationalized it. In June, AI company XBOW took the
top spot on HackerOne’s US leaderboard after submitting over 1,000 new
vulnerabilities in just a few months. In August, the seven teams
competing in DARPA’s AI Cyber Challenge collectively found 54 new
vulnerabilities in a target system, in four hours (of compute). Also in
August, Google announced that its Big Sleep AI found dozens of new
vulnerabilities in open-source projects.
It gets worse. In July Ukraine’s CERT discovered a piece of Russian
malware that used an LLM to automate the cyberattack process, generating
both system reconnaissance and data theft commands in real-time. In
August, Anthropic reported that they disrupted a threat actor that used
Claude, Anthropic’s AI model, to automate the entire cyberattack
process. It was an impressive use of the AI, which performed network
reconnaissance, penetrated networks, and harvested victims’ credentials.
The AI was able to figure out which data to steal, how much money to
extort out of the victims, and how to best write extortion emails.
Another hacker used Claude to create and market his own ransomware,
complete with “advanced evasion capabilities, encryption, and
anti-recovery mechanisms.” And in September, Checkpoint reported on
hackers using HexStrike-AI to create autonomous agents that can scan,
exploit, and persist inside target networks. Also in September, a
research team showed how they can quickly and easily reproduce hundreds
of vulnerabilities from public information. These tools are increasingly
free for anyone to use. Villager, a recently released AI pentesting tool
from Chinese company Cyberspike, uses the Deepseek model to completely
automate attack chains.
This is all well beyond AIs capabilities in 2016, at DARPA’s Cyber Grand
Challenge. The annual Chinese AI hacking challenge, Robot Hacking Games,
might be on this level, but little is known outside of China.
Tipping point on the horizon
AI agents now rival and sometimes surpass even elite human hackers in
sophistication. They automate operations at machine speed and global
scale. The scope of their capabilities allows these AI agents to
completely automate a criminal’s command to maximize profit, or
structure advanced attacks to a government’s precise specifications,
such as to avoid detection.
In this future, attack capabilities could accelerate beyond our
individual and collective capability to handle. We have long taken it
for granted that we have time to patch systems after vulnerabilities
become known, or that withholding vulnerability details prevents
attackers from exploiting them. This is no longer the case.
The cyberattack/cyberdefense balance has long skewed towards the
attackers; these developments threaten to tip the scales completely.
We’re potentially looking at a singularity event for cyber attackers.
Key parts of the attack chain are becoming automated and integrated:
persistence, obfuscation, command-and-control, and endpoint evasion.
Vulnerability research could potentially be carried out during
operations instead of months in advance.
The most skilled will likely retain an edge for now. But AI agents don’t
have to be better at a human task in order to be useful. They just have
to excel in one of four dimensions: speed, scale, scope, or
sophistication. But there is every indication that they will eventually
excel at all four. By reducing the skill, cost, and time required to
find and exploit flaws, AI can turn rare expertise into commodity
capabilities and gives average criminals an outsized advantage.
The AI-assisted evolution of cyberdefense
AI technologies can benefit defenders as well. We don’t know how the
different technologies of cyber-offense and cyber-defense will be
amenable to AI enhancement, but we can extrapolate a possible series of
overlapping developments.
Phase One: The Transformation of the Vulnerability Researcher. AI-based
hacking benefits defenders as well as attackers. In this scenario, AI
empowers defenders to do more. It simplifies capabilities, providing far
more people the ability to perform previously complex tasks, and
empowers researchers previously busy with these tasks to accelerate or
move beyond them, freeing time to work on problems that require human
creativity. History suggests a pattern. Reverse engineering was a
laborious manual process until tools such as IDA Pro made the capability
available to many. AI vulnerability discovery could follow a similar
trajectory, evolving through scriptable interfaces, automated workflows,
and automated research before reaching broad accessibility.
Phase Two: The Emergence of VulnOps. Between research breakthroughs and
enterprise adoption, a new discipline might emerge: VulnOps. Large
research teams are already building operational pipelines around their
tooling. Their evolution could mirror how DevOps professionalized
software delivery. In this scenario, specialized research tools become
developer products. These products may emerge as a SaaS platform, or
some internal operational framework, or something entirely different.
Think of it as AI-assisted vulnerability research available to everyone,
at scale, repeatable, and integrated into enterprise operations.
Phase Three: The Disruption of the Enterprise Software Model. If
enterprises adopt AI-powered security the way they adopted continuous
integration/continuous delivery (CI/CD), several paths open up. AI
vulnerability discovery could become a built-in stage in delivery
pipelines. We can envision a world where AI vulnerability discovery
becomes an integral part of the software development process, where
vulnerabilities are automatically patched even before reaching
production -- a shift we might call continuous discovery/continuous
repair (CD/CR). Third-party risk management (TPRM) offers a natural
adoption route, lower-risk vendor testing, integration into procurement
and certification gates, and a proving ground before wider rollout.
Phase Four: The Self-Healing Network. If organizations can independently
discover and patch vulnerabilities in running software, they will not
have to wait for vendors to issue fixes. Building in-house research
teams is costly, but AI agents could perform such discovery and generate
patches for many kinds of code, including third-party and vendor
products. Organizations may develop independent capabilities that create
and deploy third-party patches on vendor timelines, extending the
current trend of independent open-source patching. This would increase
security, but having customers patch software without vendor approval
raises questions about patch correctness, compatibility, liability,
right-to-repair, and long-term vendor relationships.
These are all speculations. Maybe AI-enhanced cyberattacks won’t evolve
the ways we fear. Maybe AI-enhanced cyberdefense will give us
capabilities we can’t yet anticipate. What will surprise us most might
not be the paths we can see, but the ones we can’t imagine yet.
This essay was written with Heather Adkins and Gadi Evron, and
originally appeared in CSO.
--
Roger Clarke mailto:[email protected]
T: +61 2 6288 6916 http://www.xamax.com.au http://www.rogerclarke.com
Xamax Consultancy Pty Ltd 78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Visiting Professorial Fellow UNSW Law & Justice
Visiting Professor in Computer Science Australian National University
--- End Message ---
_______________________________________________
apf-media-archive mailing list
[email protected]
https://lists.privacy.org.au/mailman/listinfo/apf-media-archive
