Google’s call-scanning AI could dial up censorship by default, privacy experts
warn
Natasha Lomas - May 15, 2024
A feature Google demoed at its I/O confab yesterday, using its generative AI
technology to scan voice calls in real time for conversational patterns
associated with financial scams, has sent a collective shiver down the spines
of privacy and security experts who are warning the feature represents the thin
end of the wedge. They warn that, once client-side scanning is baked into
mobile infrastructure, it could usher in an era of centralized censorship.
Google’s demo of the call scam-detection feature, which the tech giant said
would be built into a future version of its Android OS — estimated to run on
some three-quarters of the world’s smartphones — is powered by Gemini Nano, the
smallest of its current generation of AI models meant to run entirely on-device.
This is essentially client-side scanning: A nascent technology that’s generated
huge controversy in recent years in relation to efforts to detect child sexual
abuse material (CSAM) or even grooming activity on messaging platforms.
Apple abandoned a plan to deploy client-side scanning for CSAM in 2021 after a
huge privacy backlash. However, policymakers have continued to heap pressure on
the tech industry to find ways to detect illegal activity taking place on their
platforms. Any industry moves to build out on-device scanning infrastructure
could therefore pave the way for all-sorts of content scanning by default —
whether government-led or related to a particular commercial agenda.
Responding to Google’s call-scanning demo in a post on X, Meredith Whittaker,
president of the U.S.-based encrypted messaging app Signal, warned: “This is
incredibly dangerous. It lays the path for centralized, device-level client
side scanning.
“From detecting ‘scams’ it’s a short step to ‘detecting patterns commonly
associated w[ith] seeking reproductive care’ or ‘commonly associated w[ith]
providing LGBTQ resources’ or ‘commonly associated with tech worker
whistleblowing.’”
Cryptography expert Matthew Green, a professor at Johns Hopkins, also took to X
to raise the alarm. “In the future, AI models will run inference on your texts
and voice calls to detect and report illicit behavior,” he warned. “To get your
data to pass through service providers, you’ll need to attach a zero-knowledge
proof that scanning was conducted. This will block open clients.”
Green suggested this dystopian future of censorship by default is only a few
years out from being technically possible. “We’re a little ways from this tech
being quite efficient enough to realize, but only a few years. A decade at
most,” he suggested.
European privacy and security experts were also quick to object.
Reacting to Google’s demo on X, Lukasz Olejnik, a Poland-based independent
researcher and consultant for privacy and security issues, welcomed the
company’s anti-scam feature but warned the infrastructure could be repurposed
for social surveillance. “[T]his also means that technical capabilities have
already been, or are being developed to monitor calls, creation, writing texts
or documents, for example in search of illegal, harmful, hateful, or otherwise
undesirable or iniquitous content — with respect to someone’s standards,” he
wrote.
“Going further, such a model could, for example, display a warning. Or block
the ability to continue,” Olejnik continued with emphasis. “Or report it
somewhere. Technological modulation of social behaviour, or the like. This is a
major threat to privacy, but also to a range of basic values and freedoms. The
capabilities are already there.”
Fleshing out his concerns further, Olejnik told TechCrunch: “I haven’t seen the
technical details but Google assures that the detection would be done
on-device. This is great for user privacy. However, there’s much more at stake
than privacy. This highlights how AI/LLMs inbuilt into software and operating
systems may be turned to detect or control for various forms of human activity.
This highlights how AI/LLMs inbuilt into software and operating systems may
be turned to detect or control for various forms of human activity.
Lukasz Olejnik
“So far it’s fortunately for the better. But what’s ahead if the technical
capability exists and is built in? Such powerful features signal potential
future risks related to the ability of using AI to control the behavior of
societies at a scale or selectively. That’s probably among the most dangerous
information technology capabilities ever being developed. And we’re nearing
that point. How do we govern this? Are we going too far?”
Michael Veale, an associate professor in technology law at UCL, also raised the
chilling specter of function-creep flowing from Google’s conversation-scanning
AI — warning in a reaction post on X that it “sets up infrastructure for
on-device client side scanning for more purposes than this, which regulators
and legislators will desire to abuse.”
Privacy experts in Europe have particular reason for concern: The European
Union has had a controversial message-scanning legislative proposal on the
table since 2022, which critics — including the bloc’s own Data Protection
Supervisor — warn represents a tipping point for democratic rights in the
region as it would force platforms to scan private messages by default.
While the current legislative proposal claims to be technology agnostic, it’s
widely expected that such a law would lead to platforms deploying client-side
scanning in order to be able to respond to a so-called detection order
demanding they spot both known and unknown CSAM and also pick up grooming
activity in real time.
Earlier this month, hundreds of privacy and security experts penned an open
letter warning the plan could lead to millions of false positives per day, as
the client-side scanning technologies that are likely to be deployed by
platforms in response to a legal order are unproven, deeply flawed and
vulnerable to attacks.
Google was contacted for a response to concerns that its conversation-scanning
AI could erode people’s privacy but at press time it had not responded.
https://techcrunch.com/2024/05/15/googles-call-scanning-ai-could-dial-up-censorship-by-default-privacy-experts-warn/
