Emotional robots: Will we love them or hate them?
Software that can tell our emotional state is on the way, but will
we like machines that can sense how we feel?
by Hazel Muir
SUNDAY, 1 February 2009, and 100 million Americans have got only one
thing on their minds - the Super Bowl. The Pittsburgh Steelers are
poised to battle the Arizona Cardinals in the most popular televised
sporting event in the US. In a hotel room in New York, 46 supporters
gather to watch the game, munching burgers and downing beers. Nothing
strange about that, of course, aside from the machines that are
monitoring these sports fans' every move and every breath they take.
The viewers are wearing vests with sensors that monitor their heart
rate, movement, breathing and sweat. A market research company has
kitted out the party-goers with these sensors to measure their
emotional engagement with adverts during commercial breaks.
Advertisers pay $3 million for a 30-second slot during the Super Bowl,
so they want to be as confident as they can be that their ads are
hitting home. And they are willing to pay for the knowledge. "It's a
rapidly growing market - our revenues this year are four times what
they were last year," says Carl Marci, CEO and chief scientist for the
company running the experiment, Innerscope Research based in Boston,
Massachusetts.
Viewers are wearing vests with sensors that monitor their heart rate,
movement, breathing and sweat
Innerscope's approach is the latest in a wave of ever more
sophisticated emotion-sensing technologies. For years, computers in
some call centres have monitored our voices so that managers can home
in on what makes us fly into a wild rage. The latest technologies
could soon be built into everyday gadgets to smooth our interactions
with them. In-car alarms that jolt sleepy drivers awake, satnavs that
sense our frustration in a traffic jam and offer alternative routes,
and monitors that diagnose depression from body language are all in
the pipeline. Prepare for the era of emotionally aware gadgets.
Outside of science fiction, the idea of technology that reads emotions
has a brief, and chequered, past. Back in the mid-1990s, computer
scientist Rosalind Picard at the Massachusetts Institute of Technology
suggested pursuing this sort of research. She was greeted with
scepticism. "It was such a taboo topic back then - it was seen as very
undesirable, soft and irrelevant," she says.
Picard persevered, and in 1997 published a book called Affective
Computing, which laid out the case that many technologies would work
better if they were aware of their user's feelings. For instance, a
computerised tutor could slow down its pace or give helpful
suggestions if it sensed a student looking frustrated, just as a human
teacher would.
She also suggested that wearable computers could sense emotion in a
very direct way, by measuring your heart and breathing rate, or the
changes in the skin's electrical conductance that signal emotional
arousal. Wearable "mood detectors" could help people identify their
stress triggers or communicate how they are feeling to others.
The most established way to analyse a person's feelings is through the
tone of their voice. For several years, companies have been using
"speech analytics" software that automatically monitors conversations
between call-centre agents and customers. One supplier is NICE
Systems, based in Ra'anana, Israel. It specialises in
emotion-sensitive software and call-monitoring systems for companies
and security organisations, and claims to have more than 24,000
customers worldwide, including the New York Police Department and
Vodafone.
As well as scanning audio files for key words and phrases, such as a
competitor's name, the software measures stress levels, as indicated
by voice pitch and talking speed. Computers flag up calls in which
customers appear to get angry or stressed out, perhaps because they
are making a fraudulent insurance claim, or simply receiving poor
service.
Voice works well when the person whose feelings you are trying to
gauge is expressing themselves verbally, but that's not always the
case, so several research teams are now figuring out ways of reading a
person's feelings by analysing their posture and facial expressions
alone.
Many groups have made impressive progress in the field, first by
training computers to identify a face as such. Systems do this by
searching for skin tone and using algorithms to locate features like
the corners of the eyes and eyebrows, the nostrils and corners of the
mouth (see diagram).
The computer can then keep track of these features as they move, often
classifying the movements according to a commonly used emotion
encoding system. That system recognises 44 "action units" representing
facial movements. For instance, one might represent a smile - the
mouth stretches horizontally and its corners go up. Add to that an
eye-region movement that raises the cheeks and gives you crow's feet
and now you have a beaming, genuinely happy smile rather than a stiff,
polite one.
Using these techniques, computer programs can correctly recognise six
basic emotions - disgust, happiness, sadness, anger, fear and surprise
- more than 9 times out of 10, but only if the target face uses an
exaggerated expression. Software can accurately judge more subtle,
spontaneous facial expressions as "negative" or "positive"
three-quarters of the time, but they cannot reliably spot spontaneous
displays of the six specific emotions - yet. To accurately interpret
complex, realistic emotions, computers will need extra cues, such as
upper body posture and head motion.
Computers can recognise emotions such as disgust, happiness, sadness,
anger, fear and surprise more than nine times out of ten
That's because facial expressions alone are ambiguous. A smile on your
face might actually signal embarrassment if it's also accompanied by a
downward pitch of the head, for instance. A backward head motion is
one part of an expression of disgust. But if someone combines that
with a downward movement of the mouth and one raised shoulder, they're
conveying indifference. "If I just looked at the face and saw the
mouth going down, I would score it as sadness. But the combination
with the shoulder and head motion is 'I don't care'," says Maja
Pantic, who studies computer recognition of expressions at Imperial
College London.
Pantic's team eventually hopes to find ways of fusing information from
body gestures and facial expressions together in real time to read
emotions accurately, although she concedes it may be an impossibly
complex challenge. "This research is still so very new," she notes.
Basic emotions
In the meantime, they are studying the dynamics of how expressions
change, to see if this can help computers identify emotions more
accurately. Intuitively, most people know that a faked smile is more
exaggerated than a real one, and switches on and off more abruptly.
Facial-tracking technology has confirmed that, and also revealed some
more subtle differences.
These subtleties came to light in a 2004 study of 81 adults by Jeffrey
Cohn and Karen Schmidt at the University of Pittsburgh in Pennsylvania
(International Journal of Wavelets, Multiresolution and
Information Processing, vol 2, p 1). They used tracking technology to
compare forced smiles with spontaneous smiles provoked by comedy
videos. This showed that spontaneous smiles are surprisingly complex,
with multiple rises of the mouth corners.
Other teams have been highly successful at the opposite end of the
emotional spectrum: pain detection. Computers are surprisingly good at
distinguishing fake pain from the real thing, according to a study
published this year by Gwen Littlewort of the University of
California, San Diego, and colleagues.
Her team investigated whether facial expression software could
distinguish people in real pain (because their hands were in iced
water) from others asked to fake pain. The computer correctly
classified real or fake pain 88 per cent of the time. When the team
asked 170 untrained volunteers to make the judgement, they were right
only 49 per cent of the time - no better than complete guesswork.
This year, Pantic and her colleagues hope to find out whether
computers can accurately recognise the signs of lower back pain from
facial expressions and body posture. They hope that computers might be
able to distinguish between real physiological pain and the pain
someone might perceive, quite genuinely, if they expect to feel pain
or are depressed, but have no physiological cause for it. It could
lead to more reliable ways of assessing whether painkillers are
effective. "If you get a prescribed medication for acute pain, we
would be able to monitor whether these medicines are actually working
just by observing a person's behaviour," says Pantic.
One group of researchers has developed emotion-reading technology for
a particularly vulnerable group of people. Picard and Rana el Kaliouby
of MIT have built an "Interactive Social-Emotional Toolkit" (iSET),
designed to help children with disorders linked to sensory processing,
such as autism, to understand emotions in other people. A camera
monitors the face of someone the child is talking to, and identifies
31 facial and head movements. Software interprets the combination of
movements in terms of six states: agreeing, disagreeing,
concentrating, thinking, interested and confused.
Then a laptop-sized screen displays six labelled bubbles that grow or
shrink accordingly. If someone's nodding and smiling during the
conversation, the agreeing bubble grows. If the listener looks away, a
growing red bubble signals disagreement or disinterest. The team will
began randomised trials of the technology this month. For 15 weeks,
one group of five autistic children will use the iSET, while two
control groups will use either an interactive DVD that teaches
emotional awareness or have only standard classroom training. Before
and afterwards, the researchers will test how well the children
identify emotional expressions unaided by the iSET to see if the
technology helps them learn to identify emotions for themselves.
Patronising paperclips
Not everyone welcomes these developments. William Gaver, a designer at
Goldsmiths, University of London, concedes some of the applications
may be beneficial, but fears emotion-sensing computers will be used in
patronising ways. Who could forget Microsoft's cringe-making
"paperclip" that offered help with writing letters: Microsoft wisely
killed it off because people found it so irritating. But what if some
emotion-triggered reincarnated "Mr Clippy" started popping up
everywhere?
"The nightmare scenario is that the Microsoft paperclip starts to be
associated with anything from the force with which you're typing to
some sort of physiological measurement," says Gaver. "Then it pops up
on your screen and says: 'Oh I'm sorry you're unhappy, would you like
me to help you with that?'"
Emotion sensors could undermine personal relationships, he adds.
Monitors that track elderly people in their homes, for instance, could
leave them isolated. "Imagine being in a hurry to get home and
wondering whether to visit an older friend on the way," says Gaver.
"Wouldn't this be less likely if you had a device to reassure you not
only that they were active and safe, but showing all the physiological
and expressive signs of happiness as well?"
Picard raises another concern - that emotion-sensing technologies
might be used covertly. Security services could use face and
posture-reading systems to sense stress in people from a distance (a
common indicator a person may be lying), even when they're unaware of
it. Imagine if an unsavoury regime got hold of such technology and
used it to identify citizens who opposed it, says Picard. There has
already been progress towards stress detectors. For instance, research
by Ioannis Pavlidis at the University of Houston, Texas, has shown
that thermal imaging of people's faces can sense stress-induced
increases in blood flow around the eyes.
His team analysed thermal videos of 39 political activists given the
opportunity to commit a mock crime - stealing a cheque left in an
empty corridor, made payable to an organisation they strongly opposed.
They had to deny it during subsequent interrogation, and were
threatened with financial penalties and punishments of loud noise if
the interrogator caught them lying (empty threats at it turned out,
for ethical reasons). Computer analysis of the videos correctly
distinguished the 15 innocent and 24 guilty "suspects" 82 per cent of
the time.
Another fledgeling technique, called laser Doppler vibrometry,
measures tiny stress-related changes in respiration and heartbeat from
afar - indicators that are sometimes used to gauge whether a person is
stressed, and hence possibly lying.
Picard says that anyone utilising emotion-sensing systems should be
obliged to gain informed consent from the people they plan to "read".
At least that way, whether you find it patronising, creepy or just
plain annoying, you can hit the big "off" button and it will, or at
least should, leave you and your emotions in peace.
I know how you feel
DO YOU reckon you're a master of reading another's true feelings? Many
people think they are, but only about 1 in 100 of us are naturally
gifted at recognising emotions in someone who's trying to conceal
them, says Paul Ekman, a psychologist formerly at the University of
California, San Francisco.
Ekman made his name when he identified the facial expressions of the
seven key emotions that are universal, regardless of nationality or
culture - happiness, sadness, fear, anger, disgust, contempt and
surprise. He also acts as a consultant to law-enforcement agencies,
advising them on how to spot liars from clues in their facial
expressions, speech and body movements.
It takes considerable effort to be a good human lie detector. To begin
with, it is essential to know your subject's "baseline" behaviour when
they're not stressed. Then look for deviations from this when they're
under interrogation. Ekman points out that not everyone is the same.
For example, some people look fearful regardless of their emotions.
So there are no absolute signs that people are definitely lying, but
here are some of Ekman's top tips for spotting a fraud:
DO THEY HAVE RHYTHM?
Clues in the voice include unusually long or frequent pauses. People
who are having trouble deciding exactly what to say usually use fewer
hand gestures to reinforce their speech - they're less likely to
"conduct" their speech by waving their hands.
LOOK OUT FOR FLICKERS
People can't help showing their true feelings for a fraction of a
second. For example, a person might try to conceal their feelings of
contempt, but give it away with a fleeting raised lip on one side, so
look out for these micro-expressions. (Test your ability to interpret
micro-expressions at www.facetest.notlong.com)
SPOT THE GESTURAL SLIPS
Some gestures, called "emblems", have a precise meaning within a
cultural group. Examples include a shoulder shrug with upward palms,
communicating "who cares" or "I'm helpless". Usually people make them
obvious, but when lying, they may display an incomplete emblem. They
might rotate their hands upwards on their lap - a subconscious
fragment of the shrug that betrays their feeling of helplessness at
not lying well.
Hazel Muir is a freelance writer based in the UK
To unsubscribe send a message to [email protected] with
the subject unsubscribe.
To change your subscription to digest mode or make any other changes, please
visit the list home page at
http://accessindia.org.in/mailman/listinfo/accessindia_accessindia.org.in
