Send Link mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://mailman.anu.edu.au/mailman/listinfo/link
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Link digest..."
Today's Topics:
1. DeepSeek .. does more with less (Stephen Loosley)
2. Re: DeepSeek (Tom Worthington)
----------------------------------------------------------------------
Message: 1
Date: Thu, 30 Jan 2025 05:32:59 +0000
From: Stephen Loosley <[email protected]>
To: "[email protected]" <[email protected]>
Subject: [LINK] DeepSeek .. does more with less
Message-ID:
<sy5p282mb4409eb5bb1c11f9f4228e736c2...@sy5p282mb4409.ausp282.prod.outlook.com>
Content-Type: text/plain; charset="Windows-1252"
https://gizmodo.com/what-deepseeks-ai-did-that-everyone-else-didnt-2000555731
What DeepSeek?s AI Did That Everyone Else?s Didn?t
The Chinese AI company didn't surpass OpenAI by going bigger or inventing new
techniques.
By Todd Feathers Published January 28, 2025 | Comments (146)
[Photo caption: DeepSeek's latest AI models outperform OpenAI's on some
advanced tasks and were much cheaper to build. ? CFOTO/Getty Images]
The Chinese AI company DeepSeek exploded into the news cycle over the weekend
after it replaced OpenAI?s ChatGPT as the most downloaded app on the Apple App
Store.
Its commercial success followed the publication of several papers in which
DeepSeek announced that its newest R1 models?which cost significantly less for
the company to make and for customers to use?are equal to, and in some cases
surpass, OpenAI?s best publicly available models.
So what did DeepSeek do that deep-pocketed OpenAI didn?t?
It?s hard to say with certainty because OpenAI has been pretty cagey about how
it trained its GPT-o1 model, the previous leader on a variety of benchmark
tests. But there are some clear differences in the companies? approaches and
other areas where DeepSeek appears to have made impressive breakthroughs.
Probably the biggest difference?and certainly the one that sent the stocks of
chip makers like NVIDIA tumbling on Monday?is that DeepSeek is creating
competitive models much more efficiently than its bigger counterparts.
The company?s latest R1 and R1-Zero ?reasoning? models are built on top of
DeepSeek?s V3 base model, which the company said was trained for less than $6
million in computing costs using older NVIDIA hardware (which is legal for
Chinese companies to buy, unlike the company?s state-of-the-art chips). By
comparison, OpenAI CEO Sam Altman said that GPT-4 cost more than $100 million
to train.
Karl Freund, founder of the industry analysis firm Cambrian AI Research, told
Gizmodo that U.S. policies like the recent ban on advanced chip sales to China
have forced companies like DeepSeek to improve by optimizing the architecture
of their models rather than throwing money at better hardware and
Manhattan-sized data centers.
?You can build a model quickly or you can do the hard work to build it
efficiently,? Freund said.
?The impact on Western companies will be that they?ll be forced to do the hard
work that they?ve not been willing to undertake.?
DeepSeek didn?t invent most of the optimization techniques it used. Some, like
using data formats that use less memory, have been proposed by its bigger
competitors. The picture that emerges from DeepSeek?s papers?even for
technically ignorant readers?is of a team that pulled in every tool they could
find to make training require less computing memory and designed its model
architecture to be as efficient as possible on the older hardware it was using.
OpenAI was the first developer to introduce so-called reasoning models, which
use a technique called chain-of-thought that mimics humans? trial-and-error
method of problem solving to complete complex tasks, particularly in math and
coding. The company hasn?t said how exactly it did that.
DeepSeek, on the other hand, laid out its process.
In the past, generative AI models have been improved by incorporating what?s
known as reinforcement learning with human feedback (RLHF). Humans label the
good and bad characteristics of a bunch of AI responses and the model is
incentivized to emulate the good characteristics, like accuracy and coherency.
DeepSeek?s big innovation in building its R1 models was to do away with human
feedback and design its algorithm to recognize and correct its own mistakes.
?DeepSeekR1-Zero demonstrates capabilities such as self-verification,
reflection, and generating long [chains-of-thought], marking a significant
milestone for the research community,? the researchers wrote.
?Notably, it is the first open research to validate that reasoning capabilities
of [large language models] can be incentivized purely through [reinforcement
learning].?
The results of the pure reinforcement learning approach weren?t perfect. The
R1-Zero model?s outputs were sometimes difficult to read and switched between
languages. So DeepSeek created a new training pipeline that incorporates a
relatively small amount of labeled data to nudge the model in the preferred
direction combined with several rounds of pure reinforcement learning.
The resulting model, R1, outperformed OpenAI?s GPT-o1 model on several math and
coding problem sets designed for humans.
Bill Hannas and Huey-Meei Chang, experts on Chinese technology and policy at
the Georgetown Center for Security and Emerging Technology, said China closely
monitors the technological breakthroughs and practices of Western companies
which has helped its companies find workarounds to U.S. policies like chip
embargoes that are designed to give American companies an advantage.
DeepSeek?s success, they said, isn?t a bad thing for the domestic industry but
it is ?a wake-up call to U.S. AI companies obsessed with gargantuan (and
expensive) solutions.
?Doing more with less? underpins the approach taken at several Chinese
state-funded labs.?
--
------------------------------
Message: 2
Date: Fri, 31 Jan 2025 09:54:48 +1100
From: Tom Worthington <[email protected]>
To: [email protected]
Subject: Re: [LINK] DeepSeek
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
On 1/29/25 17:52, Stephen Loosley wrote:
> Must say .. DeepSeek-V3 seems amazing? ...
Do you have an example of amazing results from it?
I find the specifications unbelievable, in the literal sense, rather
than amazing.
If nothing else it is good to see US tech & political leaders "Hoist
with his own petard". ;-)
--
Tom Worthington http://www.tomw.net.au
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 665 bytes
Desc: OpenPGP digital signature
URL:
<https://mailman.anu.edu.au/pipermail/link/attachments/20250131/a781db73/attachment-0001.sig>
------------------------------
Subject: Digest Footer
_______________________________________________
Link mailing list
[email protected]
https://mailman.anu.edu.au/mailman/listinfo/link
------------------------------
End of Link Digest, Vol 386, Issue 16
*************************************
