On 10/20/25 5:50 AM, Hans Henrik Bergan wrote:
On Sat, 18 Oct 2025 at 12:03, ToddAndMargo via curl-users
<[email protected]> wrote:
Hi All,
curl-8.9.1-4.fc41.x86_64
How do I get around "You've been blocked" on
this web sire:
$ curl -L
https://www.softpedia.com/get/System/Back-Up-and-Recovery/Icedrive.shtml#download
-o eraseme.html
adding
--user-agent 'Mozilla/5.0 (X11; Linux x86_64; rv:143.0)
Gecko/20100101 Firefox/143.0'
Did not help.
Yours in confusion,
-T
softpedia.com use the cloudflare WAF thing.
Cloudflare WAF detect curl by minor details in its TLS handshake, and
block curl,
even if the user-agent is whitelisted, WAF detect that it's actually
not firefox because
the TLS details doesn't match.
There is a project called curl-impersonate,
https://github.com/lwthiker/curl-impersonate ,
where curl has been modified to carefully mimic the Chromium and
Firefox TLS implementation, for example:
./curl-impersonate-ff --user-agent 'Mozilla/5.0 (Windows NT 10.0;
Win64; x64; rv:140.0) Gecko/20100101 Firefox/140.0'
'https://www.softpedia.com/get/System/Back-Up-and-Recovery/Icedrive.shtml'
works. The useragent claims to be firefox, and it's been modified to
mimic Firefox TLS.
Hi Hans,
Well, my work around randomly fail. Probably dependent on what
cloutflare server I am logging into at the server farm.
But curl-impersonate is getting around it.
Thank you!
-T
My notes:
How to install and use curl-impersonate with Fedora:
Fedora:
# rpm install nss nss-pem ca-certificates
Download a precompiled binary from:
https://github.com/lwthiker/curl-impersonate/releases
https://github.com/lwthiker/curl-impersonate/releases/download/v0.6.1/curl-impersonate-v0.6.1.x86_64-linux-gnu.tar.gz
create a directory and de-tar the above into it.
The following site, about half way downm will give you what
each script impersonates.
The following examples uses
chrome116 (Chrome build 116.0.5845.180 and Windows 10)
Download a web page:
$ curl_chrome116 https://icedrive.net/apps/desktop-laptop > eraseme.html
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 4986 0 4986 0 0 17467 0 --:--:-- --:--:-- --:--:--
17494
Download a binary file:
$ curl_chrome116
https://cdn.icedrive.net/static/apps/win/IcedriveSetup-v3.56.exe >
eraseme.exe
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 256M 100 256M 0 0 525k 0 0:08:19 0:08:19 --:--:--
458k
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette: https://curl.se/mail/etiquette.html
