date:20200720

Re: Getting CAPTCHA response when download a webpage

2020-07-20 Thread Ray Satiro via curl-library


On 7/19/2020 6:38 PM, Mah. E. via curl-library wrote:

i can open this link on any browser and never ask for recaptcha


I get a captcha in every browser.

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Getting CAPTCHA response when download a webpage

2020-07-20 Thread Jiahao XU via curl-library

Then I don’t know what is missing, too.

Get Outlook for iOS

From: Mah. E. 
Sent: Monday, July 20, 2020 6:44:31 PM
To: Jiahao XU 
Cc: libcurl development ; curl-us...@cool.haxx.se 

Subject: Re: Getting CAPTCHA response when download a webpage

This is line number 8 of the sample downloaded containing captcha response 

I extracted the header used by firefox and chrome, also i captured the headers 
used by KGet download manager and used it with curl and got the same results, 
still not sure what is missing

On Mon, Jul 20, 2020 at 12:27 PM Jiahao XU 
mailto:jiahao...@outlook.com>> wrote:
I think I know why curl’s output is so different to chrome:
The request header is different.

To be specific, I think it is probably the mime types that matters.

Chrome might have sent out mime types it supported in the requested header, the 
signal that html5 video player or something like that is available.

Without such mime settings, the site probably won’t send the right html page.

I am no where farmiliar with chrome or h5 video player, but I suppose that the 
request header plays an important rule here.

Get Outlook for iOS

From: Jiahao XU mailto:jiahao...@outlook.com>>
Sent: Monday, July 20, 2020 8:22:32 PM
To: Mah. E. mailto:mahmoud.aboghaz...@gmail.com>>
Cc: libcurl development 
mailto:curl-library@cool.haxx.se>>; 
curl-us...@cool.haxx.se 
mailto:curl-us...@cool.haxx.se>>
Subject: Re: Getting CAPTCHA response when download a webpage

You seemed to have a typo: it should be captcha, not capcha.

Get Outlook for iOS

From: Mah. E. 
mailto:mahmoud.aboghaz...@gmail.com>>
Sent: Monday, July 20, 2020 8:18:09 PM
To: Jiahao XU mailto:jiahao...@outlook.com>>
Cc: libcurl development 
mailto:curl-library@cool.haxx.se>>; 
curl-us...@cool.haxx.se 
mailto:curl-us...@cool.haxx.se>>
Subject: Re: Getting CAPTCHA response when download a webpage

I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes of output.

This 9393 bytes is an html contains captcha response and not the actual web 
page which should be around 130 kb
To make sure open the downloaded content with any editor and search for word 
“capcha” or “Cloudflare”
Also you will get same result with curl without setting any headers or user 
agent

I tried user agent and cookie file from a browser and nothing seems to work

Regards,
Mahmoud

On Jul 20, 2020, at 9:55 AM, Jiahao XU 
mailto:jiahao...@outlook.com>> wrote:

IMHO it could be user agent and cookies affecting the output.

I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes of output.

However, when I tried with wget, I got 403 forbidden.

Get Outlook for iOS

From: curl-library 
mailto:curl-library-boun...@cool.haxx.se>> 
on behalf of Mah. E. via curl-library 
mailto:curl-library@cool.haxx.se>>
Sent: Monday, July 20, 2020 8:38:47 AM
To: curl-us...@cool.haxx.se 
mailto:curl-us...@cool.haxx.se>>
Cc: Mah. E. 
mailto:mahmoud.aboghaz...@gmail.com>>; 
curl-library@cool.haxx.se 
mailto:curl-library@cool.haxx.se>>
Subject: Getting CAPTCHA response when download a webpage

First, Thanks for this awesome tool.

is there anyway to download this web page using curl

https://www.crunchyroll.com/en-gb/blood-blockade-battlefront/episode-6-get-the-lock-out-754075

because i get Cloudflare CAPTCHA html response "8 kb file"
but if i use a different downloaders like KGet or idm i get the actual page 
"131 Kb file size"

i can open this link on any browser and never ask for recaptcha
i tried also to use the full headers from the browser with curl command with no 
success

any help would be appreciated

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Getting CAPTCHA response when download a webpage

2020-07-20 Thread Mah. E. via curl-library

This is line number 8 of the sample downloaded containing captcha response


I extracted the header used by firefox and chrome, also i captured the
headers used by KGet download manager and used it with curl and got the
same results, still not sure what is missing

On Mon, Jul 20, 2020 at 12:27 PM Jiahao XU  wrote:

> I think I know why curl’s output is so different to chrome:
> The request header is different.
>
> To be specific, I think it is probably the mime types that matters.
>
> Chrome might have sent out mime types it supported in the requested
> header, the signal that html5 video player or something like that is
> available.
>
> Without such mime settings, the site probably won’t send the right html
> page.
>
> I am no where farmiliar with chrome or h5 video player, but I suppose that
> the request header plays an important rule here.
>
> Get Outlook for iOS 
> --
> *From:* Jiahao XU 
> *Sent:* Monday, July 20, 2020 8:22:32 PM
> *To:* Mah. E. 
> *Cc:* libcurl development ;
> curl-us...@cool.haxx.se 
> *Subject:* Re: Getting CAPTCHA response when download a webpage
>
> You seemed to have a typo: it should be captcha, not capcha.
>
> Get Outlook for iOS 
> --
> *From:* Mah. E. 
> *Sent:* Monday, July 20, 2020 8:18:09 PM
> *To:* Jiahao XU 
> *Cc:* libcurl development ;
> curl-us...@cool.haxx.se 
> *Subject:* Re: Getting CAPTCHA response when download a webpage
>
>
> I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes
> of output.
>
>
> This 9393 bytes is an html contains captcha response and not the actual
> web page which should be around 130 kb
> To make sure open the downloaded content with any editor and search for
> word “capcha” or “Cloudflare”
> Also you will get same result with curl without setting any headers or
> user agent
>
> I tried user agent and cookie file from a browser and nothing seems to work
>
> Regards,
> Mahmoud
>
> On Jul 20, 2020, at 9:55 AM, Jiahao XU  wrote:
>
> IMHO it could be user agent and cookies affecting the output.
>
> I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes
> of output.
>
> However, when I tried with wget, I got 403 forbidden.
>
> Get Outlook for iOS 
> --
> *From:* curl-library  on behalf of
> Mah. E. via curl-library 
> *Sent:* Monday, July 20, 2020 8:38:47 AM
> *To:* curl-us...@cool.haxx.se 
> *Cc:* Mah. E. ; curl-library@cool.haxx.se <
> curl-library@cool.haxx.se>
> *Subject:* Getting CAPTCHA response when download a webpage
>
> First, Thanks for this awesome tool.
>
> is there anyway to download this web page using curl
>
>
> https://www.crunchyroll.com/en-gb/blood-blockade-battlefront/episode-6-get-the-lock-out-754075
>
> because i get Cloudflare CAPTCHA html response "8 kb file"
> but if i use a different downloaders like KGet or idm i get the actual
> page "131 Kb file size"
>
> i can open this link on any browser and never ask for recaptcha
> i tried also to use the full headers from the browser with curl command
> with no success
>
> any help would be appreciated
>
>
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Getting CAPTCHA response when download a webpage

2020-07-20 Thread Jiahao XU via curl-library

I think I know why curl’s output is so different to chrome:
The request header is different.

To be specific, I think it is probably the mime types that matters.

Chrome might have sent out mime types it supported in the requested header, the 
signal that html5 video player or something like that is available.

Without such mime settings, the site probably won’t send the right html page.

I am no where farmiliar with chrome or h5 video player, but I suppose that the 
request header plays an important rule here.

Get Outlook for iOS

From: Jiahao XU 
Sent: Monday, July 20, 2020 8:22:32 PM
To: Mah. E. 
Cc: libcurl development ; curl-us...@cool.haxx.se 

Subject: Re: Getting CAPTCHA response when download a webpage

You seemed to have a typo: it should be captcha, not capcha.

Get Outlook for iOS

From: Mah. E. 
Sent: Monday, July 20, 2020 8:18:09 PM
To: Jiahao XU 
Cc: libcurl development ; curl-us...@cool.haxx.se 

Subject: Re: Getting CAPTCHA response when download a webpage

I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes of output.

This 9393 bytes is an html contains captcha response and not the actual web 
page which should be around 130 kb
To make sure open the downloaded content with any editor and search for word 
“capcha” or “Cloudflare”
Also you will get same result with curl without setting any headers or user 
agent

I tried user agent and cookie file from a browser and nothing seems to work

Regards,
Mahmoud

On Jul 20, 2020, at 9:55 AM, Jiahao XU 
mailto:jiahao...@outlook.com>> wrote:

IMHO it could be user agent and cookies affecting the output.

I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes of output.

However, when I tried with wget, I got 403 forbidden.

Get Outlook for iOS

From: curl-library 
mailto:curl-library-boun...@cool.haxx.se>> 
on behalf of Mah. E. via curl-library 
mailto:curl-library@cool.haxx.se>>
Sent: Monday, July 20, 2020 8:38:47 AM
To: curl-us...@cool.haxx.se 
mailto:curl-us...@cool.haxx.se>>
Cc: Mah. E. 
mailto:mahmoud.aboghaz...@gmail.com>>; 
curl-library@cool.haxx.se 
mailto:curl-library@cool.haxx.se>>
Subject: Getting CAPTCHA response when download a webpage

First, Thanks for this awesome tool.

is there anyway to download this web page using curl

https://www.crunchyroll.com/en-gb/blood-blockade-battlefront/episode-6-get-the-lock-out-754075

because i get Cloudflare CAPTCHA html response "8 kb file"
but if i use a different downloaders like KGet or idm i get the actual page 
"131 Kb file size"

i can open this link on any browser and never ask for recaptcha
i tried also to use the full headers from the browser with curl command with no 
success

any help would be appreciated

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Getting CAPTCHA response when download a webpage

2020-07-20 Thread Jiahao XU via curl-library

You seemed to have a typo: it should be captcha, not capcha.

Get Outlook for iOS

From: Mah. E. 
Sent: Monday, July 20, 2020 8:18:09 PM
To: Jiahao XU 
Cc: libcurl development ; curl-us...@cool.haxx.se 

Subject: Re: Getting CAPTCHA response when download a webpage

I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes of output.

This 9393 bytes is an html contains captcha response and not the actual web 
page which should be around 130 kb
To make sure open the downloaded content with any editor and search for word 
“capcha” or “Cloudflare”
Also you will get same result with curl without setting any headers or user 
agent

I tried user agent and cookie file from a browser and nothing seems to work

Regards,
Mahmoud

On Jul 20, 2020, at 9:55 AM, Jiahao XU 
mailto:jiahao...@outlook.com>> wrote:

IMHO it could be user agent and cookies affecting the output.

I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes of output.

However, when I tried with wget, I got 403 forbidden.

Get Outlook for iOS

From: curl-library 
mailto:curl-library-boun...@cool.haxx.se>> 
on behalf of Mah. E. via curl-library 
mailto:curl-library@cool.haxx.se>>
Sent: Monday, July 20, 2020 8:38:47 AM
To: curl-us...@cool.haxx.se 
mailto:curl-us...@cool.haxx.se>>
Cc: Mah. E. 
mailto:mahmoud.aboghaz...@gmail.com>>; 
curl-library@cool.haxx.se 
mailto:curl-library@cool.haxx.se>>
Subject: Getting CAPTCHA response when download a webpage

First, Thanks for this awesome tool.

is there anyway to download this web page using curl

https://www.crunchyroll.com/en-gb/blood-blockade-battlefront/episode-6-get-the-lock-out-754075

because i get Cloudflare CAPTCHA html response "8 kb file"
but if i use a different downloaders like KGet or idm i get the actual page 
"131 Kb file size"

i can open this link on any browser and never ask for recaptcha
i tried also to use the full headers from the browser with curl command with no 
success

any help would be appreciated

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Getting CAPTCHA response when download a webpage

2020-07-20 Thread Mah. E. via curl-library

> I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 
> (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes of 
> output.

This 9393 bytes is an html contains captcha response and not the actual web 
page which should be around 130 kb
To make sure open the downloaded content with any editor and search for word 
“capcha” or “Cloudflare”
Also you will get same result with curl without setting any headers or user 
agent 

I tried user agent and cookie file from a browser and nothing seems to work

Regards,
Mahmoud

> On Jul 20, 2020, at 9:55 AM, Jiahao XU  wrote:
> 
> IMHO it could be user agent and cookies affecting the output.
> 
> I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 
> (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes of 
> output.
> 
> However, when I tried with wget, I got 403 forbidden.
> 
> Get Outlook for iOS
> From: curl-library  on behalf of Mah. E. 
> via curl-library 
> Sent: Monday, July 20, 2020 8:38:47 AM
> To: curl-us...@cool.haxx.se 
> Cc: Mah. E. ; curl-library@cool.haxx.se 
> 
> Subject: Getting CAPTCHA response when download a webpage
>  
> First, Thanks for this awesome tool.
> 
> is there anyway to download this web page using curl
> 
> https://www.crunchyroll.com/en-gb/blood-blockade-battlefront/episode-6-get-the-lock-out-754075
> 
> because i get Cloudflare CAPTCHA html response "8 kb file" 
> but if i use a different downloaders like KGet or idm i get the actual page 
> "131 Kb file size"
> 
> i can open this link on any browser and never ask for recaptcha
> i tried also to use the full headers from the browser with curl command with 
> no success
> 
> any help would be appreciated
> 
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Getting CAPTCHA response when download a webpage

2020-07-20 Thread Jiahao XU via curl-library

IMHO it could be user agent and cookies affecting the output.

I set useragent to “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/84.0.4147.89 Safari/537.36” and got  9393 bytes of output.

However, when I tried with wget, I got 403 forbidden.

Get Outlook for iOS

From: curl-library  on behalf of Mah. E. via 
curl-library 
Sent: Monday, July 20, 2020 8:38:47 AM
To: curl-us...@cool.haxx.se 
Cc: Mah. E. ; curl-library@cool.haxx.se 

Subject: Getting CAPTCHA response when download a webpage

First, Thanks for this awesome tool.

is there anyway to download this web page using curl

https://www.crunchyroll.com/en-gb/blood-blockade-battlefront/episode-6-get-the-lock-out-754075

because i get Cloudflare CAPTCHA html response "8 kb file"
but if i use a different downloaders like KGet or idm i get the actual page 
"131 Kb file size"

i can open this link on any browser and never ask for recaptcha
i tried also to use the full headers from the browser with curl command with no 
success

any help would be appreciated

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Getting CAPTCHA response when download a webpage

2020-07-20 Thread Mah. E. via curl-library

First, Thanks for this awesome tool.

is there anyway to download this web page using curl

https://www.crunchyroll.com/en-gb/blood-blockade-battlefront/episode-6-get-the-lock-out-754075

because i get Cloudflare CAPTCHA html response "8 kb file"
but if i use a different downloaders like KGet or idm i get the actual page
"131 Kb file size"

i can open this link on any browser and never ask for recaptcha
i tried also to use the full headers from the browser with curl command
with no success

any help would be appreciated
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: How to use libcurl to get the final URL of a site after a redirection ?

2020-07-20 Thread Morbius via curl-library

On Sun, 19 Jul 2020, Daniel Stenberg via curl-library wrote:

>
>On Sun, 19 Jul 2020, Morbius via curl-library wrote:
>
>> I would like to write a function to obtain the final URL from a site
>> which
>> redirects to another site using libcurl functions. I need some guidance
>> on
>> this.
>
>CURLINFO_EFFECTIVE_URL ?
>
>https://curl.haxx.se/libcurl/c/CURLINFO_EFFECTIVE_URL.html
>

Works perfectly. Many thanks, Daniel.
Problem solved.
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Getting CAPTCHA response when download a webpage

Re: Getting CAPTCHA response when download a webpage

Re: Getting CAPTCHA response when download a webpage

Re: Getting CAPTCHA response when download a webpage

Re: Getting CAPTCHA response when download a webpage

Re: Getting CAPTCHA response when download a webpage

Re: Getting CAPTCHA response when download a webpage

Getting CAPTCHA response when download a webpage

Re: How to use libcurl to get the final URL of a site after a redirection ?

9 matches

Site Navigation

Mail list logo

Footer information