Re: [Mojolicious] curl works, Mojo::UserAgent doesn't

[lackhoff via Mojolicious]

Am 14.05.2020 um 05:08 schrieb Stefan Adams:
> I'm certain you'll be able to accomplish this with M::UA. Can you share

Just to reduce complexity I changed everything to plain HTTP, so no SSL,
keys or certificates.

> what the HTML error produced is?

It is not really a normal error page, just a page from the UI. The most
relevant lines are these:
Invalid variable name.
Wiederholen Sie das Update oder starten Sie die FRITZ!Box neu.
[German for: repeat the update(sic! I wanted to backup the box, not
update) or restart your FRITZ!Box]

> Can you share the output that you are comparing that you are noticing is so
> similar between curl and M::UA? You're using -v for curl. Also set the
> MOJO_CLIENT_DEBUG env variable to 1.

Here is what curl gives (with --trace-ascii):
=> Send header, 200 bytes (0xc8)
: POST /cgi-bin/firmwarecfg HTTP/1.1
0024: Host: fritz.box
0035: User-Agent: curl/7.58.0
004e: Accept: */*
005b: Content-Length: 370
0070: Content-Type: multipart/form-data; boundary=
00b0: 565a580ebeb0c567
00c6:
=> Send data, 370 bytes (0x172)
: --565a580ebeb0c567
002c: Content-Disposition: form-data; name="sid"
0058:
005a: 4e240384c30c909d
006c: --565a580ebeb0c567
0098: Content-Disposition: form-data; name="ImportExportPassword"
00d5:
00d7: backup
00df: --565a580ebeb0c567
010b: Content-Disposition: form-data; name="ConfigExport"
0140:
0142:
0144: --565a580ebeb0c567--
<= Recv header, 17 bytes (0x11)
: HTTP/1.1 200 OK


> See how it works comparing curl to:
> 
>$ env MOJO_CLIENT_DEBUG=1 mojo get -v -M POST -f sid=$SID -f
> ImportExportPassword=$BAKPWD -f ConfigExport= http
> ://fritz.box/cgi-bin/firmwarecfg 

The empty ConfigExport parameter gets lost with this command and the
Content-Type is different:

-- Blocking request (http://fritz.box/cgi-bin/firmwarecfg)
-- Connect 7c84d2d7396d9d2ff9571982f73240cb (http://fritz.box:80)
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
Content-Length: 48\x0d
User-Agent: Mojolicious (Perl)\x0d
Accept-Encoding: gzip\x0d
Content-Type: application/x-www-form-urlencoded\x0d
Host: fritz.box\x0d
\x0d
ImportExportPassword=backup&sid=4e240384c30c909d
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

-- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)
HTTP/1.1 200 OK\x0d



With DEBUG on and my code (s. my original message and below) I get this:

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
Accept-Encoding: gzip, deflate\x0d
Content-Type: multipart/form-data; boundary=tyUPX\x0d
Accept: */*\x0d
Host: fritz.box\x0d
Content-Length: 230\x0d
User-Agent: Mojolicious (Perl)\x0d
\x0d
--tyUPX\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
Content-Disposition: form-data; name="ConfigExport"\x0d
\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
\x0d
--tyUPX\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
Content-Disposition: form-data; name="ImportExportPassword"\x0d
\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
backup
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
\x0d
--tyUPX\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
Content-Disposition: form-data; name="sid"\x0d
\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
33cb272bcd34940b
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
\x0d
--tyUPX--\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

-- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)
HTTP/1.1 200 OK\x0d

which for me looks very much the same as the curl version but as I said,
curl gives the backup file whereas M::U just gives the HTML page back,
indicating that something went wrong.
I can only spot very minor differences that is why I am running out of
ideas:
- the boundary used by M::U is quite short
- the form parameters are not in the same order
- Accept-Encoding header (is also sent from Firefox where it works)
- the different SID is normal (my script always makes a fresch login)

Looks very strange to me
-Michael

> On Wed, May 13, 2020, 4:56 PM 'Michael Lackhoff' via Mojolicious <
> mojolicious@googlegroups.com> wrote:
> 
>> I had quite a bit of success recently using Mojo::UserAgent so I tried to
>> replace a curl command to do a backup of my Fritz.box router with M::U.
>>
>> Here is the curl command:
>> curl -s -k -o $OUT --form sid=$SID --form ImportExportPassword=$BAKPWD \
>> --form ConfigExport= http://fritz.box/cgi-bin/firmwarecfg
>>
>> It should be equivalent to this M::U request:
>>
>> my $tx = $ua->build_tx(
>> POST => 'http://fritz.box/cgi-bin/firmwarecfg' =>
>> {
>> 'Accept'   => '*/*',
>> 'Content-Type' => 'multipart/form-data',
>> } => form => {
>> sid

Re: [Mojolicious] curl works, Mojo::UserAgent doesn't

[Stefan Adams]

I'm certain you'll be able to accomplish this with M::UA. Can you share
what the HTML error produced is?

Can you share the output that you are comparing that you are noticing is so
similar between curl and M::UA? You're using -v for curl. Also set the
MOJO_CLIENT_DEBUG env variable to 1.

See how it works comparing curl to:

   $ env MOJO_CLIENT_DEBUG=1 mojo get -v -M POST -f sid=$SID -f
ImportExportPassword=$BAKPWD -f ConfigExport= http
://fritz.box/cgi-bin/firmwarecfg 

On Wed, May 13, 2020, 4:56 PM 'Michael Lackhoff' via Mojolicious <
mojolicious@googlegroups.com> wrote:

> I had quite a bit of success recently using Mojo::UserAgent so I tried to
> replace a curl command to do a backup of my Fritz.box router with M::U.
>
> Here is the curl command:
> curl -s -k -o $OUT --form sid=$SID --form ImportExportPassword=$BAKPWD \
> --form ConfigExport= http://fritz.box/cgi-bin/firmwarecfg
>
> It should be equivalent to this M::U request:
>
> my $tx = $ua->build_tx(
> POST => 'http://fritz.box/cgi-bin/firmwarecfg' =>
> {
> 'Accept'   => '*/*',
> 'Content-Type' => 'multipart/form-data',
> } => form => {
> sid  => $SID,
> ImportExportPassword => $BAKPWD,
> ConfigExport => '',
> }
> );
>
> # for debugging:
> print $tx->req->to_string;
>
> $tx = $ua->start($tx);
> $tx->res->save_to($OUT);
>
> As far as I can tell both the headers and the POST body is very much the
> same (except the boundary value to separate the form fields) but to my
> surprise the curl command works ($OUT is the backup file) but with the M::U
> version $OUT consists of some HTML output indicating an error.
>
> If I could see a difference I could try to better adjust my script but as
> I said, they look very much the same (I compared it with the -v and
> --trace-ascii output of curl), so I run out of ideas what could trigger the
> differnt response of my Fritz.box.
> Any ideas? At the moment I just solve it by using the curl command with
> "system" but I would prefer a Perl-only solution and what is even more
> important to me: I want to understand what is going on here.
>
> -Michael
>
> --
> You received this message because you are subscribed to the Google Groups
> "Mojolicious" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mojolicious+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mojolicious/14df452c-e7a4-4b7c-90a2-9e1f5e15becc%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/mojolicious/CACyQ%2BFR1F7fqH3%3DFJCMftHuwH1rEKwyPZ%2BO0g7d%3DJcRaWYqDtw%40mail.gmail.com.

[Mojolicious] curl works, Mojo::UserAgent doesn't

['Michael Lackhoff' via Mojolicious]

I had quite a bit of success recently using Mojo::UserAgent so I tried to 
replace a curl command to do a backup of my Fritz.box router with M::U.

Here is the curl command:
curl -s -k -o $OUT --form sid=$SID --form ImportExportPassword=$BAKPWD \
--form ConfigExport= http://fritz.box/cgi-bin/firmwarecfg

It should be equivalent to this M::U request:

my $tx = $ua->build_tx(
POST => 'http://fritz.box/cgi-bin/firmwarecfg' =>
{
'Accept'   => '*/*',
'Content-Type' => 'multipart/form-data',
} => form => {
sid  => $SID,
ImportExportPassword => $BAKPWD,
ConfigExport => '',
}
);

# for debugging:
print $tx->req->to_string;

$tx = $ua->start($tx);
$tx->res->save_to($OUT);

As far as I can tell both the headers and the POST body is very much the 
same (except the boundary value to separate the form fields) but to my 
surprise the curl command works ($OUT is the backup file) but with the M::U 
version $OUT consists of some HTML output indicating an error.

If I could see a difference I could try to better adjust my script but as I 
said, they look very much the same (I compared it with the -v and 
--trace-ascii output of curl), so I run out of ideas what could trigger the 
differnt response of my Fritz.box.
Any ideas? At the moment I just solve it by using the curl command with 
"system" but I would prefer a Perl-only solution and what is even more 
important to me: I want to understand what is going on here.

-Michael

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/mojolicious/14df452c-e7a4-4b7c-90a2-9e1f5e15becc%40googlegroups.com.