Re: [Pharo-dev] Zinc redirect bug?

[Sven Van Caekenberghe]

> On 14 Jul 2017, at 21:12, Sven Van Caekenberghe  wrote:
> 
> Hi Aliaksei,
> 
> This is an interesting problem, thanks for reporting it. I can reproduce the 
> problem.
> 
> If you inspect 'ZnLogEvent announcer' before doing the request, you can see 
> that ZnClient does seem to do the same as 'curl -vOL 
> https://dl.bintray.com/syrel/Moz2D/osx/development/i386/libMoz2D.dylib'. It 
> follows one redirect, to the correct host and URL.
> 
> One difference that I see that might lead to the 403 Forbidden is that Zn has 
> a slight difference in the redirect URL being followed.
> 
> Curl seems to follow
> 
> /86/866a31722e3a9ea19c2bc4eeed2e1e69f7339ddd460500f2a33b4c9d041ef9b4?__gda__=exp=1500058371~hmac=820d22695ef0a821abc7e6eb7be6749ad43e8e0f74ba4b103476aa1b3663b381=attachment%3Bfilename%3D%22libMoz2D.dylib%22=application%2Fx-www-form-urlencoded=U2FsdGVkX19XiffCdVJdoADmObVHKjU4Wla5_WKFKHoNAvXQw8ponQIqWBsRUfNNDaULVXlXwB22y7xy0ls3DGkl700_9RxT-HtU92OHYdgKzSq9a6A7p5NIoMi4avCjPk_UCnVbt5U4_nxvkmKtuA
> 
> While Zn turns this (parses the URL and then renders it again) into
> 
> /86/866a31722e3a9ea19c2bc4eeed2e1e69f7339ddd460500f2a33b4c9d041ef9b4?response-content-type=application/x-www-form-urlencoded=attachment;filename%3D%22libMoz2D.dylib%22=U2FsdGVkX19XiffCdVJdoADmObVHKjU4Wla5_WKFKHoNAvXQw8ponQIqWBsRUfNNDaULVXlXwB22y7xy0ls3DGkl700_9RxT-HtU92OHYdgKzSq9a6A7p5NIoMi4avCjPk_UCnVbt5U4_nxvkmKtuA&__gda__=exp%3D1500058371~hmac%3D820d22695ef0a821abc7e6eb7be6749ad43e8e0f74ba4b103476aa1b3663b381
> 
> The difference is in the encoding of the __gda__ query parameter value. That 
> contains = characters. These get escaped by Zn (the %3D) but are left 
> untouched by curl. Maybe the server does not like that. I have to look deeper 
> into the spec to figure out if this is a bug or not. Since query parameters 
> look like key=value it seems odd to allow = in the value part.
> 
> Another possible issue might be the order of query parameters that got 
> changed.
> 
> I'll come back to you if I find out more.
> 
> Sven

By (temporarily) changing ZnResourceMetaUtils>>#writeQueryFields:with:on: to 
use #querySafeSet instead of #queryKeyValueSafeSet (so that $= no longer gets 
encoded), the request passes, i.e. the redirect is followed and returns 200 OK.

The next problem is the interpretation of the result ;-)

The returned resource is specified to have a content-type of 
application/x-www-form-urlencoded and Zn interprets it as such. However, that 
does not seem to be the case. Rather it seems to be binary data. Now, returning 
application/x-www-form-urlencoded data as response to a request is very weird, 
I have never seen that.

It seems as if browsers and curl just ignore the actual type and just dump the 
data in a file (content-disposition: attachment;filename="libMoz2D.dylib"). 
Where is such a behaviour described ?

Aliaksei, could you maybe upload something like an small JPG or PNG image using 
the same mechanism ? Then it is easier to test the correctness of the result.

Could it be that you uploaded libMoz2D.dylib incorrectly ? Because 
application/x-www-form-urlencoded is normally only used in the context of HTTP 
forms that POSTed.

>> On 14 Jul 2017, at 13:02, Aliaksei Syrel  wrote:
>> 
>> Hello,
>> 
>> I am trying to download a file from bintray with the help of ZnClient.
>> I'm doing the following:
>> 
>> ZnClient new
>>   systemPolicy;
>>   url: 
>> 'https://dl.bintray.com/syrel/Moz2D/osx/development/i386/libMoz2D.dylib';
>>   get;
>>   yourself
>> 
>> which results in Error 403.
>> 
>> At the same time wget works as expected:
>> 
>> wget https://dl.bintray.com/syrel/Moz2D/osx/development/i386/libMoz2D.dylib
>> 
>> Trying to open this url in a browser works too.
>> 
>> I tried to find out what may cause the error and it looks like it is 
>> something related to redirects because request and response headers look 
>> good...
>> 
>> Thanks,
>> Alex
> 

Re: [Pharo-dev] Zinc redirect bug?

[Sven Van Caekenberghe]

Hi Aliaksei,

This is an interesting problem, thanks for reporting it. I can reproduce the 
problem.

If you inspect 'ZnLogEvent announcer' before doing the request, you can see 
that ZnClient does seem to do the same as 'curl -vOL 
https://dl.bintray.com/syrel/Moz2D/osx/development/i386/libMoz2D.dylib'. It 
follows one redirect, to the correct host and URL.

One difference that I see that might lead to the 403 Forbidden is that Zn has a 
slight difference in the redirect URL being followed.

Curl seems to follow

/86/866a31722e3a9ea19c2bc4eeed2e1e69f7339ddd460500f2a33b4c9d041ef9b4?__gda__=exp=1500058371~hmac=820d22695ef0a821abc7e6eb7be6749ad43e8e0f74ba4b103476aa1b3663b381=attachment%3Bfilename%3D%22libMoz2D.dylib%22=application%2Fx-www-form-urlencoded=U2FsdGVkX19XiffCdVJdoADmObVHKjU4Wla5_WKFKHoNAvXQw8ponQIqWBsRUfNNDaULVXlXwB22y7xy0ls3DGkl700_9RxT-HtU92OHYdgKzSq9a6A7p5NIoMi4avCjPk_UCnVbt5U4_nxvkmKtuA

While Zn turns this (parses the URL and then renders it again) into

/86/866a31722e3a9ea19c2bc4eeed2e1e69f7339ddd460500f2a33b4c9d041ef9b4?response-content-type=application/x-www-form-urlencoded=attachment;filename%3D%22libMoz2D.dylib%22=U2FsdGVkX19XiffCdVJdoADmObVHKjU4Wla5_WKFKHoNAvXQw8ponQIqWBsRUfNNDaULVXlXwB22y7xy0ls3DGkl700_9RxT-HtU92OHYdgKzSq9a6A7p5NIoMi4avCjPk_UCnVbt5U4_nxvkmKtuA&__gda__=exp%3D1500058371~hmac%3D820d22695ef0a821abc7e6eb7be6749ad43e8e0f74ba4b103476aa1b3663b381

The difference is in the encoding of the __gda__ query parameter value. That 
contains = characters. These get escaped by Zn (the %3D) but are left untouched 
by curl. Maybe the server does not like that. I have to look deeper into the 
spec to figure out if this is a bug or not. Since query parameters look like 
key=value it seems odd to allow = in the value part.

Another possible issue might be the order of query parameters that got changed.

I'll come back to you if I find out more.

Sven

> On 14 Jul 2017, at 13:02, Aliaksei Syrel  wrote:
> 
> Hello,
> 
> I am trying to download a file from bintray with the help of ZnClient.
> I'm doing the following:
> 
> ZnClient new
>systemPolicy;
>url: 
> 'https://dl.bintray.com/syrel/Moz2D/osx/development/i386/libMoz2D.dylib';
>get;
>yourself
> 
> which results in Error 403.
> 
> At the same time wget works as expected:
> 
> wget https://dl.bintray.com/syrel/Moz2D/osx/development/i386/libMoz2D.dylib
> 
> Trying to open this url in a browser works too.
> 
> I tried to find out what may cause the error and it looks like it is 
> something related to redirects because request and response headers look 
> good...
> 
> Thanks,
> Alex

Re: [Pharo-dev] Zinc redirect bug?

[Tim Mackinnon]

I’m not sure if this is related - but the Pharo6 image is missing a ZnConstant 
initialise, and I recall redirect was one of the error codes missing. Possibly 
re-initialising ZnConstants might fix your problem?

Tim

> On 14 Jul 2017, at 12:02, Aliaksei Syrel  wrote:
> 
> Hello,
> 
> I am trying to download a file from bintray with the help of ZnClient.
> I'm doing the following:
> 
> ZnClient new
>systemPolicy;
>url: 
> 'https://dl.bintray.com/syrel/Moz2D/osx/development/i386/libMoz2D.dylib 
> ';
>get;
>yourself
> 
> which results in Error 403.
> 
> At the same time wget works as expected:
> 
> wget https://dl.bintray.com/syrel/Moz2D/osx/development/i386/libMoz2D.dylib 
> 
> 
> Trying to open this url in a browser works too.
> 
> I tried to find out what may cause the error and it looks like it is 
> something related to redirects because request and response headers look 
> good...
> 
> Thanks,
> Alex

[Pharo-dev] Zinc redirect bug?

[Aliaksei Syrel]

Hello,

I am trying to download a file from *bintray* with the help of ZnClient.
I'm doing the following:

ZnClient new
   systemPolicy;
   url: '
https://dl.bintray.com/syrel/Moz2D/osx/development/i386/libMoz2D.dylib';
   get;
   yourself

which results in Error 403.

At the same time wget works as expected:

wget https://dl.bintray.com/syrel/Moz2D/osx/development/i386/libMoz2D.dylib

Trying to open this url in a browser works too.

I tried to find out what may cause the error and it looks like it is
something related to redirects because request and response headers look
good...

Thanks,
Alex