Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Stefan Adams]

On Wed, Oct 31, 2018 at 9:34 PM Sylvain Thibault 
wrote:

> As suggested, here is a working solution using a Role::Tiny role to
> modify/override Mojo::Parameters::to_string so that it does not escape the
> caret '^' character.
> Feel free to comment/criticize.
>

That looks terrific to me and pretty much exactly what I'd do!


> Thank you all for your valuable input, this was enlightening.
>

Heck yeah!  I learned an important lesson myself!  Thanks for your
question!

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Sylvain Thibault]

As suggested, here is a working solution using a Role::Tiny role to 
modify/override Mojo::Parameters::to_string so that it does not escape the 
caret '^' character.
Feel free to comment/criticize.

Thank you all for your valuable input, this was enlightening.

use Mojo::UserAgent;
use FindBin qw($Bin);
use feature qw(say);
use lib $Bin;

$| = 1;

# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);

my $url_raw
  = 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
^GSPC=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0'
;

# Build a normal transaction
my $tx = $ua->build_tx(GET => $url_raw, => {Accept => '*/*'});

# Bypass caret '^' encoding with a Role::Tiny role as Yahoo finance servers 
are not RFC 3986 compliant yet...
$tx->req->url->query->with_roles('+Unescaped');

say $tx->req->to_string;

# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
  read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
  }
);

# Process transaction
$tx = $ua->start($tx);

say "done";


And the file ./Mojo/Parameters/Role/Unescaped.pm to modify/override 
Mojo::Parameters::to_string

package Mojo::Parameters::Role::Unescaped;

use Mojo::Base -role;

use Mojo::Util qw(decode encode url_escape url_unescape);

has charset => 'UTF-8';

# Override default Mojo::Parameters::to_string method
sub to_string {
  my $self = shift;
  # String (RFC 3986)
  my $charset = $self->charset;
  if (defined(my $str = $self->{string})) {
$str = encode $charset, $str if $charset;
# Do not encode caret '^' character
return url_escape $str, '^A-Za-z0-9\-._~%!$&\'()*+,;=:@/?i^';
  }
}

1;



On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired 
> output:
>
> GET /streamer/1.0?s=^GSPC=l86,l84,p20 HTTP/1.1
> Host: s omehost.com
> User-Agent: curl/7.61.1
> Accept: */*
>
> How does one send a caret '^' in a URL without encoding it to %5E ?
>
> Using a symbol without a caret works great in the Mojo::UserAgent version 
> of the code.
>
> Thanks,
>
> Sylvain Thibault 
>

On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired 
> output:
>
> GET /streamer/1.0?s=^GSPC=l86,l84,p20 HTTP/1.1
> Host: s omehost.com
> User-Agent: curl/7.61.1
> Accept: */*
>
> How does one send a caret '^' in a URL without encoding it to %5E ?
>
> Using a symbol without a caret works great in the Mojo::UserAgent version 
> of the code.
>
> Thanks,
>
> Sylvain Thibault 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Sylvain Thibault]

As suggested, here is a working solution using a Role::Tiny role to modify 
Mojo::Parameters::to_string so that it does not escape the caret '^' 
character.
Feel free to comment/criticize.

Thank you all for you valuable input, this was enlightening.

use Mojo::UserAgent;
use FindBin qw($Bin);
use feature qw(say);
use lib $Bin;

$| = 1;

# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);

my $url_raw
  = 
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=^GSPC=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0'
;

# Build a normal transaction
my $tx = $ua->build_tx(GET => $url_raw, => {Accept => '*/*'});

# Bypass caret '^' encoding as Yahoo finance servers are not  RFC 3986 
compliant yet...
$tx->req->url->query->with_roles('+Unescaped');

say $tx->req->to_string;

# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
  read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
  }
);

# Process transaction
$tx = $ua->start($tx);

say "done";


And the file ./Mojo/Parameters/Role/Unescaped.pm to modify/override 
Mojo::Parameters::to_string

package Mojo::Parameters::Role::Unescaped;


use Mojo::Base -role;


use Mojo::Util qw(decode encode url_escape url_unescape);


has charset => 'UTF-8';


sub to_string {
  my $self = shift;


  # String (RFC 3986)
  my $charset = $self->charset;
  if (defined(my $str = $self->{string})) {
$str = encode $charset, $str if $charset;
return url_escape $str, '^A-Za-z0-9\-._~%!$&\'()*+,;=:@/?i^';
  }
}




1;



On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired 
> output:
>
> GET /streamer/1.0?s=^GSPC=l86,l84,p20 HTTP/1.1
> Host: s omehost.com
> User-Agent: curl/7.61.1
> Accept: */*
>
> How does one send a caret '^' in a URL without encoding it to %5E ?
>
> Using a symbol without a caret works great in the Mojo::UserAgent version 
> of the code.
>
> Thanks,
>
> Sylvain Thibault 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Sylvain Thibault]

Can somebody describe with a little more detail how one would go about 
using a role to modify Mojo::Parameters::to_string
and using it with something like 
$tx->req->url->query->with_roles('+Unescaped') ?

I am looking at Joel Berger's blog 
post: https://mojolicious.io/blog/2017/12/13/day-13-more-about-roles/
but not quite getting there.

On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired 
> output:
>
> GET /streamer/1.0?s=^GSPC=l86,l84,p20 HTTP/1.1
> Host: s omehost.com
> User-Agent: curl/7.61.1
> Accept: */*
>
> How does one send a caret '^' in a URL without encoding it to %5E ?
>
> Using a symbol without a caret works great in the Mojo::UserAgent version 
> of the code.
>
> Thanks,
>
> Sylvain Thibault 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Stefan Adams]

On Wed, Oct 31, 2018 at 12:19 PM Dan Book  wrote:

> Mojo::Collection and Mojo::File are not hash-based objects so they cannot
> use Mojo::Base, thus they implement with_roles themselves. Mojo::Base based
> objects all say "inherits all methods from Mojo::Base" or a subclass of
> that.
>

This is really helpful!  Thank you for clearing that up!

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Stefan Adams]

On Wed, Oct 31, 2018 at 10:48 AM sri  wrote:

> Poking into internals usually means one day you'll get what you asked for,
>> but if there is no other way...
>>
>
> Yes, that is a very dangerous hack, a more sane approach would be to use a
> Role::Tiny role modifying
> Mojo::Parameters::to_string
> ($tx->req->url->query->with_roles('+Unescaped')) or something similar.
>

Yes!  I love this strategy!  Existing packages like Mojo::Collection and
Mojo::File already have with_roles available, but Mojo::Parameters does
not.  Does it make sense to add that, or is it really that simple to just
add it independently and the reason why with_roles is available to these
other packages is just for convenience due to the overwhelming likelihood
that they will be used more frequently?

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[sri]

>
> I do not know of a non-ugly way either but if you are okay with a hacky 
> and possibly fragile solution you might get away with adding:
>
>  $ua->on(start => sub {
>my ($ua, $tx) = @_;
>$tx->req->{'start_buffer'} =~ s/([&?]s=)%5E/$1^/o;
>  });
>
> Poking into internals usually means one day you'll get what you asked for, 
> but if there is no other way...
>

Yes, that is a very dangerous hack, a more sane approach would be to use a 
Role::Tiny role modifying
Mojo::Parameters::to_string 
($tx->req->url->query->with_roles('+Unescaped')) or something similar.

--
sebastian

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Sylvain Thibault]

Thank you Heiko,
This works for me.
This case is definitely an outlier.

On Wednesday, October 31, 2018 at 5:09:43 AM UTC-6, Heiko Jansen wrote:
>
> I do not know of a non-ugly way either but if you are okay with a hacky 
> and possibly fragile solution you might get away with adding:
>
>  $ua->on(start => sub {
>my ($ua, $tx) = @_;
>$tx->req->{'start_buffer'} =~ s/([&?]s=)%5E/$1^/o;
>  });
>
> Poking into internals usually means one day you'll get what you asked for, 
> but if there is no other way...
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Heiko Jansen]

I do not know of a non-ugly way either but if you are okay with a hacky and 
possibly fragile solution you might get away with adding:

 $ua->on(start => sub {
   my ($ua, $tx) = @_;
   $tx->req->{'start_buffer'} =~ s/([&?]s=)%5E/$1^/o;
 });

Poking into internals usually means one day you'll get what you asked for, 
but if there is no other way...

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Sylvain Thibault]

Thanks for that.
You would think Yahoo would be RFC 3986 compliant but their services have 
been on the decline for a while.

I don't know how to override Mojo::Util url_escape either.

On Sunday, October 28, 2018 at 12:46:36 PM UTC-6, Stefan Adams wrote:
>
> I see, you are building a client app, not a server app -- I missed that 
> detail.
>
> And, I see that you want to pass the ^ *without* escaping it.
>
> Your URL is getting parsed by Mojo::URL, whose parameters are getting 
> parsed by Mojo::Parameters, and when you get the parameters as a string 
> , url_escape 
>  is being applied:
>
> Percent encode unsafe characters in string as described in RFC 3986 
> , the pattern used defaults to 
> ^A-Za-z0-9\-._~.
>
>
> Reading the code for to_string in Mojo::Parameters, I see no way around 
> this.  It would seem that Yahoo Finance is not RFC 3986 compliant.  The 
> question that needs to be asked then is, "Is it possible for Mojo::URL to 
> customize the characters that are considered unsafe?"  I don't know super 
> advanced Perl like this, but maybe it's possible to override a function and 
> make Mojo::Util::url_escape not actually escape the string?
>
> There might be a more advanced way to build your transaction, one that 
> doesn't operate on a Mojo::URL object which will ultimately apply 
> url_escape without question -- but I wouldn't know it.
>
> Anyway, from what I can see, there's not a solution to your problem.  :(
>
>
> On Sun, Oct 28, 2018 at 11:49 AM Sylvain Thibault  > wrote:
>
>> Thank you for your response Stefan.
>> Here is the complete code.  Symbol SPY works, symbol ^GSPC returns not 
>> found.
>>
>> This curl command with the ^GSPC symbol works:
>>
>> curl -s -o - -N -v 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
>> ^GSPC=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0'
>>
>>
>> #! /usr/bin/env perl
>> use Mojo::UserAgent;
>> use feature qw(say);
>>
>> $| = 1;
>>
>> # Accept responses of indefinite size
>> my $ua = Mojo::UserAgent->new(max_response_size => 0);
>> $ua->inactivity_timeout(0);
>>
>> my $url_raw
>> # Symbol ^GSPC returns NOT FOUND
>> #  = 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
>> ^GSPC=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0';
>>   = '
>> https://streamerapi.finance.yahoo.com/streamer/1.0?s=SPY=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0
>> ';
>>
>>
>> # Build a normal transaction
>> my $tx = $ua->build_tx(
>>   GET => $url_raw,
>>   => {Accept => '*/*'}
>> );
>>
>> # Remove caret encoding... results is not found... ???
>> say $tx->req->to_string;
>>
>> # Replace "read" events to disable default content parser
>> $tx->res->content->unsubscribe('read')->on(
>>   read => sub {
>> my ($content, $bytes) = @_;
>> say "Streaming: $bytes";
>>   }
>> );
>>
>> # Process transaction
>> $tx = $ua->start($tx);
>>
>>
>> say "done";
>>
>>
>>
>> On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>>>
>>> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>>>
>>> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
>>> The server returns NOT FOUND.
>>>
>>> So this transaction from Mojo::UserAgent returns NOT FOUND:
>>>
>>> GET /streamer/1.0?s=%5EGSPC=l86,l84,p20 HTTP/1.1
>>> Host: somehost.com
>>> Accept: */*
>>> User-Agent: Mojolicious (Perl)
>>> Content-Length: 0
>>>
>>> This transaction using curl with the caret not encoded returns the 
>>> desired output:
>>>
>>> GET /streamer/1.0?s=^GSPC=l86,l84,p20 HTTP/1.1
>>> Host: s omehost.com
>>> User-Agent: curl/7.61.1
>>> Accept: */*
>>>
>>> How does one send a caret '^' in a URL without encoding it to %5E ?
>>>
>>> Using a symbol without a caret works great in the Mojo::UserAgent 
>>> version of the code.
>>>
>>> Thanks,
>>>
>>> Sylvain Thibault 
>>>
>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Mojolicious" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to mojolicious...@googlegroups.com .
>> To post to this group, send email to mojol...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/mojolicious.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Stefan Adams]

I see, you are building a client app, not a server app -- I missed that
detail.

And, I see that you want to pass the ^ *without* escaping it.

Your URL is getting parsed by Mojo::URL, whose parameters are getting
parsed by Mojo::Parameters, and when you get the parameters as a string
, url_escape
 is being applied:

Percent encode unsafe characters in string as described in RFC 3986
, the pattern used defaults to
^A-Za-z0-9\-._~.


Reading the code for to_string in Mojo::Parameters, I see no way around
this.  It would seem that Yahoo Finance is not RFC 3986 compliant.  The
question that needs to be asked then is, "Is it possible for Mojo::URL to
customize the characters that are considered unsafe?"  I don't know super
advanced Perl like this, but maybe it's possible to override a function and
make Mojo::Util::url_escape not actually escape the string?

There might be a more advanced way to build your transaction, one that
doesn't operate on a Mojo::URL object which will ultimately apply
url_escape without question -- but I wouldn't know it.

Anyway, from what I can see, there's not a solution to your problem.  :(


On Sun, Oct 28, 2018 at 11:49 AM Sylvain Thibault 
wrote:

> Thank you for your response Stefan.
> Here is the complete code.  Symbol SPY works, symbol ^GSPC returns not
> found.
>
> This curl command with the ^GSPC symbol works:
>
> curl -s -o - -N -v 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
> ^GSPC=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0'
>
>
> #! /usr/bin/env perl
> use Mojo::UserAgent;
> use feature qw(say);
>
> $| = 1;
>
> # Accept responses of indefinite size
> my $ua = Mojo::UserAgent->new(max_response_size => 0);
> $ua->inactivity_timeout(0);
>
> my $url_raw
> # Symbol ^GSPC returns NOT FOUND
> #  = 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
> ^GSPC=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0';
>   = '
> https://streamerapi.finance.yahoo.com/streamer/1.0?s=SPY=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0
> ';
>
>
> # Build a normal transaction
> my $tx = $ua->build_tx(
>   GET => $url_raw,
>   => {Accept => '*/*'}
> );
>
> # Remove caret encoding... results is not found... ???
> say $tx->req->to_string;
>
> # Replace "read" events to disable default content parser
> $tx->res->content->unsubscribe('read')->on(
>   read => sub {
> my ($content, $bytes) = @_;
> say "Streaming: $bytes";
>   }
> );
>
> # Process transaction
> $tx = $ua->start($tx);
>
>
> say "done";
>
>
>
> On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>>
>> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>>
>> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
>> The server returns NOT FOUND.
>>
>> So this transaction from Mojo::UserAgent returns NOT FOUND:
>>
>> GET /streamer/1.0?s=%5EGSPC=l86,l84,p20 HTTP/1.1
>> Host: somehost.com
>> Accept: */*
>> User-Agent: Mojolicious (Perl)
>> Content-Length: 0
>>
>> This transaction using curl with the caret not encoded returns the
>> desired output:
>>
>> GET /streamer/1.0?s=^GSPC=l86,l84,p20 HTTP/1.1
>> Host: s omehost.com
>> User-Agent: curl/7.61.1
>> Accept: */*
>>
>> How does one send a caret '^' in a URL without encoding it to %5E ?
>>
>> Using a symbol without a caret works great in the Mojo::UserAgent version
>> of the code.
>>
>> Thanks,
>>
>> Sylvain Thibault
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Mojolicious" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mojolicious+unsubscr...@googlegroups.com.
> To post to this group, send email to mojolicious@googlegroups.com.
> Visit this group at https://groups.google.com/group/mojolicious.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded

[Sylvain Thibault]

Thank you for your response Stefan.
Here is the complete code.  Symbol SPY works, symbol ^GSPC returns not found
.

This curl command with the ^GSPC symbol works:

curl -s -o - -N -v 
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=^GSPC=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0'


#! /usr/bin/env perl
use Mojo::UserAgent;
use feature qw(say);

$| = 1;

# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);

my $url_raw
# Symbol ^GSPC returns NOT FOUND
#  = 
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=^GSPC=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0';
  = 
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=SPY=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0'
;


# Build a normal transaction
my $tx = $ua->build_tx(
  GET => $url_raw,
  => {Accept => '*/*'}
);

# Remove caret encoding... results is not found... ???
say $tx->req->to_string;

# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
  read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
  }
);

# Process transaction
$tx = $ua->start($tx);


say "done";



On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired 
> output:
>
> GET /streamer/1.0?s=^GSPC=l86,l84,p20 HTTP/1.1
> Host: s omehost.com
> User-Agent: curl/7.61.1
> Accept: */*
>
> How does one send a caret '^' in a URL without encoding it to %5E ?
>
> Using a symbol without a caret works great in the Mojo::UserAgent version 
> of the code.
>
> Thanks,
>
> Sylvain Thibault 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.