Paul Kulchenko <[EMAIL PROTECTED]> writes:

> Hi, Gisle!

Hi, Paul!

> --- Gisle Aas <[EMAIL PROTECTED]> wrote:
> > I my view it is a bug to put content containing chars with ord() > 255
> > in the the content of a HTTP::Request.  If you want UTF8 encoded stuff
> > you should put UTF8 encoded stuff in the content.  Don't expect perl
> > to magically guess.  You should use Encode::encode_utf8($str) or
> > something like it.
> I'm not sure I follow you. I do have my string utf8 encoded using
> Perl capabilities. What do you mean "is a bug to put content
> containing chars with ord() 255 in the the content of a
> HTTP::Request"? What should I do then if I have utf8 encoded string
> to send? I expect LWP will handle it properly on wire.

If you have a utf8 encoded string then none of the chars in the string
will have ord() > 255.  It is the kind of string that
Encode::encode_utf8() would produce.

What you want is LWP to deal with strings with the _internal_ UTF8
flag set.  In my view LWP can't guess what encoding to apply to
serialize that kind of string.  UTF-8 is not really a more obvious
choice than UTF-16.  If it happens to be an image that somehow got the
UTF8 flag set then any UTF-encoding would be wrong, as the string
should simply be utf8_downgraded to be ok again.

> > If there was an easy way I would like to add a
> >   sv_utf8_downgrade($req->content, 0);
> > to the LWP::Protocol code.  This would make requests with such
> > chars in them fail early.  
> I don't understand why they should be failed. What's wrong with this:
> $utf8 = pack('U*', unpack('C*', $something_russian_latin1_encoded));
> $req = HTTP::Request
>   ->new(POST => $endpoint, HTTP::Headers->new, $utf8);
> $resp = LWP::UserAgent->new->request($req);
> request won't be properly encoded in 5.6.1 and later unless you drop
> utf8 mark from $utf8. I do need to have utf8 encoding on wire. 
> What do you expect me to do?

If you want the string UTF8 encoded, then say so explicitly:

 $req = HTTP::Request->new(POST => $endpoint);
 $req->content_type("text/plain; charset='utf8'");

> > > # drop UTF mark
> > > $str = pack('C0A*', $str) if length($str) != bytelength($str);
> > > 
> > > Ideally I would like to have it fixed in LWP::Protocol. btw, how
> > > quick is pack 'C0A*'?
> > 
> > It will certainly have to copy the string.  I think it would be
> > better to try to use one of functions the Encode module provides.
> I need it to work with all Perls starting 5.005. Encode wasn't
> available in 5.6.x, was it?

That is true.  How about something like this (untested);

   eval { require Encode; };
   if ($@) {
      # replacement
      *Encode::encode_utf8 = sub { pack('C0A*', shift) };     

Actually, I think pack is buggy if this works.  The A* really ought to
downgrade the string it packs and croak if this is not possible.

Reply via email to