This problem is probably StringContent.ToByteArray(). It uses Encoding.GetBytes(string) which obeys .NET encoding semantics and by default replaces any nonconvertible characters to '?'. And then MutableStringOps.Dump() is using it to create string representation. We could make StringContent.ToByteArray() not replacing nonconvertible characters by using EncodingFallback. BinaryContent.ToString()/ToStringBuilder() also has the same issue.
-- Oleg From: Curt Hagenlocher Sent: Monday, July 14, 2008 6:27 AM To: Michael Letterle; [email protected] Cc: IronRuby Team Subject: RE: [Ironruby-core] MutableString encoding issue MutableString can have one of three internal representations, depending on how it was last used. One of these is a byte array. This particular problem may be in the scanner or parser and not in the actual string class, as we don't otherwise have a problem storing the character: >>> $s = "\204" => "?" >>> $s[0] => 63 >>> $s[0] = 132 => 132 >>> $s => "\204" >>> From: Michael Letterle [mailto:[EMAIL PROTECTED] Sent: Monday, July 14, 2008 6:21 AM To: [email protected] Cc: IronRuby Team Subject: Re: [Ironruby-core] MutableString encoding issue This was a known issue a while back, it's the reason the Zlib library didn't work well with binary files. I'm fairly certain there was work being done on making String be backed by a byte array... and in fact I thought this was already done. On Mon, Jul 14, 2008 at 2:20 AM, Oleg Tkachenko <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote: Stumbled on this when testing yaml. >>> "\204" => "?" While irb(main):005:0>"\204" => "\204" I believe Ruby string can hold arbitrary byte values, but as we are storing content as a string we are obviously losing all values that cannot be represented in default encoding. Tomas, what do you think? -- Oleg _______________________________________________ Ironruby-core mailing list [email protected]<mailto:[email protected]> http://rubyforge.org/mailman/listinfo/ironruby-core -- Michael Letterle [Polymath Prokrammer] http://blog.prokrams.com
_______________________________________________ Ironruby-core mailing list [email protected] http://rubyforge.org/mailman/listinfo/ironruby-core
