Excerpts from Adeel Ahmad Khan's message of 2011-01-31 19:24:42 +0100: > > Tero Tilus <t...@tilus.net>: > > Adeel Ahmad Khan, 2011-01-29 05:11: > > > invalid byte sequence in UTF-8 > > ... > > > I am using a nearly fresh installation of Sup 0.12.1 with Ruby > > > 1.9.2p136. I have LOCALE="en_US.UTF-8".
What editor are you using ? Could you attach some buggy text (make sure it is transferred in binary - ie upload an archive somewhere like http://paste.xinu.at/)? > > Both the errors were from regex matches against message body. Somehow > > your editor doesn't know your locale or is not obeying it. As a > > result non-utf8 stuff gets saved to disk and sup gets confused. > > You were right. It turned out to be an issue with editor. I'm not so sure. I think it is a problem with Ruby (1.9) degrading the the text it reads to US-ASCII (8bit). And when you later try to do UTF-8 stuff with it (append, or regexp with UTF-8) it fails. Perhaps even when trying to access chars in the string (the body) that are not really US-ASCII [2]. What my patch does is tell Ruby that the string is an UTF-8 string, no matter what it deduced from reading the file in the first place - or what might have happened throughout Sup's processing of the text. Try this patch, it forces encoding on the entire body: http://ix.io/1rO (these patches are workarounds; not to be applied to source-tree) The same happened with labels.txt (or with contacts). Whenever a label string could be degraded to US-ASCII, Ruby did so (how should it know it was UTF-8 anyway?), then when trying to append, match or work with the US-ASCII string towards UTF-8 input it failed. What I think must be done (as an alternative to supporting different encodings all the way) - is to _always_ read all files in UTF-8 [1] (or transcode to UTF-8), and perhaps most difficultly _keep_ the strings UTF-8 throught the entire Sup processes. [1] http://blog.grayproductions.net/articles/ruby_19s_three_default_encodings [2] http://www.ruby-forum.com/topic/194493 Best regards, Gaute _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel
