Hi Matt, sorry for the late reply, got busy :-)
On 28 March 2015 22:18 CET, Matt Gushee wrote: > > On Sat, Mar 28, 2015 at 5:33 AM, Moritz Heidkamp <[email protected] >> wrote: > >> >> ah, that's what you are referring to, I see! It's like that because I >> didn't want to force a utf8 dependency on the user. > > > Maybe in that case it would be good if the API doc said something like: > "comparse is compatible with UTF-8, but many of the built-in combinators do > not work with UTF-8 characters, so you may need to construct your own. For > example: ..." Sure, we can do that! I didn't mention it so far as this property is implicit with how CHICKEN core strings work. Also, note that Comparse can parse arbitrary inputs, not just character strings / sequences, e.g. you can use it to parse a list of symbols into a structure. No arbitrary limitations and all that :-) >> This doesn't work because `sake' is a string and Comparse operates on >> the byte level by default (in accordance with CHICKEN core string >> procedures). > > > Well, yes. And, though I understand that if you *know* your programs > only need to process single-byte characters, it is convenient (and > better performance-wise, though I wonder how much in the year 2015) to > equate characters with bytes. It's not quite that simple: Characters may be encoded in many ways, UTF-8 is far from the only widely used one and not ideal in all cases, e.g. the algorithmic complexity of some operations on UTF-8 encoded strings is objectively worse than those on UTF-32 encoded strings. And it will remain so even till the year 2050. Not guarantees on what happens after that, though! > I'm of the opinion (shared by many I18n experts, if I'm not mistaken) > that a high-level language in the 21st century should have in its core > a rock-solid character abstraction that is never, ever conflated with > a byte. The character abstraction actually is rock-solid even in CHICKEN 4 already: A character object represents a Unicode codepoint in an encoding independent way. > There are a lot of things I love about Chicken, but the (IMHO > obsolete) string implementation is not one of them. Yeah, strings being equivalent to u8vectors / blobs is a bit messy at times. I think this is something worth addressing in CHICKEN 5. It would be a rather invasive change, though, and so far nobody seems inclined to put in the effort. For the time being, it's safest to mainly think of strings as byte arrays, i.e. mentally (or actually, as the utf8 egg does internally) replace the `string-' prefix with a `byte-' prefix in the core string API. Whenever you work with actual strings, use the extension for the respective encoding you're dealing with (though currently the only fully supported option with CHICKEN 4 is UTF-8 via the utf8 egg AFAIK). >> We could create a comparse-utf8 egg to facilitate this. It's not >> currently on my agenda but I will put it in my Comparse notes for future >> reference. If you feel inclined to create one, I'm happy to provide you >> with code review and feedback! >> > > I was thinking about that. That'd be great! I would suggest to make it a separate egg so that we keep the utf8 dependency optional. To give an example of why such dependencies matter: Medea used to depend on the utf8 egg for some things but we removed that dependency when we wanted to package it up for a mobile application. The reason was that the utf8 egg bundles some extra case map files which we didn't need at all but would have complicated the packaging considerably. > I looked through the source code to see what string handling functions > are used that are not provided by the utf8 egg, and thus would need to > be reimplemented. So far I've found: > > substring/shared > string-kmp-partial-search > make-kmp-restart-vector > > Substring/shared is not too big a deal, but that KMP stuff is a bit > daunting. Maybe I'll look into it if I have time. I do like the comparse > API, and would like to be able to use it. Substring/shared is not a big deal, indeed, since it's essentially an alias of the regular substring procedure in CHICKEN 4 anyway. And as John pointed out, you don't really need to touch any of the KMP procedures. Thinking about it some more, I believe it would be necessary to expose the char-seq-cursor API to be able to provide a fully featured comparse-utf8 module. Let me know if that turns out to be the case -- I'd be happy to extract a comparse-lolevel module for that purpose :-) Moritz
signature.asc
Description: PGP signature
_______________________________________________ Chicken-users mailing list [email protected] https://lists.nongnu.org/mailman/listinfo/chicken-users
