[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Update of bug #15377 (project freeciv): Status: Ready For Test = Fixed Open/Closed:Open = Closed ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Update of bug #15377 (project freeciv): Planned Release: = 2.1.12, 2.2.1, 2.3.0 ___ Follow-up Comment #25: Version for S2_1 and S2_2 attached. Use Ulrik's comment. (file #8803, file #8804) ___ Additional Item Attachment: File name: trunk_fc_is_ascii.diff Size:5 KB File name: S2_2_S2_1_my_is_ascii.diff Size:5 KB ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #24, bug #15377 (project freeciv): Here it is. These functions are wrappers for the libc character class functions, without any locale-dependent behavior. The character functions work as documented for ASCII. Bytes outside of the ASCII set will not be reported to belong to any character class, and will be left unchanged by transformations. This behavior is safe but not strictly correct forsingle-byte 8-bit- or UTF-8 encoded text; in UTF-8, any byte that is part of a multibyte sequence is non-ASCII. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Meddelandet skickades via/av Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Update of bug #15377 (project freeciv): Status:None = Ready For Test Assigned to:None = pepeto ___ Follow-up Comment #23: I will attempt to commit Jason's patch file #8662. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Update of bug #15377 (project freeciv): Depends on: = bugs #15658 ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
Re: [Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Good, I wrote a brief explanation that you can include to explain the functions (see previous posting). Please include it if you find any value in it. ulrik Den 1 apr 2010 kl. 19.50 skrev pepeto NO-REPLY.INVALID- addr...@gna.org: Update of bug #15377 (project freeciv): Status:None = Ready For Test Assigned to:None = pepeto ___ Follow-up Comment #23: I will attempt to commit Jason's patch file #8662. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
Re: [Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Le jeudi 01 avril 2010 à 23:18 +0200, Ulrik Sverdrup a écrit : Good, I wrote a brief explanation that you can include to explain the functions (see previous posting). Please include it if you find any value in it. ulrik What are you talking about, I don't find such explanation... ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #17, bug #15377 (project freeciv): I have no opinion about what is the best to do... ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #18, bug #15377 (project freeciv): The best thing right now would be to check in jason's latest patch, at least for fc_isspace. I bet we can find many more bugs of this type (though not crashing bugs) everywhere where isspace is used right now, and nowhere were the current behavior makes sense. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Meddelandet skickades via/av Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #19, bug #15377 (project freeciv): I guess a is_utf8_string() and ut8_string_rm_truncated() (which doesn't require any extra lib) should be good to unsure to don't send incomplete strings thought the network, and maybe loading from files. This would also fix the crash described at bug #15658. And also the empty line bug as fixed in warclient at: http://svn.gna.org/viewcvs/freeciv-warclient?rev=1401view=rev ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #20, bug #15377 (project freeciv): The patch I sent should fix bug #15658 which appears to be the same as this one. However rather than that patch I think someone should take the time to write proper ascii versions of those functions, just in case we end up with a non-superset-of-ascii locale some day. But I guess that's unlikely so it shouldn't really matter. -jason ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message sent via/by Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #21, bug #15377 (project freeciv): Oh, what a mess this is: the freeciv server of course does not call setlocale anywhere, so it should be running in the C/POSIX locale and these functions should already work with ASCII only on the server side. We missed out on a lot of server bugs here I think (phew!). One argument for not renaming the restricted functions is that we would probably replace every single use of them with the ASCII version. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Meddelandet skickades via/av Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Update of bug #15377 (project freeciv): Operating System: Microsoft Windows = None ___ Follow-up Comment #22: Please ignore my previous comment. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Meddelandet skickades via/av Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Update of bug #15377 (project freeciv): Operating System:None = Microsoft Windows ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message sent via/by Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #10, bug #15377 (project freeciv): Actually I came to the exact same conclusion Ulrik: we should simply rewrite the character functions to work only on ascii (range 0 to 127) characters, and leave others alone. Using the system-provided functions (such as tolower) which work in the current locale makes no sense and never will make sense. This means some UTF8 characters wont get properly converted which will cause minor bugs (for instance it wont do proper case comparison on non-ascii letters in player/ruler names), but will remove a much larger set of bugs (for instance it may currently do wrong case comparison on those same non-ascii letters). As for making or using a UTF-8 variant for these, it's a bit more complicated than that. You can't have isspace() or tolower() or isalpha() functions in utf8 that go byte-by-byte. This means the same job you were doing via byte iteration over the string has to be done entirely differently, in every case. Additionally, some of the places these functions are called they are given utf-8, some they are given ascii, and in some they may be given latin1 or a different character set - so again it all requires careful auditing of the users in that case, of which there are a lot. If we go that route I'd rather rewrite all freeciv core to use ucs2 or ucs4 (fixed-width unicode) strings and impose that on these functions, and then we get type-checking out of it as well. This wouldn't be that hard but then we have to convert all data files (in utf-8) and all GUI strings (also in utf-8, for gtk2) on both input and output, everywhere, which is a lot of lines of change. It's probably overkill. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message sent via/by Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #11, bug #15377 (project freeciv): And, unfortunately limiting the functions to ascii-only is not going to work very well either for the is_ functions. is_alnum is most likely used for instance to check for valid ruler/city names and will fail on any non-ascii names now. Even if we add exceptions for any ruler/city names in the ruleset (probably already done) it's not desirable for any non-latin-alphabet players. Nonetheless here's the patch for it (very simple patch; someone might want to actually rewrite the functions to do the work directly for ascii). Out of curiosity, if I made a patch to switch freeciv's internal encoding (client, server, and network encoding) to UCS2/UCS4, would the change be supported by the current developers? Also as a side note, libc does include support for wide characters. However it's not defined what character set this actually is so we still have the current locale issues which is no good since we want to use the same character set at both client and server. -jason (file #8662) ___ Additional Item Attachment: File name: char.diff Size:2 KB ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message sent via/by Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #12, bug #15377 (project freeciv): Turning these functions into ASCII only is the only sane thing, but it might only be the first step. I don't think we check ruler names or anything against character classes, that would already today fail often! Likewise, we don't add new bugs by this change, for example matching player names with non-ASCII names already now requires exact case for the non-ascii part. (For example /ai RÖDSKÄGGE won't match player Rödskägge, but /ai RöDSKäGGE will. This behavior is preserved with this change.) About the patch: Why not document more exactly how we now define the functions more or less work with UTF-8 will leave people wondering. Here's my contribution: These functions are wrappers for the libc character class functions, without any locale-dependent behavior. The character functions work as documented for ASCII. Bytes outside of the ASCII set will not be reported to belong to any character class, and will only be transformed to themselves. This behavior is safe but not strictly correct forsingle-byte 8-bit- or UTF-8 encoded text; in UTF-8, any byte that is part of a multibyte sequence is non-ASCII. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Meddelandet skickades via/av Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #13, bug #15377 (project freeciv): We could look eagerly at GLib's support for Unicode/UTF-8, they have every function we would ever need: http://library.gnome.org/devel/glib/unstable/glib-Unicode-Manipulation.html That should already be available to the GTK+ client of course. But there must be some more lighter-weight library for Unicode + UTF-8 available. If I can amend the text in the previous message, say and will be left unchanged by transformations. instead of and will only be transformed to themselves. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Meddelandet skickades via/av Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #14, bug #15377 (project freeciv): It is not a real big deal. It just mean having 2 sets of functions for ASCII and UTF-8 (maybe with different prefixes). Without big changes, it can be improved a lot. e.g., fc_snprintf() just need to cut the string after the last complete character if returning -1. We could look eagerly at GLib's support for Unicode/UTF-8 The Glib support could also be a solution. Such code is already in use in the gtk2 client. Out of curiosity, if I made a patch to switch freeciv's internal encoding (client, server, and network encoding) to UCS2/UCS4, would the change be supported by the current developers? I am not sure to know how this work, but if someone explain to me. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #15, bug #15377 (project freeciv): See also bug #14205. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #16, bug #15377 (project freeciv): Having 2 sets of functions is trickier than it sounds. For one, you have to be extremely careful which you use where. For a second, neither one should really be using the libc character functions. And for a third, the functions we're talking about _cant_ be written for variable-width character sets like utf-8 so they all have to be replaced with string variants. I took a look at doing this but it's just way too many places and too wide a spectrum of the codebase affected to do so without upping the bug rate. Glib or libutf8 can provide some help on some of these functions, I do believe. We dont have glib as a requirement for server compilation however, though it wouldnt be a big deal to add it. To change the internal encoding to ucs4 I would introduce a new typedef, typedef uint32 fc_char into fciconv.h. The majority of internal strings then get changed from char[] to fc_char[]. String conversion is mostly done already via the fciconv or gettext functions, but what would have to be added is conversion of all data files read (from utf8/ascii into ucs4) and to all gui strings (again between ucs4 and utf8, both ways and in a lot of places). Perhaps the biggest challenge though would be server-generated strings, which can't use sprintf but instead will need a ucs4 version of the same (probably available through an external library). -jason ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message sent via/by Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #7, bug #15377 (project freeciv): If I understand UTF-8 correctly, tolower, isspace and similar functions are perfectly safe, as long as the functions work with only ASCII without any windows or otherwise 8-bit charsets. UTF-8 is ASCII clean. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Meddelandet skickades via/av Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #8, bug #15377 (project freeciv): As a follow up, my ASCII comments means that it is simple to write well-defined substitutions if we only care about the ascii part. For example, if we'd uppercase Perpinyà we would get PERPINYà etc. We can patch fc_tolower, fc_isspace etc to all return early if we have a byte outside [0, 127]. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Meddelandet skickades via/av Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Additional Item Attachment, bug #15377 (project freeciv): File name: ascii_str_funcs.diff Size:2 KB ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Meddelandet skickades via/av Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #9, bug #15377 (project freeciv): Isn't any utility in the standard C99 library to perform character operations like tolower on UTF-8 strings? ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #5, bug #15377 (project freeciv): Yep, that is surely correct. These functions check whether c, which must have the value of an unsigned char or EOF, falls into a certain character class according to the current locale. Each of the character function wrappers in support.[ch] works only on characters in the current locale, which in freeciv parlance is the local encoding, and generally not the same as UTF-8 at all. In other words, these functions cannot be used on nearly all freeciv strings without causing these problems. Replacing these is not an easy task. Their very nature is to operate byte-by-byte, but with freeciv using utf-8 internally this is not possible. One alternative used elsewhere is to convert the utf-8 string to a fixed-width encoding such as UCS2 (similar to utf-16) or UCS4 (similar to utf-32) which can then be operated on byte-by-byte, but since that is also not the current locale the functions will not work on those characters either. Having all of Freeciv use ucs4 internally could even be an option, discussed previously that would prevent all such problems as there could then be no confusing internal strings from current locale strings; however, this would require a LOT of work and has been dismissed in the past. Additionally there may be a few locations where the text is ascii and the character functions will and should be used. Maybe. I'll take a look at where these functions are used and what might be done about them. -jason ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message sent via/by Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #3, bug #15377 (project freeciv): The problem is that one byte of the last (multibyte) UTF-8 character of the city name gets stripped by remove_trailing_spaces(), because this function checks single bytes with isspace() and one byte of the last character in the city name means non-breakable space in the Windows character set and thus it gets stripped. So we would need a check for UTF-8 spaces instead of single character spaces. See also http://rt.freeciv.org/Ticket/Display.html?id=40601 ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Nachricht geschickt von/durch Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Follow-up Comment #1, bug #15377 (project freeciv): The catalan.ruleset is in utf8, in both S2_2 and trunk. This is correct and all characters are utf8. It's loaded into the server where it remains utf-8. It's then sent over the network in utf-8. It's received by the client and converted by iconv into the client encoding which is also utf-8. It's then sent to gtk2 as utf8. The isutf8 program can be used to confirm that files are in utf8. All ruleset, data, and code files should be in utf8. In fact the only text files that may be non-UTF8 within freeciv is the translations which have their codeset listed at top (even these may be utf-8, which should be encouraged). But using this utility does not show up any invalid files. So I dont see where this can be going wrong. It could be caused by a bug in windows GTK or windows iconv as i have heard this is not reproducable on other platforms with the GTK2 client. It might also be a bug in windows use of iconv, if it is trying to convert it out of utf-8 for some reason, but in that case the bug should probably show up on other platforms and other rulesets. Thirdly it could be an issue only triggered by 64bit; the other bug reports windows7 64bit but this one just says windows vista. I do notice the documentation on this is limited and in some cases incorrect. I'll make a patch to improve this. -jason ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message sent via/by Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
Update of bug #15377 (project freeciv): Release: = 2.0.10, 2.2.0 ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message posté via/par Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
[Freeciv-Dev] [bug #15377] freeciv-gtk2.exe hangs
URL: http://gna.org/bugs/?15377 Summary: freeciv-gtk2.exe hangs Project: Freeciv Submitted by: None Submitted on: Monday 02/15/10 at 23:02 CET Category: client-gtk-2.0 Severity: 3 - Normal Priority: 5 - Normal Status: None Assigned to: None Originator Email: mborr...@hotmail.com Open/Closed: Open Release: Discussion Lock: Any Operating System: Microsoft Windows Planned Release: ___ Details: start freeciv 2.2.0 on windows vista (or XP) selecting catalan nation. All the rest of choices default. begin game, first city built Barcelona. next, build colons. the rest of the built cities are: Tarragona LLeida Girona Ripoll Perpinyà just creating Perpinyà the freeciv-gtk2.exe hangs i think is caused by à character on the city name. Catalan cites could contain àéèiòóu ïü charset. the same error appears on freeciv 2.0.10 release. ___ Reply to this item at: http://gna.org/bugs/?15377 ___ Message sent via/by Gna! http://gna.org/ ___ Freeciv-dev mailing list Freeciv-dev@gna.org https://mail.gna.org/listinfo/freeciv-dev
