Hello, In recent weeks, I was working on some changes in my posix32 library which had to do with handling of LCID locales. In this message I want to summarize some inconsistencies between mingw-w64's and MSVC's winnt.h, as well as point out some special/problematic cases when it comes to using LCID locales. The latter may be useful for people who write software which makes use of LCID locales.
A`LCID` locale is constructed from two pieces: - A `LANGID`; constructed by `MAKELANGID` macro using `LANG_*` (primary language id) and `SUBLANG_*` (sublanguage id) constants. - A `SORT_*` (sort id) constant; using non-default (`SORT_DEFAULT`) allows to change how functions such as `ComapreString` sort strings when using that `LCID` locale. There are two special `SUBLANG_*` values: 1. SUBLANG_NEUTRAL (0x00) 2. SUBALNG_DEFAULT (0x01) The `SUBLANG_NEUTRAL` creates what's called a neutral (or partial) locale; partial locales are deprecated. Such locales are only useful for use with `ConvertDefaultLocale` to obtain default locale for some language. In most cases, however, it will produce the same results as using `SUBLANG_DEFAULT`. The `SUBLANG_DEFAULT` creates "default" locale for a specific language. Examples: 1. MAKELANG(LANG_ENGLISH, SUBLANG_DEFAULT) is the same as using `SUBLANG_ENGLISH_US` (en-US) 2. MAKELANG(LANG_PORTUGUESE, SUBLANG_DEFAULT) is the same as using `SUBLANG_PORTUGUESE_BRAZILIAN` (pt-BR) 3. MAKELANG(LANG_IRISH, SUBLANG_DEFAULT) crates an invalid `LCID` locale (see below) As can be seen, `SUBLANG_DEFAULT` may result in unexpected results, and in particular, with the following primary language IDs: 1. 0x03: `LANG_CATALAN` and `LANG_VALENCIAN` 2. 0x1A: `LANG_CROATIAN`, `LANG_BOSNIAN` and `LANG_SERBIAN` 3. 0x2E: `LANG_UPPER_SORBIAN` and `LANG_LOWER_SORBIAN` First case will create `LANGID` as if MAKELANGID(LANG_CATALAN, SUBLANG_CATALAN_CATALAN). Second case will create `LANGID` as if MAKELANGID(LANG_CROATIAN, SUBLANG_CROATIAN_CROATIA). Third case will create `LANGID` as if MAKELANGID(LANG_UPPER_SORBIAN, SUBLANG_UPPER_SORBIAN_GERMANY). In all these cases, it may create `LANGID` for a different language than was intended. I addressed this issue in my posix32 library in this commit: https://github.com/maiddaisuki/posix32/commit/9e9904f0072dedfc82d863243d8dfbdd88417c53 Another thing about `SUBLANG_*` constants is that, for specific `LANG_*`, they usually start at value 0x01 and simply increment. They are, however, a few exceptions to this pattern as they have "gaps": 1. LANG_ENGLISH (0x09): valid sublanguage Ids are 0x01-0x16, except 0x14 and 0x15 2. LANG_IRISH (0x3C): the only valid sublanguage ID is 0x02 3. LANG_TIBETAN (0x51): valid sublanguage Ids are 0x01 and 0x03 4. LANG_TAMAZIGHT (0x5F): valid sublanguage Ids are 0x01-0x04, except 0x03 5. LANG_KASHMIRI (0x60): the only valid sublanguage ID is 0x02 In case 2 and 5, it has an effect that you cannot use `SUBLANG_DEFAULT` (0x01) to create a valid `LCID` locale. In addition to `LANG_*` and `SUBLANG_*` constants defined in winnt.h, there are also valid `LCID` locales for which there is no corresponding constants (either one or both may be missing). I discovered this by using `EnumSystemLocales` function, and simply deconstructing returned `LCID` values. I think after introducing locale names in Windows Vista, and deprecating `LCID` locales, in some cases Microsoft kept adding `LCID` for newly supported locales, but not adding new constants to `winnt.h`. For those who are interested, you can refer to the following source files in posix32: language.c: https://github.com/maiddaisuki/posix32/blob/master/lib/posix32-crt/src/locale/locale_map/language.c sublanguage.c: https://github.com/maiddaisuki/posix32/blob/master/lib/posix32-crt/src/locale/locale_map/sublanguage.c The first file lists all languages supported by Windows, as well corresponding `LANG_*` values. The second file lists all known valid `LCID` locales: it stores a `SUBLANG_*` value for that locale and an index to language list in `language.c`. Combining those `LANG_*` and `SUBLANG_*` values, you can construct corresponding `LCID` locale. In order to check whether constructed `LCID` is supported, you can use Windows function `IsValidLocale`. Some constants have aliases. Some of them are simply an alternative spelling, while some are preferred over legacy ones. The following are such `LANG_*` constants: 1. LANG_AZERBAIJANI is a preferred over legacy LANG_AZERI 2. LANG_BANGLA is an alias for LANG_BENGALI 3. LANG_PERSIAN is a preferred over legacy LANG_FARSI 4. LANG_FULAH is a preferred over legacy LANG_PULAR 5. LANG_ODIA is a preferred over legacy LANG_ORIYA 6. LANG_SAKHA is a preferred over legacy LANG_YAKUT 7. LANG_TIGRINYA is an alias for LANG_TIGRIGNA As the result, there are also similar `SUBLANG_*` aliases: 1. SUBLANG_AZERBAIJANI_AZERBAIJAN_CYRILLIC and SUBLANG_AZERI_CYRILLIC 2. SUBLANG_AZERBAIJANI_AZERBAIJAN_LATIN and SUBLANG_AZERI_LATIN 3. SUBLANG_BANGLA_BANGLADESH and SUBLANG_BENGALI_BANGLADESH 4. SUBLANG_BANGLA_INDIA and SUBLANG_BENGALI_INDIA 5. SUBLANG_FULAH_SENEGAL and SUBLANG_PULAR_SENEGAL 6. SUBLANG_KASHMIRI_SASIA and SUBLANG_KASHMIRI_INDIA 7. SUBLANG_ODIA_INDIA and SUBLANG_ORIYA_INDIA 8. SUBLANG_SAKHA_RUSSIA and SUBLANG_YAKUT_RUSSIA 9. SUBLANG_SINDHI_PAKISTAN and SUBLANG_SINDHI_AFGHANISTAN 10. SUBLANG_TIGRINYA_ERITREA and SUBLANG_TIGRIGNA_ERITREA The cases 6 and 9 feel a bit weird, but Microsoft's winnt.h claims they are really just aliases. However, mingw-w64's winnt.h defines aliases which are not found in MSVC headers: 1. SUBLANG_ENGLISH_IRELAND; same as SUBLANG_ENGLISH_EIRE 2. SUBLANG_HAUSA_NIGERIA; same as SUBLANG_HAUSA_NIGERIA_LATIN 3. SUBLANG_LAO_LAO_PDR; same as SUBLANG_LAO_LAO 4. SUBLANG_LITHUANIAN_LITHUANIA; same as SUBLANG_LITHUANIAN 5. SUBLANG_PORTUGUESE_PORTUGAL; same as SUBLANG_PORTUGUESE 6. SUBLANG_SWEDISH_SWEDEN; same as SUBLANG_SWEDISH 7. SUBLANG_SYRIAC; same as SUBLANG_SYRIAC_SYRIA Cases 4, 5 and 6 are guarded with `WINVER >= 0x0600`. Since the above 7 constants are missing from recent MSVC headers, I propose that we remove them. There are two more constants which are not found in MSVC headers: 1. SUBLANG_ROMANIAN_MOLDOVA; commented out with value 0x01 2. SUBLANG_TIBETAN_BHUTAN; has value 0x02 As for the first one, there is actually `LCID` locale for "ro-MD", but it has sublanguage ID 0x02. As for the second one, I mentioned above that `LANG_TIBETAN` with sublanguage ID 0x02 does not create valid `LCID` locale. However, with sublanguage ID 0x03, it creates `LCID` locale for "dz-BT" ("Dzongkha_Bhutan"). The country name matches the constant... In both cases, it seems to me like the author just got confused in all this mess. Similar to 7 constants above, I propose we simply remove these 2 constants from winnt.h. One more thing about `SORT_*` values. There are two locales "es-ES" and "es-ES_tradnl": the latter is created using a different `SUBLANG_*` value rather than `SORT_*` value: 1. es-ES: MAKELCID (MAKELANGID (LANG_SPANISH, SUBLANG_SPANISH_MODERN), SORT_DEFAULT) 2. es-ES_tradnl: MAKELCID (MAKELANGID (LANG_SPANISH, SUBLANG_SPANISH), SORT_DEFAULT) I had to extend logic in handling sorting orders for `LCID` in order to be able handle "es-ES_tradnl" locale: https://github.com/maiddaisuki/posix32/commit/43cbc6db20f5c83cc891d24baccdea9d6eb3fd84 - Kirill Makurin _______________________________________________ Mingw-w64-public mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/mingw-w64-public
