Re: Need help fixing failing locale tests

[John Marino]

On 11/15/2015 1:56 PM, Andrey Chernov wrote:
> On 15.11.2015 15:46, Baptiste Daroussin wrote:
>> On Sun, Nov 15, 2015 at 03:24:19PM +0300, Andrey Chernov wrote:
>>> On 15.11.2015 10:09, John Marino wrote:
>>> ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
>>> with that). Lots of ports (even at configure stage!) have checks for
>>> them. Since we generate locales from CLDR now, it will be no cost to
>>> bring all 8859-1 back to not violate POLA and not fix every failing port.
>>>
>> Exp-run have been made and no ports were failing with the removed locales.
> 
> There is soft-fail, configure just marks that locales are not supported
> and use "C". Sorry I don't remember port names where I saw it right now
> and don't have a time to search for them right now too. Soft-fails (like
> in tcl with nl_langinfo) are almost impossible to detect excepting
> specific situation happens or source code inspection. Do we ever need
> them when there is no harm to keep 8859-1 locales?
> 

Well, there is "harm".  The -1/-15 confusion happens a lot.

Is the plan to keep every locale ever created for ever and ever?  Never
do any kind of kind of cleanup or reorg?

This is not really any kind of problem.  Update the regression test and
nobody will even notice after that in all liklihood.

John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

[John Marino]

On 11/15/2015 2:10 PM, Andrey Chernov wrote:
>> Well, there is "harm".  The -1/-15 confusion happens a lot.
> 
> It is user confusion and his responsibility. It not leads to wrong
> program build f.e. Moreover, you can't protect users who set 8859-1 that
> way, they do not convert to 8859-15 as you assume but start to complain
> everywhere that FreeBSD is not working instead.

Invalid.  "locale -a" shows what locales are available.
The confusion is not with one user.  It's with one user that produces
document in one encoding and a second user that choses the wrong one
(usually -1).  -15 was designed to replace -1.

OpenBSD removed ISO8859* completely.


> 
>> Is the plan to keep every locale ever created for ever and ever?  Never
>> do any kind of kind of cleanup or reorg?
> 
> It will be nice to do it that way. FreeBSD have a little part of world
> locales, which indirectly assumes that they are really used.

Also invalid.  Locales are not standardized with regard to encoding, so
maintaining a museum of locales is specific to FreeBSD.  Linux calls
them differently.

John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

[John Marino]

On 11/15/2015 3:39 PM, Andrey Chernov wrote:
> As I already say, I don't want to insist on any particular point of view
> in such area as human behavior. I just say that it is POLA violation
> (even while it is upgrade) and we can let users decide by themselves,
> what it best for them (without me at least).

I am starting to think "POLA" as an acronym is subject to abuse.  By
this definition, *any* change would "astonish" a user (picturing the
most incompetent user impossible too).

POLA is meant for unreasonable and unexplained changes.  I don't think
tidying up locales for the first time in a decade is unreasonable or
unexplained.

Let's not dilute "POLA".  It's pretty good but you can apply it to anything.

>>> I tell about is not different text document encoding, but program
>>> failure due to inability to set his 8859-1 locale.
>>
>> Please provide an example of such a program (in ports).
> 
> See gettext-0.19.6/gettext-tools/configure, part started with
> # Test for the usual locale name.
> I don't have time to dig more such code now, but I remember I saw more
> for sure.
> 
>>> In any case it is related to the user behavior an various views on it.
>>> They can be different and I see no point to insist on my particular view
>>> at all. But... Programs configure soft-fails shows real danger here.
>>
>> and the impact of this is ... ?
> 
> Various. It depends on for what reason program sense locale.

If the ports framework is not overriding locale to "C" during the build
then that's probably something that should be introduced.  Ports should
not be producing different results based on the configuration of the
building machine at the time.

Right?  It's also good practice because C/POSIX never results in illegal
sequence errors while locales such as UTF-8 easily can.

John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

[John Marino]

On 11/15/2015 2:04 PM, Andrey Chernov wrote:
> On 15.11.2015 16:00, Baptiste Daroussin wrote:
>> On Sun, Nov 15, 2015 at 03:56:26PM +0300, Andrey Chernov wrote:
>>> On 15.11.2015 15:46, Baptiste Daroussin wrote:
>>>> On Sun, Nov 15, 2015 at 03:24:19PM +0300, Andrey Chernov wrote:
>>>>> On 15.11.2015 10:09, John Marino wrote:
>>>>> ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
>>>>> with that). Lots of ports (even at configure stage!) have checks for
>>>>> them. Since we generate locales from CLDR now, it will be no cost to
>>>>> bring all 8859-1 back to not violate POLA and not fix every failing port.
>>>>>
>>>> Exp-run have been made and no ports were failing with the removed locales.
>>>
>>> There is soft-fail, configure just marks that locales are not supported
>>> and use "C". Sorry I don't remember port names where I saw it right now
>>> and don't have a time to search for them right now too. Soft-fails (like
>>> in tcl with nl_langinfo) are almost impossible to detect excepting
>>> specific situation happens or source code inspection. Do we ever need
>>> them when there is no harm to keep 8859-1 locales?
>>
>> Is it ok if I readd those locales as aliases on 8859-15?
> 
> It is hacking solution leads to wrong collating order and character
> classes. It is better to generate true 8859-1 just in the same way you
> already do for 8859-15.
> 
> BTW, I can't check right now, but in case 8859-5 is removed too, it is
> better to restore it, it was used in Suns as their standard Russian
> encoding.
> 

DragonFly:

muscles# locale -a | grep 8859-5
uk_UA.ISO8859-5
be_BY.ISO8859-5
ru_RU.ISO8859-5
sr_Cyrl_RS.ISO8859-5

I agree that if -1 is brought back, it needs to be changed at the
charset.xml level.  At that point FreeBSD and DragonFly will diverge.  I
also agree using ports as justification for keeping -1 in western europe
is invalid (as in, it's not causing problems in ports)


John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

[John Marino]

On 11/15/2015 2:36 PM, Andrey Chernov wrote:
> On 15.11.2015 16:14, John Marino wrote:
>>> It is user confusion and his responsibility. It not leads to wrong
>>> program build f.e. Moreover, you can't protect users who set 8859-1 that
>>> way, they do not convert to 8859-15 as you assume but start to complain
>>> everywhere that FreeBSD is not working instead.
>>
>> Invalid.  "locale -a" shows what locales are available.
>> The confusion is not with one user.  It's with one user that produces
>> document in one encoding and a second user that choses the wrong one
>> (usually -1).  -15 was designed to replace -1.
> 
> No end-user use 'locale -a' to check locales. 

To quote you, that means it "user confusion and responsibility".
Fine.  He can "ls -d /usr/share/locales".  Otherwise, how does said user
set locales for the FIRST time.

We are talking about people that install FreeBSD 11 as a release.  If
it's an upgrade, it's documented in UPDATING (it will be) and anybody on
-CURRENT is taking responsibility for knowing what they are doing.  *IF*
this is an obstacle, it's either trivial or that user shouldn't be using
-CURRENT to begin with.


> An end-user keep some
> 8859-1 version is set many years ago, and "FreeBSD not working" problem
> I tell about is not different text document encoding, but program
> failure due to inability to set his 8859-1 locale.

Please provide an example of such a program (in ports).


> In any case it is related to the user behavior an various views on it.
> They can be different and I see no point to insist on my particular view
> at all. But... Programs configure soft-fails shows real danger here.

and the impact of this is ... ?


> 
>> OpenBSD removed ISO8859* completely.
> 
> OpenBSD was never good in locales in any case for all that years and
> contributes nothing in that area (f.e. Citrus was made by NetBSD). Now
> they try to simplify their life of supporting them, but since for us now
> all locales are autogenerated from CLDR I see no room for simplification
> in that way. Our "cleaned" locales (against to POLA) can be restored by
> autogeneration with minimal efforts, and they even took very little disk
> space.

It's not a technical issue.  -1 (and some -15) weren't removed for
technical issues, but to keep order.

>> Also invalid.  Locales are not standardized with regard to encoding, so
>> maintaining a museum of locales is specific to FreeBSD.  Linux calls
>> them differently.
> 
> Most of our (old) locales have the same names as linux ones.

Actually, none of the ISO* encodings.  Linux will accept "ISO8859-1"
because it's hyphen-insensitive, it it calls it "ISO-8859-1"

Maybe obsolete ones like cp866, GB2312 ... but there are more different
than the same.

John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

[John Marino]

On 11/15/2015 4:46 AM, NGie Cooper wrote:
> 
>> On Nov 14, 2015, at 19:28, NGie Cooper  wrote:
> 
> …
> 
>> Why were these locales removed?
>>
>>  58 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_COLLATE
>>  59 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_CTYPE
>>  60 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_TIME
>>  61 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-1
>>  62 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_COLLATE
>>  63 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_CTYPE
>>  64 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-13
> 
> la_LN.ISO8859-1 is the old Latin locale, which is no longer installed. 
> Copying over locale files from a stable/10 host works, but I’m confused as to 
> why a bunch of locales weren’t ported over in their non-UTF-8 forms.
> Thanks,
> 

We (DragonFly) didn't just update locales.  We took the opportunity to
do spring cleaning.  We didn't want to be as drastic as OpenBSD which
removed all encodings except for C/POSIX and UTF, but we did remove
several locales intentionally.

In the case of ISO8859-1:
All ISO8859-* is basically obsolete.
In western Europe, if somebody wants ISO-8859, they want ISO8859-15, not
ISO8859-1.  They are similar, but the former is tailored for western
europe with "Euro" currency and 9 other symbols.  It comes at the
expense of removing 10 characters from ISO8859-1.  There's also a common
problem that users view -15 documents with -1 accidently.  So there was
a conscience decision to have either ISO8859-1 or ISO8859-15 but not
both.  For western Europe this means the ISO8859-1 versions were dropped.

ISO8859-15:
In the case of USA and other non-European countries, they keep -1 and
dropped -15.

currency based:
In the case of countries where the currency symbols is not part of
ISO8859, it was dropped.  E.g. Costa Rica uses the Colon which is only
in UTF-8, so there's no ISO8859-* encoding at all for CR.

Latin:
Who speaks Latin today?
This was mainly an alias for 7-bit ascii.  We originally dropped that,
but later moved it to US-ASCII (which was just a symlink to Latin before)

Bapt liked DF approach well enough that he adopted it.  Even Edwin was
first in desiring to clean up locales.  The major update was a perfect time.

Bottom line:
The testsuite needs to be updated.
e.g. use de_DE.ISO8859-15 intead of de_DE.ISO8859-1
For latin, replace with US-ASCII equivalent.


John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

[John Marino]

On 11/15/2015 8:24 AM, NGie Cooper wrote:
>> On Nov 14, 2015, at 23:09, John Marino <dragonfly...@marino.st> wrote:
>> Bapt liked DF approach well enough that he adopted it.  Even Edwin was
>> first in desiring to clean up locales.  The major update was a perfect time.
>>
>> Bottom line:
>> The testsuite needs to be updated.
>> e.g. use de_DE.ISO8859-15 intead of de_DE.ISO8859-1
>> For latin, replace with US-ASCII equivalent.
> 
>   I wish this had been clearly communicated here: 
> https://svnweb.freebsd.org/changeset/base/290494 … instead, there are some 
> POLA violations:
>   1. No RelNotes.
>   2. No summary of locales that have been removed.
>   3. No UPDATING entry for people to migrate from a locale to another.
>   4. Deprecated locales (as you described it above) are still present on 
> my system after running `make delete-old` (assuming I have my acronyms right, 
> for example, ca_ES, it_CH, etc should be -1, not -15 based on your claim 
> above).
>   I will update the testcases to use locales if possible, but more work 
> needs to be done finishing off this project and documenting for end-users 
> what has changed.

With 4.) they have been removed from FreeBSD when "Make upgrade" is run
after rebuilding.  Does FreeBSD have a cleanup like that, and did you
run it?  If not, then maybe that needs to be updated.

with 3.) That's a bit like saying changing the color of shoe laces needs
instructions for people to learn how to adjust.  Run "locale -a" to see
what's available on the system.  This has always been true.  Attempting
to switch to a locale that doesn't exist results in staying in "C"
locale, which is quite obvious.

with 2.) Could be combined with 1.) but in any case, there really
shouldn't be POLA on anything that has no standard.  Should you announce
every time you add a locale too, in release notes?
Really "locale -a" should tell the whole story.

1.) A paragraph on removed locales is fine.  Those removed have been
removed because they are a) likely unused and b) if used, they are used
by mistake.  in this case you hit c) where the testsuite probably chose
the "wrong" locale in the first place.  Note that GCC did this too with
their libstdc++ testsuite and that was fixed this weekend (a huge patch
to GCC 6 upstream).

John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

[John Marino]

On 11/15/2015 12:23 PM, Craig Rodrigues wrote:
> 
> 
> On Sat, Nov 14, 2015 at 11:47 PM, John Marino <freebsd.cont...@marino.st
> <mailto:freebsd.cont...@marino.st>> wrote:
> 
> 
> With 4.) they have been removed from FreeBSD when "Make upgrade" is run
> after rebuilding.  Does FreeBSD have a cleanup like that, and did you
> run it?  If not, then maybe that needs to be updated.

by they way, this was meant to be "removed from DragonFly".  I don't
know if "make upgrade" is set right for FreeBSD.  (it appears not)

> 
> I did not run "make upgrade".
> I did this:
> cd /usr/src
> svn up
> 
> I then ran the 11 steps listed in /usr/src/Makefile:
> 
> https://github.com/freebsd/freebsd/blob/master/Makefile#L73
> 
> It looks like after running those steps, my system is in an inconsistent
> state.  I see that the following symlinks point to nonexistent files.

[snip]

It looks like the deleted locales were not removed.
They should be.


> What is the fix for this?

I think somebody needs to add these locales to be reaped by make upgrade.


> It looks like there is some Makefile logic which is wrong.


John


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

[John Marino]

On 11/15/2015 4:38 PM, Andrey Chernov wrote:
> On 15.11.2015 18:17, John Marino wrote:
>>
>> For the traditional checks, It's ironic, DragonFly uses short locales
>> like "fr_FR" which link to the approprioprate ISO8859 or UTF-8 locale.
>> Bapt removed them to avoid a bike shed and if he had not done that, this
>> gettext configure would not be an issue as "fr_FR" is the very first check.
> 
> All parts are optional excepting language itself, so fr_FR is not short
> enough, but fr is. From POSIX:
> 
> language[_territory][.codeset][@modifier]

"fr" (or "french") by itself isn't very useful, so IMO
language_territory is the minimum.

> But I dislike short locales due to their uncertainty and remember some
> programs and shell scripts that attempts to parse locale name directly
> by itself for reasons I don't remember now.

Absolutely true about the uncertainty.  It's just subjective.  We did
what we thought made sense but bapt recognized an impeding bikeshed and
just did away with it.

For some reason gettext-tools only cares about french and japanese (thus
easily patched).  Here's what it looks like on current DragonFly:

http://muscles.dragonflybsd.org/bulk/bleeding-edge-potential/latest-per-pkg/gettext-tools-0.19.6.log

Note the identified traditional french is "fr_FR" which maps to
fr_FR.ISO8859-15 on DragonFly.

It's a moot point now I guess.  ISO8859-1 has returned to FreeBSD for
western europe (probably forever).

John




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

[John Marino]

On 11/15/2015 3:59 PM, Andrey Chernov wrote:
> On 15.11.2015 17:47, John Marino wrote:
>>>> Please provide an example of such a program (in ports).
>>>
>>> See gettext-0.19.6/gettext-tools/configure, part started with
>>> # Test for the usual locale name.
>>> I don't have time to dig more such code now, but I remember I saw more
>>> for sure.
> 
>> If the ports framework is not overriding locale to "C" during the build
>> then that's probably something that should be introduced.  Ports should
>> not be producing different results based on the configuration of the
>> building machine at the time.
> 
> This is not about resetting locale to "C" during the build. Please
> really look at "configure" specific part mentioned above, it is you who
> ask at least one specific example and then it is you who ignore the
> answer takes my time to find it.

Hmmm?
Of course it's the same topic.
If the configure of a port is affected by the locale, that's a big
problem.  How is this not the same topic?

But I did look at gettext, which is configuring package based on what's
on the system (not environment).  For the unicode checks, there is
obviously no issue.

For the traditional checks, It's ironic, DragonFly uses short locales
like "fr_FR" which link to the approprioprate ISO8859 or UTF-8 locale.
Bapt removed them to avoid a bike shed and if he had not done that, this
gettext configure would not be an issue as "fr_FR" is the very first check.

John

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: gccmakedep fix - 20130619

[John Marino]

On 6/19/2013 01:00, Oliver Pinter wrote:

Hi all!

Attached a fix to gccmakedep.



So
1) There's already multiple PRs on this, including one I just submitted:
http://www.freebsd.org/cgi/query-pr.cgi?pr=179571

2) Your patch adds an .endif which is already there

3) Is not it better to submit via PR (GNATS) rather than a mailing list? 
 That's what PRs are for...


John

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org