Re: Need help fixing failing locale tests

2015-11-15 Thread John Marino 
On 11/15/2015 1:56 PM, Andrey Chernov wrote:
> On 15.11.2015 15:46, Baptiste Daroussin wrote:
>> On Sun, Nov 15, 2015 at 03:24:19PM +0300, Andrey Chernov wrote:
>>> On 15.11.2015 10:09, John Marino wrote:
>>> ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
>>> with that). Lots of ports (even at configure stage!) have checks for
>>> them. Since we generate locales from CLDR now, it will be no cost to
>>> bring all 8859-1 back to not violate POLA and not fix every failing port.
>>>
>> Exp-run have been made and no ports were failing with the removed locales.
> 
> There is soft-fail, configure just marks that locales are not supported
> and use "C". Sorry I don't remember port names where I saw it right now
> and don't have a time to search for them right now too. Soft-fails (like
> in tcl with nl_langinfo) are almost impossible to detect excepting
> specific situation happens or source code inspection. Do we ever need
> them when there is no harm to keep 8859-1 locales?
> 

Well, there is "harm".  The -1/-15 confusion happens a lot.

Is the plan to keep every locale ever created for ever and ever?  Never
do any kind of kind of cleanup or reorg?

This is not really any kind of problem.  Update the regression test and
nobody will even notice after that in all liklihood.

John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread sthaug 
> > There is soft-fail, configure just marks that locales are not supported
> > and use "C". Sorry I don't remember port names where I saw it right now
> > and don't have a time to search for them right now too. Soft-fails (like
> > in tcl with nl_langinfo) are almost impossible to detect excepting
> > specific situation happens or source code inspection. Do we ever need
> > them when there is no harm to keep 8859-1 locales?
> 
> Is it ok if I readd those locales as aliases on 8859-15?

Why on earth would you want to do that when we can have the real thing?

I still see no good reason for dropping 8859-1.

Steinar Haug, Nethelp consulting, sth...@nethelp.no
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
On 15.11.2015 16:14, John Marino wrote:
>> It is user confusion and his responsibility. It not leads to wrong
>> program build f.e. Moreover, you can't protect users who set 8859-1 that
>> way, they do not convert to 8859-15 as you assume but start to complain
>> everywhere that FreeBSD is not working instead.
> 
> Invalid.  "locale -a" shows what locales are available.
> The confusion is not with one user.  It's with one user that produces
> document in one encoding and a second user that choses the wrong one
> (usually -1).  -15 was designed to replace -1.

No end-user use 'locale -a' to check locales. An end-user keep some
8859-1 version is set many years ago, and "FreeBSD not working" problem
I tell about is not different text document encoding, but program
failure due to inability to set his 8859-1 locale.

In any case it is related to the user behavior an various views on it.
They can be different and I see no point to insist on my particular view
at all. But... Programs configure soft-fails shows real danger here.

> OpenBSD removed ISO8859* completely.

OpenBSD was never good in locales in any case for all that years and
contributes nothing in that area (f.e. Citrus was made by NetBSD). Now
they try to simplify their life of supporting them, but since for us now
all locales are autogenerated from CLDR I see no room for simplification
in that way. Our "cleaned" locales (against to POLA) can be restored by
autogeneration with minimal efforts, and they even took very little disk
space.

> Also invalid.  Locales are not standardized with regard to encoding, so
> maintaining a museum of locales is specific to FreeBSD.  Linux calls
> them differently.

Most of our (old) locales have the same names as linux ones.

-- 
http://ache.vniz.net/
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread John Marino 
On 11/15/2015 2:10 PM, Andrey Chernov wrote:
>> Well, there is "harm".  The -1/-15 confusion happens a lot.
> 
> It is user confusion and his responsibility. It not leads to wrong
> program build f.e. Moreover, you can't protect users who set 8859-1 that
> way, they do not convert to 8859-15 as you assume but start to complain
> everywhere that FreeBSD is not working instead.

Invalid.  "locale -a" shows what locales are available.
The confusion is not with one user.  It's with one user that produces
document in one encoding and a second user that choses the wrong one
(usually -1).  -15 was designed to replace -1.

OpenBSD removed ISO8859* completely.


> 
>> Is the plan to keep every locale ever created for ever and ever?  Never
>> do any kind of kind of cleanup or reorg?
> 
> It will be nice to do it that way. FreeBSD have a little part of world
> locales, which indirectly assumes that they are really used.

Also invalid.  Locales are not standardized with regard to encoding, so
maintaining a museum of locales is specific to FreeBSD.  Linux calls
them differently.

John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
On 15.11.2015 16:54, John Marino wrote:
> We are talking about people that install FreeBSD 11 as a release.  If
> it's an upgrade, it's documented in UPDATING (it will be) and anybody on
> -CURRENT is taking responsibility for knowing what they are doing.  *IF*
> this is an obstacle, it's either trivial or that user shouldn't be using
> -CURRENT to begin with.

As I already say, I don't want to insist on any particular point of view
in such area as human behavior. I just say that it is POLA violation
(even while it is upgrade) and we can let users decide by themselves,
what it best for them (without me at least).

>> I tell about is not different text document encoding, but program
>> failure due to inability to set his 8859-1 locale.
> 
> Please provide an example of such a program (in ports).

See gettext-0.19.6/gettext-tools/configure, part started with
# Test for the usual locale name.
I don't have time to dig more such code now, but I remember I saw more
for sure.

>> In any case it is related to the user behavior an various views on it.
>> They can be different and I see no point to insist on my particular view
>> at all. But... Programs configure soft-fails shows real danger here.
> 
> and the impact of this is ... ?

Various. It depends on for what reason program sense locale.

-- 
http://ache.vniz.net/
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread John Marino 
On 11/15/2015 3:39 PM, Andrey Chernov wrote:
> As I already say, I don't want to insist on any particular point of view
> in such area as human behavior. I just say that it is POLA violation
> (even while it is upgrade) and we can let users decide by themselves,
> what it best for them (without me at least).

I am starting to think "POLA" as an acronym is subject to abuse.  By
this definition, *any* change would "astonish" a user (picturing the
most incompetent user impossible too).

POLA is meant for unreasonable and unexplained changes.  I don't think
tidying up locales for the first time in a decade is unreasonable or
unexplained.

Let's not dilute "POLA".  It's pretty good but you can apply it to anything.

>>> I tell about is not different text document encoding, but program
>>> failure due to inability to set his 8859-1 locale.
>>
>> Please provide an example of such a program (in ports).
> 
> See gettext-0.19.6/gettext-tools/configure, part started with
> # Test for the usual locale name.
> I don't have time to dig more such code now, but I remember I saw more
> for sure.
> 
>>> In any case it is related to the user behavior an various views on it.
>>> They can be different and I see no point to insist on my particular view
>>> at all. But... Programs configure soft-fails shows real danger here.
>>
>> and the impact of this is ... ?
> 
> Various. It depends on for what reason program sense locale.

If the ports framework is not overriding locale to "C" during the build
then that's probably something that should be introduced.  Ports should
not be producing different results based on the configuration of the
building machine at the time.

Right?  It's also good practice because C/POSIX never results in illegal
sequence errors while locales such as UTF-8 easily can.

John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
On 15.11.2015 18:38, Andrey Chernov wrote:
>> For the traditional checks, It's ironic, DragonFly uses short locales
>> like "fr_FR" which link to the approprioprate ISO8859 or UTF-8 locale.
>> Bapt removed them to avoid a bike shed and if he had not done that, this
>> gettext configure would not be an issue as "fr_FR" is the very first check.
>
> But I dislike short locales due to their uncertainty

And gettext-0.19.6/gettext-tools/configure is good example of that
uncertainty. If DragonFly short locale fr_FR links to 8859-1, configure
test does what it is intended, but if it links to UTF-8 it does not.

-- 
http://ache.vniz.net/
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread John Marino 
On 11/15/2015 2:04 PM, Andrey Chernov wrote:
> On 15.11.2015 16:00, Baptiste Daroussin wrote:
>> On Sun, Nov 15, 2015 at 03:56:26PM +0300, Andrey Chernov wrote:
>>> On 15.11.2015 15:46, Baptiste Daroussin wrote:
 On Sun, Nov 15, 2015 at 03:24:19PM +0300, Andrey Chernov wrote:
> On 15.11.2015 10:09, John Marino wrote:
> ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
> with that). Lots of ports (even at configure stage!) have checks for
> them. Since we generate locales from CLDR now, it will be no cost to
> bring all 8859-1 back to not violate POLA and not fix every failing port.
>
 Exp-run have been made and no ports were failing with the removed locales.
>>>
>>> There is soft-fail, configure just marks that locales are not supported
>>> and use "C". Sorry I don't remember port names where I saw it right now
>>> and don't have a time to search for them right now too. Soft-fails (like
>>> in tcl with nl_langinfo) are almost impossible to detect excepting
>>> specific situation happens or source code inspection. Do we ever need
>>> them when there is no harm to keep 8859-1 locales?
>>
>> Is it ok if I readd those locales as aliases on 8859-15?
> 
> It is hacking solution leads to wrong collating order and character
> classes. It is better to generate true 8859-1 just in the same way you
> already do for 8859-15.
> 
> BTW, I can't check right now, but in case 8859-5 is removed too, it is
> better to restore it, it was used in Suns as their standard Russian
> encoding.
> 

DragonFly:

muscles# locale -a | grep 8859-5
uk_UA.ISO8859-5
be_BY.ISO8859-5
ru_RU.ISO8859-5
sr_Cyrl_RS.ISO8859-5

I agree that if -1 is brought back, it needs to be changed at the
charset.xml level.  At that point FreeBSD and DragonFly will diverge.  I
also agree using ports as justification for keeping -1 in western europe
is invalid (as in, it's not causing problems in ports)


John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
On 15.11.2015 17:47, John Marino wrote:
>>> Please provide an example of such a program (in ports).
>>
>> See gettext-0.19.6/gettext-tools/configure, part started with
>> # Test for the usual locale name.
>> I don't have time to dig more such code now, but I remember I saw more
>> for sure.

> If the ports framework is not overriding locale to "C" during the build
> then that's probably something that should be introduced.  Ports should
> not be producing different results based on the configuration of the
> building machine at the time.

This is not about resetting locale to "C" during the build. Please
really look at "configure" specific part mentioned above, it is you who
ask at least one specific example and then it is you who ignore the
answer takes my time to find it.

-- 
http://ache.vniz.net/
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
On 15.11.2015 18:01, Baptiste Daroussin wrote:
> I have restored the 8859-1

Thanx!
BTW, speaking with John I find at least one configure script from ports
which depends on 8859-1 presence.
See gettext-0.19.6/gettext-tools/configure, part started with
# Test for the usual locale name.

-- 
http://ache.vniz.net/



signature.asc
Description: OpenPGP digital signature

Re: Need help fixing failing locale tests

2015-11-15 Thread Pedro Giffuni 
FWIW;

While I personally don’t use it, Latin is the official language for the
Holy See [1]. I think it is still taught in schools in Italy for cultural
reasons and because it’s supposed to make easier to learn other
“romance” languages.

It shouldn't hurt to keep it but I have no strong opinion.

Pedro.

[1] https://en.wikipedia.org/wiki/Holy_See
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Miroslav Lachman 

John Marino wrote on 11/15/2015 15:47:

On 11/15/2015 3:39 PM, Andrey Chernov wrote:

As I already say, I don't want to insist on any particular point of view
in such area as human behavior. I just say that it is POLA violation
(even while it is upgrade) and we can let users decide by themselves,
what it best for them (without me at least).


I am starting to think "POLA" as an acronym is subject to abuse.  By
this definition, *any* change would "astonish" a user (picturing the
most incompetent user impossible too).

POLA is meant for unreasonable and unexplained changes.  I don't think
tidying up locales for the first time in a decade is unreasonable or
unexplained.

Let's not dilute "POLA".  It's pretty good but you can apply it to anything.


I agree. Everytime FreeBSD is changing something in the base, there are 
voices with "POLA". Removing Perl from base, removing BIND from base... 
these were more significant changes than changing something in locales.


Miroslav Lachman

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Michael Mitchell 
Documentation on any existing regressions and how they are expected to
fail would be a good thing… Change is necessary, but so is modification of
a regression to continue to pass (or fail) as expected. Demonstration of the
unit testing process can also be very beneficial.

: mdm

> On Nov 15, 2015, at 8:10 AM, Miroslav Lachman <000.f...@quip.cz> wrote:
> 
>>> can



smime.p7s
Description: S/MIME cryptographic signature

Re: Need help fixing failing locale tests

2015-11-15 Thread John Marino 
On 11/15/2015 2:36 PM, Andrey Chernov wrote:
> On 15.11.2015 16:14, John Marino wrote:
>>> It is user confusion and his responsibility. It not leads to wrong
>>> program build f.e. Moreover, you can't protect users who set 8859-1 that
>>> way, they do not convert to 8859-15 as you assume but start to complain
>>> everywhere that FreeBSD is not working instead.
>>
>> Invalid.  "locale -a" shows what locales are available.
>> The confusion is not with one user.  It's with one user that produces
>> document in one encoding and a second user that choses the wrong one
>> (usually -1).  -15 was designed to replace -1.
> 
> No end-user use 'locale -a' to check locales. 

To quote you, that means it "user confusion and responsibility".
Fine.  He can "ls -d /usr/share/locales".  Otherwise, how does said user
set locales for the FIRST time.

We are talking about people that install FreeBSD 11 as a release.  If
it's an upgrade, it's documented in UPDATING (it will be) and anybody on
-CURRENT is taking responsibility for knowing what they are doing.  *IF*
this is an obstacle, it's either trivial or that user shouldn't be using
-CURRENT to begin with.


> An end-user keep some
> 8859-1 version is set many years ago, and "FreeBSD not working" problem
> I tell about is not different text document encoding, but program
> failure due to inability to set his 8859-1 locale.

Please provide an example of such a program (in ports).


> In any case it is related to the user behavior an various views on it.
> They can be different and I see no point to insist on my particular view
> at all. But... Programs configure soft-fails shows real danger here.

and the impact of this is ... ?


> 
>> OpenBSD removed ISO8859* completely.
> 
> OpenBSD was never good in locales in any case for all that years and
> contributes nothing in that area (f.e. Citrus was made by NetBSD). Now
> they try to simplify their life of supporting them, but since for us now
> all locales are autogenerated from CLDR I see no room for simplification
> in that way. Our "cleaned" locales (against to POLA) can be restored by
> autogeneration with minimal efforts, and they even took very little disk
> space.

It's not a technical issue.  -1 (and some -15) weren't removed for
technical issues, but to keep order.

>> Also invalid.  Locales are not standardized with regard to encoding, so
>> maintaining a museum of locales is specific to FreeBSD.  Linux calls
>> them differently.
> 
> Most of our (old) locales have the same names as linux ones.

Actually, none of the ISO* encodings.  Linux will accept "ISO8859-1"
because it's hyphen-insensitive, it it calls it "ISO-8859-1"

Maybe obsolete ones like cp866, GB2312 ... but there are more different
than the same.

John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Baptiste Daroussin 
On Sun, Nov 15, 2015 at 04:04:33PM +0300, Andrey Chernov wrote:
> On 15.11.2015 16:00, Baptiste Daroussin wrote:
> > On Sun, Nov 15, 2015 at 03:56:26PM +0300, Andrey Chernov wrote:
> >> On 15.11.2015 15:46, Baptiste Daroussin wrote:
> >>> On Sun, Nov 15, 2015 at 03:24:19PM +0300, Andrey Chernov wrote:
>  On 15.11.2015 10:09, John Marino wrote:
>  ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
>  with that). Lots of ports (even at configure stage!) have checks for
>  them. Since we generate locales from CLDR now, it will be no cost to
>  bring all 8859-1 back to not violate POLA and not fix every failing port.
> 
> >>> Exp-run have been made and no ports were failing with the removed locales.
> >>
> >> There is soft-fail, configure just marks that locales are not supported
> >> and use "C". Sorry I don't remember port names where I saw it right now
> >> and don't have a time to search for them right now too. Soft-fails (like
> >> in tcl with nl_langinfo) are almost impossible to detect excepting
> >> specific situation happens or source code inspection. Do we ever need
> >> them when there is no harm to keep 8859-1 locales?
> > 
> > Is it ok if I readd those locales as aliases on 8859-15?
> 
> It is hacking solution leads to wrong collating order and character
> classes. It is better to generate true 8859-1 just in the same way you
> already do for 8859-15.
> 
> BTW, I can't check right now, but in case 8859-5 is removed too, it is
> better to restore it, it was used in Suns as their standard Russian
> encoding.
> 
I have restored the 8859-1

The 8859-5 were never removed:
be_BY.ISO8859-5
ru_RU.ISO8859-5
sr_Cyrl_RS.ISO8859-5
uk_UA.ISO8859-5

Best regards,
Bapt


signature.asc
Description: PGP signature

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
On 15.11.2015 18:17, John Marino wrote:
 See gettext-0.19.6/gettext-tools/configure, part started with
 # Test for the usual locale name.
 I don't have time to dig more such code now, but I remember I saw more
 for sure.

> Of course it's the same topic.
> If the configure of a port is affected by the locale, that's a big
> problem.  How is this not the same topic?

This is environment problem, not presence or absence of some locales in
the system (f.e. 8859-1 ones) we talked about. And can be easily fixed
in the environment, unlike configure's silent script fallbacks, which
can be fixed by recompiling only.

About environment problem, we already have LANG=C somewhere in the
/usr/ports/Mk/* (probably not in every place) and many ports set LANG=C
or LC_ALL by themselves.

> But I did look at gettext, which is configuring package based on what's
> on the system (not environment).  For the unicode checks, there is
> obviously no issue.
> 
> For the traditional checks, It's ironic, DragonFly uses short locales
> like "fr_FR" which link to the approprioprate ISO8859 or UTF-8 locale.
> Bapt removed them to avoid a bike shed and if he had not done that, this
> gettext configure would not be an issue as "fr_FR" is the very first check.

All parts are optional excepting language itself, so fr_FR is not short
enough, but fr is. From POSIX:

language[_territory][.codeset][@modifier]

But I dislike short locales due to their uncertainty and remember some
programs and shell scripts that attempts to parse locale name directly
by itself for reasons I don't remember now.

-- 
http://ache.vniz.net/
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread NGie Cooper 

> On Nov 15, 2015, at 03:30, John Marino  wrote:

…

> by they way, this was meant to be "removed from DragonFly".  I don't
> know if "make upgrade" is set right for FreeBSD.  (it appears not)

make upgrade doesn’t exist on FreeBSD:

$ (cd /usr/src/svn/; make upgrade)
make: don't know how to make upgrade. Stop

make: stopped in /usr/src/svn
$

> It looks like the deleted locales were not removed.
> They should be.

Hence my previous reply…

> I think somebody needs to add these locales to be reaped by make upgrade.

ObsoleteFiles.inc is missing old locale files/directories.

Cheers,
-NGie
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread sthaug 
> >> There's also a common
> >> problem that users view -15 documents with -1 accidently.  So there was
> >> a conscience decision to have either ISO8859-1 or ISO8859-15 but not
> >> both.  For western Europe this means the ISO8859-1 versions were dropped.
> 
> It is pure user problem choosing its own locale (self footshooting), it
> not leads to program build failures (like configure checks) or tests
> failures etc. BTW, linux keeps 8859-1 locales.

I see no good reason whatsoever to drop ISO8859-1.

Steinar Haug, Nethelp consulting, sth...@nethelp.no
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread John Marino 
On 11/15/2015 4:46 AM, NGie Cooper wrote:
> 
>> On Nov 14, 2015, at 19:28, NGie Cooper  wrote:
> 
> …
> 
>> Why were these locales removed?
>>
>>  58 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_COLLATE
>>  59 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_CTYPE
>>  60 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_TIME
>>  61 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-1
>>  62 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_COLLATE
>>  63 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_CTYPE
>>  64 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-13
> 
> la_LN.ISO8859-1 is the old Latin locale, which is no longer installed. 
> Copying over locale files from a stable/10 host works, but I’m confused as to 
> why a bunch of locales weren’t ported over in their non-UTF-8 forms.
> Thanks,
> 

We (DragonFly) didn't just update locales.  We took the opportunity to
do spring cleaning.  We didn't want to be as drastic as OpenBSD which
removed all encodings except for C/POSIX and UTF, but we did remove
several locales intentionally.

In the case of ISO8859-1:
All ISO8859-* is basically obsolete.
In western Europe, if somebody wants ISO-8859, they want ISO8859-15, not
ISO8859-1.  They are similar, but the former is tailored for western
europe with "Euro" currency and 9 other symbols.  It comes at the
expense of removing 10 characters from ISO8859-1.  There's also a common
problem that users view -15 documents with -1 accidently.  So there was
a conscience decision to have either ISO8859-1 or ISO8859-15 but not
both.  For western Europe this means the ISO8859-1 versions were dropped.

ISO8859-15:
In the case of USA and other non-European countries, they keep -1 and
dropped -15.

currency based:
In the case of countries where the currency symbols is not part of
ISO8859, it was dropped.  E.g. Costa Rica uses the Colon which is only
in UTF-8, so there's no ISO8859-* encoding at all for CR.

Latin:
Who speaks Latin today?
This was mainly an alias for 7-bit ascii.  We originally dropped that,
but later moved it to US-ASCII (which was just a symlink to Latin before)

Bapt liked DF approach well enough that he adopted it.  Even Edwin was
first in desiring to clean up locales.  The major update was a perfect time.

Bottom line:
The testsuite needs to be updated.
e.g. use de_DE.ISO8859-15 intead of de_DE.ISO8859-1
For latin, replace with US-ASCII equivalent.


John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Craig Rodrigues 
On Sat, Nov 14, 2015 at 11:47 PM, John Marino 
wrote:

>
> With 4.) they have been removed from FreeBSD when "Make upgrade" is run
> after rebuilding.  Does FreeBSD have a cleanup like that, and did you
> run it?  If not, then maybe that needs to be updated.
>
>
I did not run "make upgrade".
I did this:
cd /usr/src
svn up

I then ran the 11 steps listed in /usr/src/Makefile:

https://github.com/freebsd/freebsd/blob/master/Makefile#L73

It looks like after running those steps, my system is in an inconsistent
state.  I see that the following symlinks point to nonexistent files.

/usr/share/locale/nl_BE.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/nl_BE.ISO8859-1/LC_COLLATE ->
../la_LN.ISO8859-1/LC_COLLATE
/usr/share/locale/en_CA.ISO8859-15/LC_CTYPE -> ../la_LN.ISO8859-15/LC_CTYPE
/usr/share/locale/en_CA.ISO8859-15/LC_COLLATE ->
../la_LN.ISO8859-15/LC_COLLATE
/usr/share/locale/en_US.ISO8859-15/LC_COLLATE ->
../la_LN.ISO8859-15/LC_COLLATE
/usr/share/locale/en_US.ISO8859-15/LC_CTYPE -> ../la_LN.ISO8859-15/LC_CTYPE
/usr/share/locale/ca_FR.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/nb_NO.ISO8859-1/LC_MONETARY ->
../no_NO.ISO8859-1/LC_MONETARY
/usr/share/locale/nb_NO.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/nb_NO.ISO8859-1/LC_COLLATE ->
../no_NO.ISO8859-1/LC_COLLATE
/usr/share/locale/nb_NO.ISO8859-1/LC_MESSAGES ->
../no_NO.ISO8859-1/LC_MESSAGES
/usr/share/locale/nb_NO.ISO8859-1/LC_NUMERIC ->
../no_NO.ISO8859-1/LC_NUMERIC
/usr/share/locale/sr_YU.ISO8859-2/LC_COLLATE ->
../la_LN.ISO8859-2/LC_COLLATE
/usr/share/locale/sr_YU.ISO8859-2/LC_CTYPE -> ../la_LN.ISO8859-2/LC_CTYPE
/usr/share/locale/is_IS.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/zh_TW.Big5/zh_Hant_TW.Big5 -> zh_Hant_TW.Big5
/usr/share/locale/af_ZA.ISO8859-15/LC_CTYPE -> ../la_LN.ISO8859-15/LC_CTYPE
/usr/share/locale/af_ZA.ISO8859-15/LC_COLLATE ->
../la_LN.ISO8859-15/LC_COLLATE
/usr/share/locale/en_AU.ISO8859-15/LC_COLLATE ->
../la_LN.ISO8859-15/LC_COLLATE
/usr/share/locale/en_AU.ISO8859-15/LC_CTYPE -> ../la_LN.ISO8859-15/LC_CTYPE
/usr/share/locale/fr_CH.ISO8859-1/LC_COLLATE ->
../la_LN.ISO8859-1/LC_COLLATE
/usr/share/locale/fr_CH.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/zh_CN.UTF-8/LC_COLLATE -> ../la_LN.US-ASCII/LC_COLLATE
/usr/share/locale/zh_CN.UTF-8/zh_Hans_CN.UTF-8 -> zh_Hans_CN.UTF-8
/usr/share/locale/zh_CN.UTF-8/LC_CTYPE -> ../UTF-8/LC_CTYPE
/usr/share/locale/sv_SE.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/it_IT.ISO8859-1/LC_COLLATE ->
../la_LN.ISO8859-1/LC_COLLATE
/usr/share/locale/it_IT.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/it_CH.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/it_CH.ISO8859-1/LC_COLLATE ->
../la_LN.ISO8859-1/LC_COLLATE
/usr/share/locale/nn_NO.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/nn_NO.ISO8859-1/LC_COLLATE ->
../no_NO.ISO8859-1/LC_COLLATE
/usr/share/locale/nn_NO.ISO8859-1/LC_MONETARY ->
../no_NO.ISO8859-1/LC_MONETARY
/usr/share/locale/nn_NO.ISO8859-1/LC_NUMERIC ->
../no_NO.ISO8859-1/LC_NUMERIC
/usr/share/locale/nn_NO.ISO8859-1/LC_MESSAGES ->
../no_NO.ISO8859-1/LC_MESSAGES
/usr/share/locale/nl_NL.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/nl_NL.ISO8859-1/LC_COLLATE ->
../la_LN.ISO8859-1/LC_COLLATE
/usr/share/locale/zh_TW.UTF-8/LC_COLLATE -> ../la_LN.US-ASCII/LC_COLLATE
/usr/share/locale/zh_TW.UTF-8/zh_Hant_TW.UTF-8 -> zh_Hant_TW.UTF-8
/usr/share/locale/zh_TW.UTF-8/LC_CTYPE -> ../UTF-8/LC_CTYPE
/usr/share/locale/de_AT.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/en_GB.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/en_GB.ISO8859-1/LC_COLLATE ->
../la_LN.ISO8859-1/LC_COLLATE
/usr/share/locale/ca_ES.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/zh_CN.GBK/zh_Hans_CN.GBK -> zh_Hans_CN.GBK
/usr/share/locale/zh_CN.GBK/LC_COLLATE -> ../la_LN.US-ASCII/LC_COLLATE
/usr/share/locale/de_DE.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/ca_IT.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/fi_FI.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
/usr/share/locale/fi_FI.ISO8859-1/LC_COLLATE ->
../la_LN.ISO8859-1/LC_COLLATE
/usr/share/locale/zh_CN.eucCN/zh_Hans_CN.eucCN -> zh_Hans_CN.eucCN
/usr/share/locale/zh_CN.eucCN/LC_COLLATE -> ../la_LN.US-ASCII/LC_COLLATE
/usr/share/locale/zh_HK.UTF-8/LC_COLLATE -> ../la_LN.US-ASCII/LC_COLLATE
/usr/share/locale/zh_HK.UTF-8/LC_CTYPE -> ../UTF-8/LC_CTYPE
/usr/share/locale/sr_YU.UTF-8/LC_COLLATE -> ../la_LN.US-ASCII/LC_COLLATE
/usr/share/locale/sr_YU.UTF-8/LC_CTYPE -> ../UTF-8/LC_CTYPE
/usr/share/locale/zh_CN.GB18030/zh_Hans_CN.GB18030 -> zh_Hans_CN.GB18030
/usr/share/locale/zh_CN.GB18030/LC_COLLATE -> ../la_LN.US-ASCII/LC_COLLATE
/usr/share/locale/fr_CA.ISO8859-15/LC_CTYPE -> ../la_LN.ISO8859-15/LC_CTYPE

Re: Need help fixing failing locale tests

2015-11-15 Thread John Marino 
On 11/15/2015 8:24 AM, NGie Cooper wrote:
>> On Nov 14, 2015, at 23:09, John Marino  wrote:
>> Bapt liked DF approach well enough that he adopted it.  Even Edwin was
>> first in desiring to clean up locales.  The major update was a perfect time.
>>
>> Bottom line:
>> The testsuite needs to be updated.
>> e.g. use de_DE.ISO8859-15 intead of de_DE.ISO8859-1
>> For latin, replace with US-ASCII equivalent.
> 
>   I wish this had been clearly communicated here: 
> https://svnweb.freebsd.org/changeset/base/290494 … instead, there are some 
> POLA violations:
>   1. No RelNotes.
>   2. No summary of locales that have been removed.
>   3. No UPDATING entry for people to migrate from a locale to another.
>   4. Deprecated locales (as you described it above) are still present on 
> my system after running `make delete-old` (assuming I have my acronyms right, 
> for example, ca_ES, it_CH, etc should be -1, not -15 based on your claim 
> above).
>   I will update the testcases to use locales if possible, but more work 
> needs to be done finishing off this project and documenting for end-users 
> what has changed.

With 4.) they have been removed from FreeBSD when "Make upgrade" is run
after rebuilding.  Does FreeBSD have a cleanup like that, and did you
run it?  If not, then maybe that needs to be updated.

with 3.) That's a bit like saying changing the color of shoe laces needs
instructions for people to learn how to adjust.  Run "locale -a" to see
what's available on the system.  This has always been true.  Attempting
to switch to a locale that doesn't exist results in staying in "C"
locale, which is quite obvious.

with 2.) Could be combined with 1.) but in any case, there really
shouldn't be POLA on anything that has no standard.  Should you announce
every time you add a locale too, in release notes?
Really "locale -a" should tell the whole story.

1.) A paragraph on removed locales is fine.  Those removed have been
removed because they are a) likely unused and b) if used, they are used
by mistake.  in this case you hit c) where the testsuite probably chose
the "wrong" locale in the first place.  Note that GCC did this too with
their libstdc++ testsuite and that was fixed this weekend (a huge patch
to GCC 6 upstream).

John
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread John Marino 
On 11/15/2015 12:23 PM, Craig Rodrigues wrote:
> 
> 
> On Sat, Nov 14, 2015 at 11:47 PM, John Marino  > wrote:
> 
> 
> With 4.) they have been removed from FreeBSD when "Make upgrade" is run
> after rebuilding.  Does FreeBSD have a cleanup like that, and did you
> run it?  If not, then maybe that needs to be updated.

by they way, this was meant to be "removed from DragonFly".  I don't
know if "make upgrade" is set right for FreeBSD.  (it appears not)

> 
> I did not run "make upgrade".
> I did this:
> cd /usr/src
> svn up
> 
> I then ran the 11 steps listed in /usr/src/Makefile:
> 
> https://github.com/freebsd/freebsd/blob/master/Makefile#L73
> 
> It looks like after running those steps, my system is in an inconsistent
> state.  I see that the following symlinks point to nonexistent files.

[snip]

It looks like the deleted locales were not removed.
They should be.


> What is the fix for this?

I think somebody needs to add these locales to be reaped by make upgrade.


> It looks like there is some Makefile logic which is wrong.


John


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
On 15.11.2015 15:24, Andrey Chernov wrote:
> On 15.11.2015 10:09, John Marino wrote:
>> We (DragonFly) didn't just update locales.  We took the opportunity to
>> do spring cleaning.  We didn't want to be as drastic as OpenBSD which
>> removed all encodings except for C/POSIX and UTF, but we did remove
>> several locales intentionally.
>>
>> In the case of ISO8859-1:
>> All ISO8859-* is basically obsolete.
>> In western Europe, if somebody wants ISO-8859, they want ISO8859-15, not
>> ISO8859-1.  They are similar, but the former is tailored for western
>> europe with "Euro" currency and 9 other symbols.  It comes at the
>> expense of removing 10 characters from ISO8859-1. 

Forget to reply on that part:

>> There's also a common
>> problem that users view -15 documents with -1 accidently.  So there was
>> a conscience decision to have either ISO8859-1 or ISO8859-15 but not
>> both.  For western Europe this means the ISO8859-1 versions were dropped.

It is pure user problem choosing its own locale (self footshooting), it
not leads to program build failures (like configure checks) or tests
failures etc. BTW, linux keeps 8859-1 locales.

> ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
> with that). Lots of ports (even at configure stage!) have checks for
> them. Since we generate locales from CLDR now, it will be no cost to
> bring all 8859-1 back to not violate POLA and not fix every failing port.

-- 
http://ache.vniz.net/
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
On 15.11.2015 15:46, Baptiste Daroussin wrote:
> On Sun, Nov 15, 2015 at 03:24:19PM +0300, Andrey Chernov wrote:
>> On 15.11.2015 10:09, John Marino wrote:
>> ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
>> with that). Lots of ports (even at configure stage!) have checks for
>> them. Since we generate locales from CLDR now, it will be no cost to
>> bring all 8859-1 back to not violate POLA and not fix every failing port.
>>
> Exp-run have been made and no ports were failing with the removed locales.

There is soft-fail, configure just marks that locales are not supported
and use "C". Sorry I don't remember port names where I saw it right now
and don't have a time to search for them right now too. Soft-fails (like
in tcl with nl_langinfo) are almost impossible to detect excepting
specific situation happens or source code inspection. Do we ever need
them when there is no harm to keep 8859-1 locales?

-- 
http://ache.vniz.net/



signature.asc
Description: OpenPGP digital signature

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
On 15.11.2015 16:00, Baptiste Daroussin wrote:
> On Sun, Nov 15, 2015 at 03:56:26PM +0300, Andrey Chernov wrote:
>> On 15.11.2015 15:46, Baptiste Daroussin wrote:
>>> On Sun, Nov 15, 2015 at 03:24:19PM +0300, Andrey Chernov wrote:
 On 15.11.2015 10:09, John Marino wrote:
 ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
 with that). Lots of ports (even at configure stage!) have checks for
 them. Since we generate locales from CLDR now, it will be no cost to
 bring all 8859-1 back to not violate POLA and not fix every failing port.

>>> Exp-run have been made and no ports were failing with the removed locales.
>>
>> There is soft-fail, configure just marks that locales are not supported
>> and use "C". Sorry I don't remember port names where I saw it right now
>> and don't have a time to search for them right now too. Soft-fails (like
>> in tcl with nl_langinfo) are almost impossible to detect excepting
>> specific situation happens or source code inspection. Do we ever need
>> them when there is no harm to keep 8859-1 locales?
> 
> Is it ok if I readd those locales as aliases on 8859-15?

It is hacking solution leads to wrong collating order and character
classes. It is better to generate true 8859-1 just in the same way you
already do for 8859-15.

BTW, I can't check right now, but in case 8859-5 is removed too, it is
better to restore it, it was used in Suns as their standard Russian
encoding.

-- 
http://ache.vniz.net/



signature.asc
Description: OpenPGP digital signature

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
On 15.11.2015 10:09, John Marino wrote:
> We (DragonFly) didn't just update locales.  We took the opportunity to
> do spring cleaning.  We didn't want to be as drastic as OpenBSD which
> removed all encodings except for C/POSIX and UTF, but we did remove
> several locales intentionally.
> 
> In the case of ISO8859-1:
> All ISO8859-* is basically obsolete.
> In western Europe, if somebody wants ISO-8859, they want ISO8859-15, not
> ISO8859-1.  They are similar, but the former is tailored for western
> europe with "Euro" currency and 9 other symbols.  It comes at the
> expense of removing 10 characters from ISO8859-1.  There's also a common
> problem that users view -15 documents with -1 accidently.  So there was
> a conscience decision to have either ISO8859-1 or ISO8859-15 but not
> both.  For western Europe this means the ISO8859-1 versions were dropped.

ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
with that). Lots of ports (even at configure stage!) have checks for
them. Since we generate locales from CLDR now, it will be no cost to
bring all 8859-1 back to not violate POLA and not fix every failing port.

-- 
http://ache.vniz.net/
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Baptiste Daroussin 
On Sun, Nov 15, 2015 at 03:24:19PM +0300, Andrey Chernov wrote:
> On 15.11.2015 10:09, John Marino wrote:
> > We (DragonFly) didn't just update locales.  We took the opportunity to
> > do spring cleaning.  We didn't want to be as drastic as OpenBSD which
> > removed all encodings except for C/POSIX and UTF, but we did remove
> > several locales intentionally.
> > 
> > In the case of ISO8859-1:
> > All ISO8859-* is basically obsolete.
> > In western Europe, if somebody wants ISO-8859, they want ISO8859-15, not
> > ISO8859-1.  They are similar, but the former is tailored for western
> > europe with "Euro" currency and 9 other symbols.  It comes at the
> > expense of removing 10 characters from ISO8859-1.  There's also a common
> > problem that users view -15 documents with -1 accidently.  So there was
> > a conscience decision to have either ISO8859-1 or ISO8859-15 but not
> > both.  For western Europe this means the ISO8859-1 versions were dropped.
> 
> ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
> with that). Lots of ports (even at configure stage!) have checks for
> them. Since we generate locales from CLDR now, it will be no cost to
> bring all 8859-1 back to not violate POLA and not fix every failing port.
> 
Exp-run have been made and no ports were failing with the removed locales.

Best regards,
Bapt


signature.asc
Description: PGP signature

Re: Need help fixing failing locale tests

2015-11-15 Thread Baptiste Daroussin 
On Sun, Nov 15, 2015 at 03:56:26PM +0300, Andrey Chernov wrote:
> On 15.11.2015 15:46, Baptiste Daroussin wrote:
> > On Sun, Nov 15, 2015 at 03:24:19PM +0300, Andrey Chernov wrote:
> >> On 15.11.2015 10:09, John Marino wrote:
> >> ISO8859-1 locales are legacy even if obsoleted in modern world (I agree
> >> with that). Lots of ports (even at configure stage!) have checks for
> >> them. Since we generate locales from CLDR now, it will be no cost to
> >> bring all 8859-1 back to not violate POLA and not fix every failing port.
> >>
> > Exp-run have been made and no ports were failing with the removed locales.
> 
> There is soft-fail, configure just marks that locales are not supported
> and use "C". Sorry I don't remember port names where I saw it right now
> and don't have a time to search for them right now too. Soft-fails (like
> in tcl with nl_langinfo) are almost impossible to detect excepting
> specific situation happens or source code inspection. Do we ever need
> them when there is no harm to keep 8859-1 locales?

Is it ok if I readd those locales as aliases on 8859-15?

Best regards,
Bapt


signature.asc
Description: PGP signature

Re: Need help fixing failing locale tests

2015-11-15 Thread Andrey Chernov 
> Well, there is "harm".  The -1/-15 confusion happens a lot.

It is user confusion and his responsibility. It not leads to wrong
program build f.e. Moreover, you can't protect users who set 8859-1 that
way, they do not convert to 8859-15 as you assume but start to complain
everywhere that FreeBSD is not working instead.

> Is the plan to keep every locale ever created for ever and ever?  Never
> do any kind of kind of cleanup or reorg?

It will be nice to do it that way. FreeBSD have a little part of world
locales, which indirectly assumes that they are really used.

-- 
http://ache.vniz.net/
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread Simon J. Gerraty 
Craig Rodrigues  wrote:
> I don't know much about locales, so don't know what to do.

I find LANG=C LC_ALL=C solves most locale induced issues.
I suspect the tests in question assume the above anyway.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread John Marino 
On 11/15/2015 4:38 PM, Andrey Chernov wrote:
> On 15.11.2015 18:17, John Marino wrote:
>>
>> For the traditional checks, It's ironic, DragonFly uses short locales
>> like "fr_FR" which link to the approprioprate ISO8859 or UTF-8 locale.
>> Bapt removed them to avoid a bike shed and if he had not done that, this
>> gettext configure would not be an issue as "fr_FR" is the very first check.
> 
> All parts are optional excepting language itself, so fr_FR is not short
> enough, but fr is. From POSIX:
> 
> language[_territory][.codeset][@modifier]

"fr" (or "french") by itself isn't very useful, so IMO
language_territory is the minimum.

> But I dislike short locales due to their uncertainty and remember some
> programs and shell scripts that attempts to parse locale name directly
> by itself for reasons I don't remember now.

Absolutely true about the uncertainty.  It's just subjective.  We did
what we thought made sense but bapt recognized an impeding bikeshed and
just did away with it.

For some reason gettext-tools only cares about french and japanese (thus
easily patched).  Here's what it looks like on current DragonFly:

http://muscles.dragonflybsd.org/bulk/bleeding-edge-potential/latest-per-pkg/gettext-tools-0.19.6.log

Note the identified traditional french is "fr_FR" which maps to
fr_FR.ISO8859-15 on DragonFly.

It's a moot point now I guess.  ISO8859-1 has returned to FreeBSD for
western europe (probably forever).

John




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-15 Thread John Marino 
On 11/15/2015 3:59 PM, Andrey Chernov wrote:
> On 15.11.2015 17:47, John Marino wrote:
 Please provide an example of such a program (in ports).
>>>
>>> See gettext-0.19.6/gettext-tools/configure, part started with
>>> # Test for the usual locale name.
>>> I don't have time to dig more such code now, but I remember I saw more
>>> for sure.
> 
>> If the ports framework is not overriding locale to "C" during the build
>> then that's probably something that should be introduced.  Ports should
>> not be producing different results based on the configuration of the
>> building machine at the time.
> 
> This is not about resetting locale to "C" during the build. Please
> really look at "configure" specific part mentioned above, it is you who
> ask at least one specific example and then it is you who ignore the
> answer takes my time to find it.

Hmmm?
Of course it's the same topic.
If the configure of a port is affected by the locale, that's a big
problem.  How is this not the same topic?

But I did look at gettext, which is configuring package based on what's
on the system (not environment).  For the unicode checks, there is
obviously no issue.

For the traditional checks, It's ironic, DragonFly uses short locales
like "fr_FR" which link to the approprioprate ISO8859 or UTF-8 locale.
Bapt removed them to avoid a bike shed and if he had not done that, this
gettext configure would not be an issue as "fr_FR" is the very first check.

John

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-14 Thread NGie Cooper 

> On Nov 14, 2015, at 19:18, Craig Rodrigues  wrote:
> 
> On Sat, Nov 14, 2015 at 7:05 PM, Craig Rodrigues 
> wrote:
> 
>> Hi,
>> 
>> After the recent locale commits, some of the tests are failing:
>> 
>> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1675/testReport/
>> 
>> I can reproduce two failures quite easily by doing with a newly built
>> world:
>> 
>> /bin/sh /usr/tests/bin/sh/builtins/case7.0
>> /bin/sh /usr/tests/bin/sh/builtins/locale1.0
>> 
>> Can someone look into this and help fix this?
>> 
>> I don't know much about locales, so don't know what to do.
>> 
> 
> 
> I ran the two tests using bash, and got different error messages:
> 
> /usr/local/bin/bash /usr/tests/bin/sh/builtins/case7.0
> /usr/tests/bin/sh/builtins/case7.0: line 7: warning: setlocale: LC_CTYPE:
> cannot change locale (de_DE.ISO8859-1): Invalid argument
> /usr/tests/bin/sh/builtins/case7.0: line 9: warning: setlocale: LC_COLLATE:
> cannot change locale (de_DE.ISO8859-1): Invalid argument
> wrong at 18
> wrong at 23
> 
> /usr/local/bin/bash /usr/tests/bin/sh/builtins/locale1.0
> /usr/tests/bin/sh/builtins/locale1.0: line 45: warning: setlocale:
> LC_CTYPE: cannot change locale (nl_NL.ISO8859-1): Invalid argument
> Failed: $ok -eq 1 at 56
> /usr/tests/bin/sh/builtins/locale1.0: line 64: warning: setlocale: LC_ALL:
> cannot change locale (nl_NL.ISO8859-1): No such file or directory
> Failed: $ok -eq 1 at 68
> Failed: $ok -eq 1 at 74
> /usr/tests/bin/sh/builtins/locale1.0: regel 82: waarschuwing: setlocale():
> LC_ALL: kan niet van taalregio veranderen (nl_NL.ISO8859-1): No such file
> or directory
> Failed: $ok -eq 1 at 86
> Failed: $ok -eq 1 at 99
> /usr/tests/bin/sh/builtins/locale1.0: regel 107: waarschuwing: setlocale():
> LC_ALL: kan niet van taalregio veranderen (nl_NL.ISO8859-1): No such file
> or directory
> Failed: $ok -eq 1 at 111
> /usr/tests/bin/sh/builtins/locale1.0: regel 114: waarschuwing: setlocale():
> LC_ALL: kan niet van taalregio veranderen (nl_NL.ISO8859-1): No such file
> or directory
> Failed: $ok -eq 1 at 118
> /usr/tests/bin/sh/builtins/locale1.0: regel 122: waarschuwing: setlocale():
> LC_ALL: kan niet van taalregio veranderen (nl_NL.ISO8859-1): No such file
> or directory
> /usr/tests/bin/sh/builtins/locale1.0: regel 128: waarschuwing: setlocale():
> LC_ALL: kan niet van taalregio veranderen (nl_NL.ISO8859-1): No such file
> or directory
> Failed: $ok -eq 1 at 132
> 
> 
> On my system, I did:
> ls -l /usr/share/locale/de_DE.ISO8859-1/*
> -r--r--r--  1 root  wheel  4642 Nov  6 12:53
> /usr/share/locale/de_DE.ISO8859-1/LC_COLLATE
> lrwxr-xr-x  1 root  wheel27 Nov  6 12:53
> /usr/share/locale/de_DE.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
> -r--r--r--  1 root  wheel18 Nov  6 12:53
> /usr/share/locale/de_DE.ISO8859-1/LC_MESSAGES
> -r--r--r--  1 root  wheel35 Nov  6 12:53
> /usr/share/locale/de_DE.ISO8859-1/LC_MONETARY
> -r--r--r--  1 root  wheel 6 Nov  6 12:53
> /usr/share/locale/de_DE.ISO8859-1/LC_NUMERIC
> -r--r--r--  1 root  wheel   367 Nov  6 12:53
> /usr/share/locale/de_DE.ISO8859-1/LC_TIME
> 
> ls -l /usr/share/locale/nl_NL.ISO8859-1/*
> lrwxr-xr-x  1 root  wheel   29 Nov  6 12:53
> /usr/share/locale/nl_NL.ISO8859-1/LC_COLLATE ->
> ../la_LN.ISO8859-1/LC_COLLATE
> lrwxr-xr-x  1 root  wheel   27 Nov  6 12:53
> /usr/share/locale/nl_NL.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
> -r--r--r--  1 root  wheel   18 Nov  6 12:53
> /usr/share/locale/nl_NL.ISO8859-1/LC_MESSAGES
> -r--r--r--  1 root  wheel   35 Nov  6 12:53
> /usr/share/locale/nl_NL.ISO8859-1/LC_MONETARY
> -r--r--r--  1 root  wheel6 Nov  6 12:53
> /usr/share/locale/nl_NL.ISO8859-1/LC_NUMERIC
> -r--r--r--  1 root  wheel  376 Nov  6 12:53
> /usr/share/locale/nl_NL.ISO8859-1/LC_TIME
> 
> I saw that la_LN.ISO8859-1 does not exist, so the LC_CTYPE symlink is
> pointing to  nothing.

Why were these locales removed?

  58 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_COLLATE
  59 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_CTYPE
  60 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_TIME
  61 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-1
  62 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_COLLATE
  63 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_CTYPE
  64 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-13

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-14 Thread NGie Cooper 

> On Nov 14, 2015, at 19:28, NGie Cooper  wrote:

…

> Why were these locales removed?
> 
>  58 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_COLLATE
>  59 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_CTYPE
>  60 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_TIME
>  61 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-1
>  62 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_COLLATE
>  63 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_CTYPE
>  64 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-13

la_LN.ISO8859-1 is the old Latin locale, which is no longer installed. Copying 
over locale files from a stable/10 host works, but I’m confused as to why a 
bunch of locales weren’t ported over in their non-UTF-8 forms.
Thanks,
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-14 Thread NGie Cooper 

> On Nov 14, 2015, at 23:09, John Marino  wrote:
> 
> On 11/15/2015 4:46 AM, NGie Cooper wrote:
>> 
>>> On Nov 14, 2015, at 19:28, NGie Cooper  wrote:
>> 
>> …
>> 
>>> Why were these locales removed?
>>> 
>>> 58 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_COLLATE
>>> 59 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_CTYPE
>>> 60 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_TIME
>>> 61 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-1
>>> 62 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_COLLATE
>>> 63 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_CTYPE
>>> 64 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-13
>> 
>> la_LN.ISO8859-1 is the old Latin locale, which is no longer installed. 
>> Copying over locale files from a stable/10 host works, but I’m confused as 
>> to why a bunch of locales weren’t ported over in their non-UTF-8 forms.
>> Thanks,
>> 
> 
> We (DragonFly) didn't just update locales.  We took the opportunity to
> do spring cleaning.  We didn't want to be as drastic as OpenBSD which
> removed all encodings except for C/POSIX and UTF, but we did remove
> several locales intentionally.
> 
> In the case of ISO8859-1:
> All ISO8859-* is basically obsolete.
> In western Europe, if somebody wants ISO-8859, they want ISO8859-15, not
> ISO8859-1.  They are similar, but the former is tailored for western
> europe with "Euro" currency and 9 other symbols.  It comes at the
> expense of removing 10 characters from ISO8859-1.  There's also a common
> problem that users view -15 documents with -1 accidently.  So there was
> a conscience decision to have either ISO8859-1 or ISO8859-15 but not
> both.  For western Europe this means the ISO8859-1 versions were dropped.
> 
> ISO8859-15:
> In the case of USA and other non-European countries, they keep -1 and
> dropped -15.
> 
> currency based:
> In the case of countries where the currency symbols is not part of
> ISO8859, it was dropped.  E.g. Costa Rica uses the Colon which is only
> in UTF-8, so there's no ISO8859-* encoding at all for CR.
> 
> Latin:
> Who speaks Latin today?
> This was mainly an alias for 7-bit ascii.  We originally dropped that,
> but later moved it to US-ASCII (which was just a symlink to Latin before)
> 
> Bapt liked DF approach well enough that he adopted it.  Even Edwin was
> first in desiring to clean up locales.  The major update was a perfect time.
> 
> Bottom line:
> The testsuite needs to be updated.
> e.g. use de_DE.ISO8859-15 intead of de_DE.ISO8859-1
> For latin, replace with US-ASCII equivalent.

I wish this had been clearly communicated here: 
https://svnweb.freebsd.org/changeset/base/290494 … instead, there are some POLA 
violations:
1. No RelNotes.
2. No summary of locales that have been removed.
3. No UPDATING entry for people to migrate from a locale to another.
4. Deprecated locales (as you described it above) are still present on 
my system after running `make delete-old` (assuming I have my acronyms right, 
for example, ca_ES, it_CH, etc should be -1, not -15 based on your claim above).
I will update the testcases to use locales if possible, but more work 
needs to be done finishing off this project and documenting for end-users what 
has changed.
Thanks,
-NGie
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Need help fixing failing locale tests

2015-11-14 Thread Craig Rodrigues 
Hi,

After the recent locale commits, some of the tests are failing:

https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1675/testReport/

I can reproduce two failures quite easily by doing with a newly built world:

/bin/sh /usr/tests/bin/sh/builtins/case7.0
/bin/sh /usr/tests/bin/sh/builtins/locale1.0

Can someone look into this and help fix this?

I don't know much about locales, so don't know what to do.

--
Craig
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-14 Thread NGie Cooper 

> On Nov 14, 2015, at 19:46, NGie Cooper  wrote:
> 
> 
>> On Nov 14, 2015, at 19:28, NGie Cooper  wrote:
> 
> …
> 
>> Why were these locales removed?
>> 
>> 58 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_COLLATE
>> 59 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_CTYPE
>> 60 OLD_FILES+=usr/share/locale/la_LN.ISO8859-1/LC_TIME
>> 61 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-1
>> 62 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_COLLATE
>> 63 OLD_FILES+=usr/share/locale/la_LN.ISO8859-13/LC_CTYPE
>> 64 OLD_DIRS+=usr/share/locale/la_LN.ISO8859-13
> 
> la_LN.ISO8859-1 is the old Latin locale, which is no longer installed. 
> Copying over locale files from a stable/10 host works, but I’m confused as to 
> why a bunch of locales weren’t ported over in their non-UTF-8 forms.
> Thanks,

/usr/share/locale/en_US.ISO8859-15/LC_CTYPE is also broken; I opened up the 
file and it was empty.
Thanks,
-NGie
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-14 Thread Craig Rodrigues 
On Sat, Nov 14, 2015 at 7:05 PM, Craig Rodrigues 
wrote:

> Hi,
>
> After the recent locale commits, some of the tests are failing:
>
> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1675/testReport/
>
> I can reproduce two failures quite easily by doing with a newly built
> world:
>
> /bin/sh /usr/tests/bin/sh/builtins/case7.0
> /bin/sh /usr/tests/bin/sh/builtins/locale1.0
>
> Can someone look into this and help fix this?
>
> I don't know much about locales, so don't know what to do.
>


I ran the two tests using bash, and got different error messages:

/usr/local/bin/bash /usr/tests/bin/sh/builtins/case7.0
/usr/tests/bin/sh/builtins/case7.0: line 7: warning: setlocale: LC_CTYPE:
cannot change locale (de_DE.ISO8859-1): Invalid argument
/usr/tests/bin/sh/builtins/case7.0: line 9: warning: setlocale: LC_COLLATE:
cannot change locale (de_DE.ISO8859-1): Invalid argument
wrong at 18
wrong at 23

/usr/local/bin/bash /usr/tests/bin/sh/builtins/locale1.0
/usr/tests/bin/sh/builtins/locale1.0: line 45: warning: setlocale:
LC_CTYPE: cannot change locale (nl_NL.ISO8859-1): Invalid argument
Failed: $ok -eq 1 at 56
/usr/tests/bin/sh/builtins/locale1.0: line 64: warning: setlocale: LC_ALL:
cannot change locale (nl_NL.ISO8859-1): No such file or directory
Failed: $ok -eq 1 at 68
Failed: $ok -eq 1 at 74
/usr/tests/bin/sh/builtins/locale1.0: regel 82: waarschuwing: setlocale():
LC_ALL: kan niet van taalregio veranderen (nl_NL.ISO8859-1): No such file
or directory
Failed: $ok -eq 1 at 86
Failed: $ok -eq 1 at 99
/usr/tests/bin/sh/builtins/locale1.0: regel 107: waarschuwing: setlocale():
LC_ALL: kan niet van taalregio veranderen (nl_NL.ISO8859-1): No such file
or directory
Failed: $ok -eq 1 at 111
/usr/tests/bin/sh/builtins/locale1.0: regel 114: waarschuwing: setlocale():
LC_ALL: kan niet van taalregio veranderen (nl_NL.ISO8859-1): No such file
or directory
Failed: $ok -eq 1 at 118
/usr/tests/bin/sh/builtins/locale1.0: regel 122: waarschuwing: setlocale():
LC_ALL: kan niet van taalregio veranderen (nl_NL.ISO8859-1): No such file
or directory
/usr/tests/bin/sh/builtins/locale1.0: regel 128: waarschuwing: setlocale():
LC_ALL: kan niet van taalregio veranderen (nl_NL.ISO8859-1): No such file
or directory
Failed: $ok -eq 1 at 132


On my system, I did:
ls -l /usr/share/locale/de_DE.ISO8859-1/*
-r--r--r--  1 root  wheel  4642 Nov  6 12:53
/usr/share/locale/de_DE.ISO8859-1/LC_COLLATE
lrwxr-xr-x  1 root  wheel27 Nov  6 12:53
/usr/share/locale/de_DE.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
-r--r--r--  1 root  wheel18 Nov  6 12:53
/usr/share/locale/de_DE.ISO8859-1/LC_MESSAGES
-r--r--r--  1 root  wheel35 Nov  6 12:53
/usr/share/locale/de_DE.ISO8859-1/LC_MONETARY
-r--r--r--  1 root  wheel 6 Nov  6 12:53
/usr/share/locale/de_DE.ISO8859-1/LC_NUMERIC
-r--r--r--  1 root  wheel   367 Nov  6 12:53
/usr/share/locale/de_DE.ISO8859-1/LC_TIME

ls -l /usr/share/locale/nl_NL.ISO8859-1/*
lrwxr-xr-x  1 root  wheel   29 Nov  6 12:53
/usr/share/locale/nl_NL.ISO8859-1/LC_COLLATE ->
../la_LN.ISO8859-1/LC_COLLATE
lrwxr-xr-x  1 root  wheel   27 Nov  6 12:53
/usr/share/locale/nl_NL.ISO8859-1/LC_CTYPE -> ../la_LN.ISO8859-1/LC_CTYPE
-r--r--r--  1 root  wheel   18 Nov  6 12:53
/usr/share/locale/nl_NL.ISO8859-1/LC_MESSAGES
-r--r--r--  1 root  wheel   35 Nov  6 12:53
/usr/share/locale/nl_NL.ISO8859-1/LC_MONETARY
-r--r--r--  1 root  wheel6 Nov  6 12:53
/usr/share/locale/nl_NL.ISO8859-1/LC_NUMERIC
-r--r--r--  1 root  wheel  376 Nov  6 12:53
/usr/share/locale/nl_NL.ISO8859-1/LC_TIME

I saw that la_LN.ISO8859-1 does not exist, so the LC_CTYPE symlink is
pointing to  nothing.

--
Craig
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Need help fixing failing locale tests

2015-11-14 Thread NGie Cooper 

> On Nov 14, 2015, at 19:05, Craig Rodrigues  wrote:
> 
> Hi,
> 
> After the recent locale commits, some of the tests are failing:
> 
> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1675/testReport/
> 
> I can reproduce two failures quite easily by doing with a newly built world:
> 
> /bin/sh /usr/tests/bin/sh/builtins/case7.0
> /bin/sh /usr/tests/bin/sh/builtins/locale1.0
> 
> Can someone look into this and help fix this?

bapt’s locale work looks like it broke the following locales with /bin/sh:

de_DE.ISO8859-1 [LC_CTYPE,LC_COLLATE]
nl_NL.ISO8859-1

> I don't know much about locales, so don't know what to do.

I’ll look at the other failing tests.

-NGie
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"