Re: awk(1) is locale unaware (was: Re: buildworld breakage during make depend at usr.bin/kdump)
On Wed, Nov 20, 2002 at 14:54:12 +0200, Ruslan Ermilov wrote: Index: b.c === RCS file: /home/ncvs/src/contrib/one-true-awk/b.c,v retrieving revision 1.1.1.2 diff -u -p -r1.1.1.2 b.c David, this variant is nice enough. Please, commit. -- Andrey A. Chernov http://ache.pp.ru/ msg48017/pgp0.pgp Description: PGP signature
Re: awk(1) is locale unaware (was: Re: buildworld breakage during make depend at usr.bin/kdump)
On Tue, Dec 03, 2002 at 11:09:20AM +0300, Andrey A. Chernov wrote: On Wed, Nov 20, 2002 at 14:54:12 +0200, Ruslan Ermilov wrote: Index: b.c === RCS file: /home/ncvs/src/contrib/one-true-awk/b.c,v retrieving revision 1.1.1.2 diff -u -p -r1.1.1.2 b.c David, this variant is nice enough. Please, commit. One needs to catch up to his email. :-) A new version of one-true-awk was released which includes these fixes. Cheers, -- Ruslan Ermilov Sysadmin and DBA, [EMAIL PROTECTED] Sunbay Software AG, [EMAIL PROTECTED] FreeBSD committer, +380.652.512.251Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age msg48021/pgp0.pgp Description: PGP signature
awk(1) is locale unaware (was: Re: buildworld breakage during make depend at usr.bin/kdump)
On Thu, Nov 01, 2001 at 05:58:08PM -0800, David O'Brien wrote:
On Fri, Nov 02, 2001 at 04:44:12AM +0300, Andrey A. Chernov wrote:
Next bad thing discovered about new awk just looking at sourse code: it
not support locale (collating in regexp ranges too, of course). We just
make great backward step switching to it.
I have a patch for that.
It seems that this patch has never been committed. This is a critical
bug that should be fixed before 5.0-RELEASE is out.
/usr/bin/env LC_ALL=cs_CZ.ISO8859-2 sh -c echo a | grep '[A-Z]'
/usr/bin/env LC_ALL=cs_CZ.ISO8859-2 sh -c echo a | awk '/[A-Z]/ {print}'
On a related note, fixing this bug would extrapolate PR misc/45460 to
5.0-CURRENT as well.
Cheers,
--
Ruslan Ermilov Sysadmin and DBA,
[EMAIL PROTECTED] Sunbay Software AG,
[EMAIL PROTECTED] FreeBSD committer,
+380.652.512.251Simferopol, Ukraine
http://www.FreeBSD.org The Power To Serve
http://www.oracle.com Enabling The Information Age
msg46927/pgp0.pgp
Description: PGP signature
Re: awk(1) is locale unaware (was: Re: buildworld breakage during make depend at usr.bin/kdump)
On Tue, Nov 19, 2002 at 14:52:02 +0200, Ruslan Ermilov wrote: It seems that this patch has never been committed. This is a critical bug that should be fixed before 5.0-RELEASE is out. I agree. There is no locale yet and I never see that patch. -- Andrey A. Chernov http://ache.pp.ru/ msg46960/pgp0.pgp Description: PGP signature
Re: awk(1) is locale unaware (was: Re: buildworld breakage during make depend at usr.bin/kdump)
On Wed, Nov 20, 2002 at 04:38:38AM +0300, Andrey A. Chernov wrote:
On Tue, Nov 19, 2002 at 14:52:02 +0200, Ruslan Ermilov wrote:
It seems that this patch has never been committed. This is a critical
bug that should be fixed before 5.0-RELEASE is out.
I agree. There is no locale yet and I never see that patch.
This patch seems to work, I used the logic from regcomp.c in libc.
Long lines make it ugly, but it was like that when I got here ;)
Tim
Index: src/usr.bin/awk/Makefile
===
RCS file: /x/freebsd/src/usr.bin/awk/Makefile,v
retrieving revision 1.9
diff -u -r1.9 Makefile
--- src/usr.bin/awk/Makefile10 May 2002 20:36:21 - 1.9
+++ src/usr.bin/awk/Makefile20 Nov 2002 03:13:50 -
@@ -6,7 +6,7 @@
PROG= nawk
SRCS= awkgram.y b.c lex.c lib.c main.c parse.c proctab.c run.c tran.c ytab.h
-CFLAGS+= -I. -I${AWKSRC}
+CFLAGS+= -I. -I${AWKSRC} -I${.CURDIR}/../../lib/libc/locale
DPADD= ${LIBM}
LDADD= -lm
Index: src/contrib/one-true-awk/b.c
===
RCS file: /x/freebsd/src/contrib/one-true-awk/b.c,v
retrieving revision 1.1.1.2
diff -u -r1.1.1.2 b.c
--- src/contrib/one-true-awk/b.c19 Feb 2002 09:35:24 - 1.1.1.2
+++ src/contrib/one-true-awk/b.c20 Nov 2002 03:16:10 -
@@ -32,6 +32,7 @@
#include stdlib.h
#include awk.h
#include ytab.h
+#include collate.h
#defineHAT (NCHARS-2) /* matches ^ in regular expr */
/* NCHARS is 2**n */
@@ -284,7 +285,7 @@
char *cclenter(char *argp) /* add a character class */
{
- int i, c, c2;
+ int i, j, c, c2;
uschar *p = (uschar *) argp;
uschar *op, *bp;
static uschar *buf = 0;
@@ -308,12 +309,24 @@
i--;
continue;
}
- while (c c2) {
- if (!adjbuf((char **) buf, bufsz, bp-buf+2,
100, (char **) bp, 0))
- FATAL(out of space for character
class [%.10s...] 2, p);
- *bp++ = ++c;
- i++;
- }
+ if (__collate_load_error) {
+ while (c c2) {
+ if (!adjbuf((char **) buf, bufsz,
+bp-buf+2, 100, (char **) bp, 0))
+ FATAL(out of space for
+character class [%.10s...] 2, p);
+ *bp++ = ++c;
+ i++;
+ }
+ } else {
+ for (j = CHAR_MIN; j = CHAR_MAX; j++) {
+ if (!adjbuf((char **) buf, bufsz,
+bp-buf+2, 100, (char **) bp, 0))
+ FATAL(out of space for
+character class [%.10s...] 2, p);
+ if (__collate_range_cmp(c, j) = 0
+__collate_range_cmp(j, c2) =
+0) {
+ *bp++ = j;
+ i++;
+ }
+ }
+}
continue;
}
}
Index: src/contrib/one-true-awk/main.c
===
RCS file: /x/freebsd/src/contrib/one-true-awk/main.c,v
retrieving revision 1.1.1.3
diff -u -r1.1.1.3 main.c
--- src/contrib/one-true-awk/main.c 16 Mar 2002 16:50:56 - 1.1.1.3
+++ src/contrib/one-true-awk/main.c 20 Nov 2002 03:03:38 -
@@ -27,6 +27,7 @@
#define DEBUG
#include stdio.h
#include ctype.h
+#include locale.h
#include stdlib.h
#include string.h
#include signal.h
@@ -55,6 +56,7 @@
char *fs = NULL;
cmdname = argv[0];
+ setlocale(LC_ALL, );
if (argc == 1) {
fprintf(stderr, Usage: %s [-f programfile | 'program'] [-Ffieldsep]
[-v var=value] [files]\n, cmdname);
exit(1);
Index: src/contrib/one-true-awk/run.c
===
RCS file: /x/freebsd/src/contrib/one-true-awk/run.c,v
retrieving revision 1.1.1.2
diff -u -r1.1.1.2 run.c
--- src/contrib/one-true-awk/run.c 19 Feb 2002 09:35:25 - 1.1.1.2
+++ src/contrib/one-true-awk/run.c 20 Nov 2002 03:02:29 -
@@ -1504,11 +1504,11 @@
if (t == FTOUPPER) {
for (p = buf; *p; p++)
Re: awk(1) is locale unaware (was: Re: buildworld breakage during make depend at usr.bin/kdump)
On Wed, Nov 20, 2002 at 14:27:53 +1100, Tim Robbins wrote: On Wed, Nov 20, 2002 at 04:38:38AM +0300, Andrey A. Chernov wrote: On Tue, Nov 19, 2002 at 14:52:02 +0200, Ruslan Ermilov wrote: It seems that this patch has never been committed. This is a critical bug that should be fixed before 5.0-RELEASE is out. I agree. There is no locale yet and I never see that patch. This patch seems to work, I used the logic from regcomp.c in libc. Long lines make it ugly, but it was like that when I got here ;) Looks good, but it is not enough. Please look in b.c to see how weird character classes, i.e. [:alpha:] are implemented there, this stuff must be rewritted too. -- Andrey A. Chernov http://ache.pp.ru/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
On Thu, 1 Nov 2001, Alfred Perlstein wrote: Although I admit the fallout has been somewhat painful, let's try to make do with it, if we disconnect the new awk I feel that we will keep repeating this cycle, basically each activation will see new problems requiring another disconnect. Let's just get it fixed. :) For what it's worth, I agree with Alb^Hfred about this. This is -current after all. :) -- We will not tire, we will not falter, and we will not fail. - George W. Bush, President of the United States September 20, 2001 Do YOU Yahoo!? To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
Dag-Erling Smorgrav wrote: David O'Brien [EMAIL PROTECTED] writes: because `echo' nicely removes \n's from env vars when it prints them. des@des ~% foo='bar quote baz' des@des ~% echo $foo bar baz des@des ~% /bin/echo $foo bar baz DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] humm! what shell ($SHELL) are you using ? Cyrille. -- Cyrille Lefevre mailto:[EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
On 02-Nov-2001 (12:58:55/GMT) Cyrille Lefevre wrote: because `echo' nicely removes \n's from env vars when it prints them. des@des ~% foo='bar quote baz' des@des ~% echo $foo bar baz des@des ~% /bin/echo $foo bar baz humm! what shell ($SHELL) are you using ? Here (5.0-CURRENT 31-Oct and 4.4-STABLE 12-Oct), with /bin/sh works fine: % /bin/sh # foobar=foo bar # echo $foobar foo bar # echo $SHELL /bin/sh Riccardo. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
David O'Brien [EMAIL PROTECTED] writes: because `echo' nicely removes \n's from env vars when it prints them. des@des ~% foo='bar quote baz' des@des ~% echo $foo bar baz des@des ~% /bin/echo $foo bar baz DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
This is a quick heads-up. If you have built -CURRENT within the last couple of days, and if you try to use that (recently-built -CURRENT) as the host system for building -CURRENT, the 3 patches I posted last night do appear to get through the build process, but the result is a system that does not boot. I was just recently able to hit Pause on the keyboard at just the right moment to see that a message is issued very early in the boot process: name not found Assertion failed: (FALSE), funtion ficlCompileSoftCore, file softcor [and that appears to be where I hit Pause]. My *very* strong suspicion is that there are portions of the ficl build that use awk constructs that recent -CURRENT awk now treats differently, and usually silently. I will investigate and see what I can turn up for causes fixes. More later, david -- David H. Wolfskill [EMAIL PROTECTED] As a computing professional, I believe it would be unethical for me to advise, recommend, or support the use (save possibly for personal amusement) of any product that is or depends on any Microsoft product. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
On Thu, Nov 01, 2001 at 11:42:13 -0800, David O'Brien wrote: Mostly agreed. This has not been at all as smooth as I thought it would be. Before going down this path, I would like to see if the current state of the world isn't usable. I think (hope) all the nits are out now. #6 is definately needed if Bell-Labs awk stays. Next bad thing discovered about new awk just looking at sourse code: it not support locale (collating in regexp ranges too, of course). We just make great backward step switching to it. -- Andrey A. Chernov http://ache.pp.ru/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
On Fri, Nov 02, 2001 at 04:44:12AM +0300, Andrey A. Chernov wrote: Next bad thing discovered about new awk just looking at sourse code: it not support locale (collating in regexp ranges too, of course). We just make great backward step switching to it. I have a patch for that. -- -- David ([EMAIL PROTECTED]) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
On Thu, Nov 01, 2001 at 11:04:13 -0800, David Wolfskill wrote: Circumvention is to use /boot/loader.old, if it was built with gawk, or somehow build a new loader after applying a patch that accomplishes what this one does: # -# Note! This script uses strftime() which is a gawk-ism, and the -# POSIX [[:space:]] character class. No, awk should be fixed instead to conform POSIX specs, or switched back to gawk. -- Andrey A. Chernov http://ache.pp.ru/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
* Sheldon Hearn [EMAIL PROTECTED] [011101 13:27] wrote: On Thu, 01 Nov 2001 22:08:36 +0300, Andrey A. Chernov wrote: No, awk should be fixed instead to conform POSIX specs, or switched back to gawk. It's not a binary decision. What we should probably do is: 1) Disconnect bwk-awk from the build. 2) Connect gawk to the build. 3) Fix bwk-awk and / or in-tree scripts that rely on gawkisms. 4) Disconnect gawk from the build. 5) Connect bwk-awk to the build. 6) Issue a fat HEADS UP to -current (and probably -ports). Although I admit the fallout has been somewhat painful, let's try to make do with it, if we disconnect the new awk I feel that we will keep repeating this cycle, basically each activation will see new problems requiring another disconnect. Let's just get it fixed. :) -- -Alfred Perlstein [[EMAIL PROTECTED]] 'Instead of asking why a piece of software is using 1970s technology, start asking why software is ignoring 30 years of accumulated wisdom.' http://www.morons.org/rants/gpl-harmful.php3 To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
On Thu, Nov 01, 2001 at 09:23:12PM +0200, Sheldon Hearn wrote: No, awk should be fixed instead to conform POSIX specs, or switched back to gawk. It's not a binary decision. What we should probably do is: 1) Disconnect bwk-awk from the build. 2) Connect gawk to the build. 3) Fix bwk-awk and / or in-tree scripts that rely on gawkisms. 4) Disconnect gawk from the build. 5) Connect bwk-awk to the build. 6) Issue a fat HEADS UP to -current (and probably -ports). Mostly agreed. This has not been at all as smooth as I thought it would be. Before going down this path, I would like to see if the current state of the world isn't usable. I think (hope) all the nits are out now. #6 is definately needed if Bell-Labs awk stays. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
On Thu, 01 Nov 2001 13:31:04 CST, Alfred Perlstein wrote: Although I admit the fallout has been somewhat painful, let's try to make do with it, if we disconnect the new awk I feel that we will keep repeating this cycle, basically each activation will see new problems requiring another disconnect. Let's just get it fixed. :) But David's not getting it fixed. He's adjusting our awk source to work with the non-conformant bwk-awk, which doesn't solve the problem. Ciao, Sheldon. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
On Thu, Nov 01, 2001 at 09:42:18PM +0200, Sheldon Hearn wrote: Although I admit the fallout has been somewhat painful, let's try to make do with it, if we disconnect the new awk I feel that we will keep repeating this cycle, basically each activation will see new problems requiring another disconnect. Let's just get it fixed. :) But David's not getting it fixed. He's adjusting our awk source to work with the non-conformant bwk-awk, which doesn't solve the problem. It gets us back to a usable system, in which we have time to look into the issues. Getting the system usable is 1st priorty over cleaniness. -- -- David ([EMAIL PROTECTED]) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: buildworld breakage during make depend at usr.bin/kdump
Date: Wed, 31 Oct 2001 08:34:16 -0800 (PST) From: David Wolfskill [EMAIL PROTECTED] mkdep -f .depend -a-I/usr/obj/usr/src/i386/usr/include /usr/src/usr.bin/jot/jot.c cd /usr/src/usr.bin/jot; make _EXTRADEPEND echo jot: /usr/obj/usr/src/i386/usr/lib/libc.a .depend === usr.bin/kdump sh /usr/src/usr.bin/kdump/mkioctls /usr/obj/usr/src/i386/usr/include ioctl.c awk: newline in string #include cam/scsi/s... at source line 1 ... Stop in /usr/src/usr.bin/kdump. *** Error code 1 However, in reviewing that log, I didn't see the awk complaints (awk: newline in string #include cam/scsi/s... at source line 1). Yup; that is the error message that accompanies awk's termination (from within the src/usr.bin/kdump/mkioctls script) with status 2. At this stage, I'll welcome pointers/suggestions while I do some research on my own. By breaking down the parts of the pipeline in src/usr.bin/kdump/mkioctls, it looks as if the problem is that the (shell) variable ioctl_includes is being set to a list of #include directives, one per line, such as: #include cam/scsi/scsi_pass.h #include cam/scsi/scsi_ses.h #include cam/scsi/scsi_targetio.h #include dev/ppbus/lptio.h #include dev/ppbus/ppi.h ... That shell variable is being used to set the value of a similarly-named awk variable: awk -v ioctl_includes=$ioctl_includes -v use_switch=$use_switch ' And awk does *not* seem to like having embedded newline characters in there. By generating the output to a separate file, the replacing each real newline with a C/awk/Perl representation thereof (i.e., \n), I was able to get the awk part of the pipeline to terminate with a status of 0, while generating what looks like reasonable output (though I'm hardly an expert on that). If there is a way to tell awk to quit whining and *like* the newlines, that would seem to be ideal. Failing that, I can work up some way to transmogrify the #include list so awk will like it better Am I really the only one who has seen this? Cheers, david -- David H. Wolfskill [EMAIL PROTECTED] As a computing professional, I believe it would be unethical for me to advise, recommend, or support the use (save possibly for personal amusement) of any product that is or depends on any Microsoft product. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
buildworld breakage during make depend at usr.bin/kdump
=== usr.bin/jot rm -f .depend mkdep -f .depend -a-I/usr/obj/usr/src/i386/usr/include /usr/src/usr.bin/jot/jot.c cd /usr/src/usr.bin/jot; make _EXTRADEPEND echo jot: /usr/obj/usr/src/i386/usr/lib/libc.a .depend === usr.bin/kdump sh /usr/src/usr.bin/kdump/mkioctls /usr/obj/usr/src/i386/usr/include ioctl.c awk: newline in string #include cam/scsi/s... at source line 1 awk: newline in string #include cam/scsi/s... at source line 1 awk: newline in string #include cam/scsi/s... at source line 1 In file included from :72: /usr/obj/usr/src/i386/usr/include/sys/memrange.h:18: warning: `MDF_ACTIVE' redefined /usr/obj/usr/src/i386/usr/include/pccard/cardinfo.h:115: warning: this is the location of the previous definition cpp0: output pipe has been closed *** Error code 2 Stop in /usr/src/usr.bin/kdump. *** Error code 1 Stop in /usr/src/usr.bin. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. Recent CVSup history: CVSup begin from cvsup14.freebsd.org at Sat Oct 27 03:47:00 PDT 2001 CVSup ended from cvsup14.freebsd.org at Sat Oct 27 03:53:43 PDT 2001 CVSup begin from cvsup14.freebsd.org at Sun Oct 28 03:47:00 PST 2001 CVSup ended from cvsup14.freebsd.org at Sun Oct 28 03:52:52 PST 2001 CVSup begin from cvsup14.freebsd.org at Mon Oct 29 03:47:00 PST 2001 CVSup ended from cvsup14.freebsd.org at Mon Oct 29 03:53:47 PST 2001 CVSup begin from cvsup14.freebsd.org at Tue Oct 30 03:47:00 PST 2001 CVSup ended from cvsup14.freebsd.org at Tue Oct 30 03:53:02 PST 2001 CVSup begin from cvsup14.freebsd.org at Wed Oct 31 03:47:01 PST 2001 CVSup ended from cvsup14.freebsd.org at Wed Oct 31 03:53:25 PST 2001 The whine about duplicate definitions for MDF_ACTIVE is old news -- it was certainly present in yesterday's build, which completed normally (for me), and the most recent change to either was back around 28 July (and the last change to the other was near the end of 1999). However, in reviewing that log, I didn't see the awk complaints (awk: newline in string #include cam/scsi/s... at source line 1). Changing directories to src/usr.bin/kdump manually issuing make depend re-creates the symptoms, with an additional whine about /usr/include/sys/wormio.h:102 /usr/include/sys/cdrio.h:77 each having definitions of CDRIOCBLANK. As I recall, the fact that this was noticed in usr.bin/kdump is something that isn't likely of much material value: rather, that's an area that happens to be fairly sensitive, and one of the first to get hit when something goes weird elsewhere. At this stage, I'll welcome pointers/suggestions while I do some research on my own. Thanks, david -- David H. Wolfskill [EMAIL PROTECTED] As a computing professional, I believe it would be unethical for me to advise, recommend, or support the use (save possibly for personal amusement) of any product that is or depends on any Microsoft product. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
