Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-26 Thread Arnaud Lacombe
Hi,

On Tue, May 24, 2011 at 4:09 PM, Arnaud Lacombe  wrote:
> Many Makefile (espectially under sys/boot/) overwrite the value of CFLAGS.
> This is an issue if you want to generate code for a specific CPU as before the
> Makefile is interpreted, CFLAGS might already have been set with CPU specific
> settings by , which is source from sys.mk.
>
For the record, I maintain the bootloader part of that patch. So far,
it has not been (technically) proven that optimizing the bootloader
for a specific CPU breaks it in any way. So when I, as a FreeBSD
integrator, explicitly tells the build to target a given CPU, I expect
_all_ the component to be optimized for such a CPU.

Beside that contention point, I'd hope that the gvinum part can be committed.

 - Arnaud

> Signed-off-by: Arnaud Lacombe 
> ---
>  sbin/gvinum/Makefile                 |    2 +-
>  sys/boot/arm/at91/Makefile.inc       |    2 +-
>  sys/boot/i386/boot2/Makefile         |    2 +-
>  sys/boot/i386/gptboot/Makefile       |    2 +-
>  sys/boot/i386/gptzfsboot/Makefile    |    2 +-
>  sys/boot/i386/kgzldr/Makefile        |    2 +-
>  sys/boot/i386/zfsboot/Makefile       |    2 +-
>  sys/boot/pc98/boot2/Makefile         |    2 +-
>  sys/boot/pc98/kgzldr/Makefile        |    2 +-
>  sys/boot/powerpc/boot1.chrp/Makefile |    2 +-
>  sys/boot/sparc64/boot1/Makefile      |    2 +-
>  11 files changed, 11 insertions(+), 11 deletions(-)
>
> diff --git a/sbin/gvinum/Makefile b/sbin/gvinum/Makefile
> index 8cccf56..465b617 100644
> --- a/sbin/gvinum/Makefile
> +++ b/sbin/gvinum/Makefile
> @@ -5,7 +5,7 @@ SRCS=   gvinum.c gvinum.h geom_vinum_share.c
>  MAN=   gvinum.8
>
>  WARNS?=        2
> -CFLAGS=        -I${.CURDIR}/../../sys -I${DESTDIR}/${INCLUDEDIR}/edit
> +CFLAGS+=-I${.CURDIR}/../../sys -I${DESTDIR}/${INCLUDEDIR}/edit
>
>  DPADD= ${LIBEDIT} ${LIBTERMCAP} ${LIBDEVSTAT} ${LIBKVM} ${LIBGEOM}
>  LDADD= -ledit -ltermcap -ldevstat -lkvm -lgeom
> diff --git a/sys/boot/arm/at91/Makefile.inc b/sys/boot/arm/at91/Makefile.inc
> index 2f528e3..db37c7d 100644
> --- a/sys/boot/arm/at91/Makefile.inc
> +++ b/sys/boot/arm/at91/Makefile.inc
> @@ -8,7 +8,7 @@ __at91_boot_Makefile.inc__:
>  # tsc, bwct, kb920x, centipad are the supported flavors
>  BOOT_FLAVOR?=kb920x
>
> -CFLAGS=-Os -mcpu=arm9 -ffreestanding \
> +CFLAGS+=-Os -mcpu=arm9 -ffreestanding \
>        -I${.CURDIR}/../libat91 \
>        -I${.CURDIR}/../../../.. \
>        -I${.CURDIR}/../../../../arm \
> diff --git a/sys/boot/i386/boot2/Makefile b/sys/boot/i386/boot2/Makefile
> index 9568c1c..fc231e6 100644
> --- a/sys/boot/i386/boot2/Makefile
> +++ b/sys/boot/i386/boot2/Makefile
> @@ -22,7 +22,7 @@ BOOT2_UFS?=   UFS1_AND_UFS2
>  #BOOT2_UFS?=   UFS2_ONLY
>  #BOOT2_UFS?=   UFS1_ONLY
>
> -CFLAGS=        -Os \
> +CFLAGS+=-Os \
>        -fno-guess-branch-probability \
>        -fomit-frame-pointer \
>        -fno-unit-at-a-time \
> diff --git a/sys/boot/i386/gptboot/Makefile b/sys/boot/i386/gptboot/Makefile
> index 5642220..18c0f35 100644
> --- a/sys/boot/i386/gptboot/Makefile
> +++ b/sys/boot/i386/gptboot/Makefile
> @@ -19,7 +19,7 @@ GPTBOOT_UFS?= UFS1_AND_UFS2
>  #GPTBOOT_UFS?= UFS2_ONLY
>  #GPTBOOT_UFS?= UFS1_ONLY
>
> -CFLAGS=        -DBOOTPROG=\"gptboot\" \
> +CFLAGS+=-DBOOTPROG=\"gptboot\" \
>        -Os \
>        -fno-guess-branch-probability \
>        -fomit-frame-pointer \
> diff --git a/sys/boot/i386/gptzfsboot/Makefile 
> b/sys/boot/i386/gptzfsboot/Makefile
> index f0ee578..6e7d543 100644
> --- a/sys/boot/i386/gptzfsboot/Makefile
> +++ b/sys/boot/i386/gptzfsboot/Makefile
> @@ -16,7 +16,7 @@ REL1= 0x700
>  ORG1=  0x7c00
>  ORG2=  0x0
>
> -CFLAGS=        -DBOOTPROG=\"gptzfsboot\" \
> +CFLAGS+=-DBOOTPROG=\"gptzfsboot\" \
>        -Os \
>        -fno-guess-branch-probability \
>        -fomit-frame-pointer \
> diff --git a/sys/boot/i386/kgzldr/Makefile b/sys/boot/i386/kgzldr/Makefile
> index a124474..e8a4157 100644
> --- a/sys/boot/i386/kgzldr/Makefile
> +++ b/sys/boot/i386/kgzldr/Makefile
> @@ -7,7 +7,7 @@ BINDIR= ${LIBDIR}
>  NO_MAN=
>
>  SRCS=  start.s boot.c inflate.c lib.c crt.s sio.s
> -CFLAGS=        -Os
> +CFLAGS+=-Os
>  CFLAGS+=-DKZIP
>  NO_SHARED=
>  LDFLAGS=-Wl,-r
> diff --git a/sys/boot/i386/zfsboot/Makefile b/sys/boot/i386/zfsboot/Makefile
> index 06ff863..aa1bfbe 100644
> --- a/sys/boot/i386/zfsboot/Makefile
> +++ b/sys/boot/i386/zfsboot/Makefile
> @@ -14,7 +14,7 @@ REL1= 0x700
>  ORG1=  0x7c00
>  ORG2=  0x2000
>
> -CFLAGS=        -DBOOTPROG=\"zfsboot\" \
> +CFLAGS+=-DBOOTPROG=\"zfsboot\" \
>        -Os -g \
>        -fno-guess-branch-probability \
>        -fomit-frame-pointer \
> diff --git a/sys/boot/pc98/boot2/Makefile b/sys/boot/pc98/boot2/Makefile
> index 18bf251..e9f1506 100644
> --- a/sys/boot/pc98/boot2/Makefile
> +++ b/sys/boot/pc98/boot2/Makefile
> @@ -22,7 +22,7 @@ BOOT2_UFS?=   UFS1_AND_UFS2
>  #BOOT2_UFS?=   UFS2_ONLY
>  #BOOT2_UFS?=   UFS1_ONLY
>
> -CFLAGS=        -Os \
> +CFLAGS+=-Os \
>        -fno-guess-branch-probability \
>        -fomit-frame-pointer \
>        -fno-unit-at-a

Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread Arnaud Lacombe
Hi,

On Wed, May 25, 2011 at 4:10 PM, Arnaud Lacombe  wrote:
> I'm now trying to track down the original instruction triggering the
> SIGILL, but it is in a library and that section of the memory does not
> seem to be included in the core. Moreover I do not think I have any
> way on a broken system to get the address at which libraries get
> loaded (understand that ldd(1) is dynamically linked, and as the libc
> the likely culprit, rendering ldd(1) useless).
>
ok, the reason the whole system appears broken is because
`/libexec/ld-elf.so.1' is broken, it contains several CMOV
instructions. The objects it is made from are clear from any CMOV, so
this code likely comes from the jail's lib. I guess that the
responsible for building the jail environment had the wonderful idea
to use the host /usr/lib32 and put them in the jail /usr/lib, without
knowing that those libraries are i686-optimized, That also explain the
/boot/loader brokenness which links against the jail's libstand.a.

 - Arnaud
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread Arnaud Lacombe
Hi,

On Wed, May 25, 2011 at 3:44 PM, John Baldwin  wrote:
> On Wednesday, May 25, 2011 1:03:10 pm Arnaud Lacombe wrote:
>> Hi,
>>
>> On Wed, May 25, 2011 at 12:28 PM, John Baldwin  wrote:
>> > On Wednesday, May 25, 2011 11:34:29 am Arnaud Lacombe wrote:
>> >> Hi,
>> >>
>> >> On Wed, May 25, 2011 at 9:43 AM, John Baldwin  wrote:
>> >> >> The original trouble I met, is that building for an i586 target in a
>> >> >> 32bits jail, on top of an amd64 system[0] (I do not have control over
>> >> >> that setup) produces incorrect binaries. The current fix I've got is
>> >> >> to define MACHINE_ARCH=i386 and CPUTYPE=i586. This enforces
>> >> >> `-march=i586' to be passed to the compiler, for all except the
>> >> >> bootloader (because it overwrites CFLAGS). With this, binaries
>> >> >> produced works fine (ie. /bin/sh no longer SIGILL when bringing up the
>> >> >> system). So I suspect that gcc default to i686 in this setup and
>> >> >> corrupt all the binaries, thus the attached patch.
>> >> >
>> >> > Wait.  You must have something wrong in your jail if you can't do a
>> > buildworld
>> >> > with CPUTYPE set to none and have it do the right thing.  You need to
> find
>> >> > your root problem.  Forcing CPUCFLAGS for the boot code is a band-aid,
>> > it's
>> >> > not the right solution to your problem.
>> >> >
>> >> Unless error of my part, I never mentioned it was using `buildworld',
>> >> which it is not. The system uses bare calls to make(1) in the
>> >> sys/boot/ directory. As the jail is 32bits, it was expected not to be
>> >> an issue, but the jail compiler uses /lib/libstand.a to link the
>> >> loader, and it obviously contains i686-only instructions, which
>> >> trigger a reset of an i586-only CPU.
>> >>
>> >> The more broad issue with the setup is that gcc within that
>> >> environment, without being told -march=i586, produces i686
>> >> instructions which are incompatible with the target CPU.
>> >
>> > Huh?  GCC does not generate i686 instructions by default on FreeBSD/i386.
>  It
>> > generates i486 instructions but that is all.
>> something is odd somewhere.
>>
>> > Are you sure you aren't running
>> > the 64-bit gcc (which will generate i686 instructions by default)?
>> >
>> yes.
>>
>> # which gcc
>> /usr/bin/gcc
>>
>> # file /usr/bin/gcc
>> /usr/bin/gcc: ELF 32-bit LSB executable, Intel 80386, version 1
>> (FreeBSD), for FreeBSD 7.1, statically linked, FreeBSD-style, stripped
>>
>> # gcc -v
>> Using built-in specs.
>> Target: i386-undermydesk-freebsd
>> Configured with: FreeBSD/i386 system compiler
>> Thread model: posix
>> gcc version 4.2.1 20070719  [FreeBSD]
>>
>> # uname -a
>> FreeBSD build 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May  1 07:18:07
>> UTC 2009     r...@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
>>  amd64
>   ^
>
> I think this is probably going to confuse make and some other things as well.
>
This is what I meant when I said "canadian setup". HOST=amd64,
BUIILD=i386 and TARGET=i586.

I'm now trying to track down the original instruction triggering the
SIGILL, but it is in a library and that section of the memory does not
seem to be included in the core. Moreover I do not think I have any
way on a broken system to get the address at which libraries get
loaded (understand that ldd(1) is dynamically linked, and as the libc
the likely culprit, rendering ldd(1) useless).

 - Arnaud
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread John Baldwin
On Wednesday, May 25, 2011 1:03:10 pm Arnaud Lacombe wrote:
> Hi,
> 
> On Wed, May 25, 2011 at 12:28 PM, John Baldwin  wrote:
> > On Wednesday, May 25, 2011 11:34:29 am Arnaud Lacombe wrote:
> >> Hi,
> >>
> >> On Wed, May 25, 2011 at 9:43 AM, John Baldwin  wrote:
> >> >> The original trouble I met, is that building for an i586 target in a
> >> >> 32bits jail, on top of an amd64 system[0] (I do not have control over
> >> >> that setup) produces incorrect binaries. The current fix I've got is
> >> >> to define MACHINE_ARCH=i386 and CPUTYPE=i586. This enforces
> >> >> `-march=i586' to be passed to the compiler, for all except the
> >> >> bootloader (because it overwrites CFLAGS). With this, binaries
> >> >> produced works fine (ie. /bin/sh no longer SIGILL when bringing up the
> >> >> system). So I suspect that gcc default to i686 in this setup and
> >> >> corrupt all the binaries, thus the attached patch.
> >> >
> >> > Wait.  You must have something wrong in your jail if you can't do a
> > buildworld
> >> > with CPUTYPE set to none and have it do the right thing.  You need to 
find
> >> > your root problem.  Forcing CPUCFLAGS for the boot code is a band-aid,
> > it's
> >> > not the right solution to your problem.
> >> >
> >> Unless error of my part, I never mentioned it was using `buildworld',
> >> which it is not. The system uses bare calls to make(1) in the
> >> sys/boot/ directory. As the jail is 32bits, it was expected not to be
> >> an issue, but the jail compiler uses /lib/libstand.a to link the
> >> loader, and it obviously contains i686-only instructions, which
> >> trigger a reset of an i586-only CPU.
> >>
> >> The more broad issue with the setup is that gcc within that
> >> environment, without being told -march=i586, produces i686
> >> instructions which are incompatible with the target CPU.
> >
> > Huh?  GCC does not generate i686 instructions by default on FreeBSD/i386. 
 It
> > generates i486 instructions but that is all.
> something is odd somewhere.
> 
> > Are you sure you aren't running
> > the 64-bit gcc (which will generate i686 instructions by default)?
> >
> yes.
> 
> # which gcc
> /usr/bin/gcc
> 
> # file /usr/bin/gcc
> /usr/bin/gcc: ELF 32-bit LSB executable, Intel 80386, version 1
> (FreeBSD), for FreeBSD 7.1, statically linked, FreeBSD-style, stripped
> 
> # gcc -v
> Using built-in specs.
> Target: i386-undermydesk-freebsd
> Configured with: FreeBSD/i386 system compiler
> Thread model: posix
> gcc version 4.2.1 20070719  [FreeBSD]
> 
> # uname -a
> FreeBSD build 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May  1 07:18:07
> UTC 2009 r...@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
>  amd64
   ^

I think this is probably going to confuse make and some other things as well.

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread Kostik Belousov
On Wed, May 25, 2011 at 09:43:15AM -0400, John Baldwin wrote:
> It may be that your jail is not a pure 32-bit jail (some things like a 32-bit 
> ps won't really work in with a 64-bit host for example).  Also, until 
Err, is it broken (again) ? I committed the 32bit compat shims for kinfo_proc
long time ago, and do have ps working in 32bit jail in stable/8.

> recently, the machine_arch sysctl reported amd64 for 32-bit binaries.  That 
> was recently changed and might also help your world build to be more correct.
> You could also try doing 'make buildworld TARGET=i386' in a 64-bit jail.  
> However, one question is what problem are you trying to solve?


pgpFpeV5knWr5.pgp
Description: PGP signature


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread Arnaud Lacombe
Hi,

On Wed, May 25, 2011 at 12:28 PM, John Baldwin  wrote:
> On Wednesday, May 25, 2011 11:34:29 am Arnaud Lacombe wrote:
>> Hi,
>>
>> On Wed, May 25, 2011 at 9:43 AM, John Baldwin  wrote:
>> >> The original trouble I met, is that building for an i586 target in a
>> >> 32bits jail, on top of an amd64 system[0] (I do not have control over
>> >> that setup) produces incorrect binaries. The current fix I've got is
>> >> to define MACHINE_ARCH=i386 and CPUTYPE=i586. This enforces
>> >> `-march=i586' to be passed to the compiler, for all except the
>> >> bootloader (because it overwrites CFLAGS). With this, binaries
>> >> produced works fine (ie. /bin/sh no longer SIGILL when bringing up the
>> >> system). So I suspect that gcc default to i686 in this setup and
>> >> corrupt all the binaries, thus the attached patch.
>> >
>> > Wait.  You must have something wrong in your jail if you can't do a
> buildworld
>> > with CPUTYPE set to none and have it do the right thing.  You need to find
>> > your root problem.  Forcing CPUCFLAGS for the boot code is a band-aid,
> it's
>> > not the right solution to your problem.
>> >
>> Unless error of my part, I never mentioned it was using `buildworld',
>> which it is not. The system uses bare calls to make(1) in the
>> sys/boot/ directory. As the jail is 32bits, it was expected not to be
>> an issue, but the jail compiler uses /lib/libstand.a to link the
>> loader, and it obviously contains i686-only instructions, which
>> trigger a reset of an i586-only CPU.
>>
>> The more broad issue with the setup is that gcc within that
>> environment, without being told -march=i586, produces i686
>> instructions which are incompatible with the target CPU.
>
> Huh?  GCC does not generate i686 instructions by default on FreeBSD/i386.  It
> generates i486 instructions but that is all.
something is odd somewhere.

> Are you sure you aren't running
> the 64-bit gcc (which will generate i686 instructions by default)?
>
yes.

# which gcc
/usr/bin/gcc

# file /usr/bin/gcc
/usr/bin/gcc: ELF 32-bit LSB executable, Intel 80386, version 1
(FreeBSD), for FreeBSD 7.1, statically linked, FreeBSD-style, stripped

# gcc -v
Using built-in specs.
Target: i386-undermydesk-freebsd
Configured with: FreeBSD/i386 system compiler
Thread model: posix
gcc version 4.2.1 20070719  [FreeBSD]

# uname -a
FreeBSD build 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May  1 07:18:07
UTC 2009 r...@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
 amd64

The  fact that /usr/bin/gcc is statically linked made me think we may
have built internally, but it is also statically linked on a 8.2
machine from release packages.

 - Arnaud
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread Andriy Gapon
on 25/05/2011 19:28 John Baldwin said the following:
> On Wednesday, May 25, 2011 11:34:29 am Arnaud Lacombe wrote:
>> The more broad issue with the setup is that gcc within that
>> environment, without being told -march=i586, produces i686
>> instructions which are incompatible with the target CPU.
> 
> Huh?  GCC does not generate i686 instructions by default on FreeBSD/i386.  It 
> generates i486 instructions but that is all.  Are you sure you aren't running 
> the 64-bit gcc (which will generate i686 instructions by default)?

Just curious if Arnaud has forgot to set
UNAME_m="i386"
UNAME_p="i386"
in the environment of his build shell within the jail.

-- 
Andriy Gapon
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread John Baldwin
On Wednesday, May 25, 2011 11:34:29 am Arnaud Lacombe wrote:
> Hi,
> 
> On Wed, May 25, 2011 at 9:43 AM, John Baldwin  wrote:
> >> The original trouble I met, is that building for an i586 target in a
> >> 32bits jail, on top of an amd64 system[0] (I do not have control over
> >> that setup) produces incorrect binaries. The current fix I've got is
> >> to define MACHINE_ARCH=i386 and CPUTYPE=i586. This enforces
> >> `-march=i586' to be passed to the compiler, for all except the
> >> bootloader (because it overwrites CFLAGS). With this, binaries
> >> produced works fine (ie. /bin/sh no longer SIGILL when bringing up the
> >> system). So I suspect that gcc default to i686 in this setup and
> >> corrupt all the binaries, thus the attached patch.
> >
> > Wait.  You must have something wrong in your jail if you can't do a 
buildworld
> > with CPUTYPE set to none and have it do the right thing.  You need to find
> > your root problem.  Forcing CPUCFLAGS for the boot code is a band-aid, 
it's
> > not the right solution to your problem.
> >
> Unless error of my part, I never mentioned it was using `buildworld',
> which it is not. The system uses bare calls to make(1) in the
> sys/boot/ directory. As the jail is 32bits, it was expected not to be
> an issue, but the jail compiler uses /lib/libstand.a to link the
> loader, and it obviously contains i686-only instructions, which
> trigger a reset of an i586-only CPU.
> 
> The more broad issue with the setup is that gcc within that
> environment, without being told -march=i586, produces i686
> instructions which are incompatible with the target CPU.

Huh?  GCC does not generate i686 instructions by default on FreeBSD/i386.  It 
generates i486 instructions but that is all.  Are you sure you aren't running 
the 64-bit gcc (which will generate i686 instructions by default)?

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread Arnaud Lacombe
Hi,

On Wed, May 25, 2011 at 9:43 AM, John Baldwin  wrote:
> On Tuesday, May 24, 2011 5:30:37 pm Arnaud Lacombe wrote:
>> Hi,
>>
>> On Tue, May 24, 2011 at 4:41 PM, Dimitry Andric  wrote:
>> > On 2011-05-24 22:09, Arnaud Lacombe wrote:
>> >>
>> >> Many Makefile (espectially under sys/boot/) overwrite the value of
> CFLAGS.
>> >> This is an issue if you want to generate code for a specific CPU as
> before
>> >> the
>> >> Makefile is interpreted, CFLAGS might already have been set with CPU
>> >> specific
>> >> settings by, which is source from sys.mk.
>> >
>> > ...
>> >>
>> >> --- a/sys/boot/i386/boot2/Makefile
>> >> +++ b/sys/boot/i386/boot2/Makefile
>> >
>> > ...
>> >
>> > The problem with this patch is that for some of the things you fixed,
>> > stuff like boot-time programs, you NEVER want any CPU specific settings!
>> > You must use the default, lowest common denominator setting instead, or
>> > there is no guarantee the boot program will be correct.
>> >
>> To use your argument against you: with the default, the boot program
>> is not correct (see below).
>
> Thousands of machines succesfully booting FreeBSD would seem to contradict
> your assertion.
>
my assertion is backed by experiment, so I'm fine with the contradiction.

>> > So that is why these Makefiles purposefully overwrite CFLAGS. it is not
>> > by accident.
>> You just might be right, but unless the code say the overwrite is
>> _on_purpose_, I would not assume the state of mind of the author, one
>> way or another.
>
> The boot code is certainly intended to be something that works across the
> board.  Also, I doubt you will see any user-visible performance difference
> from changing the optimization options for the boot code.
>
Saving a few kilobytes of space might be more important than speed in
some environment.

>> > Besides, for space-constrained things like boot2, you
>> > might not even be able to compile it when using non-standard settings,
>> > since the code might grow too large.
>> >
>> or can shrink by using more optimized instructions.
>
> Well, your test in a later e-mail is a bit flawed.  GCC tends to insert
> a lot of padding for newer CPUs to align things on more optimal boundaries.
>
That does not explain at all why the default setting always produces
the worst code, size-wise.

> We run 'sed' over the assembly version of boot2 to strip all that out.
>
... to save 4 bytes.

> However, the more important point for the boot code is that it needs to
> just work.
>
and it does not, because gcc defaults are _bad_.

 - Arnaud
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread Arnaud Lacombe
Hi,

On Wed, May 25, 2011 at 9:43 AM, John Baldwin  wrote:
>> The original trouble I met, is that building for an i586 target in a
>> 32bits jail, on top of an amd64 system[0] (I do not have control over
>> that setup) produces incorrect binaries. The current fix I've got is
>> to define MACHINE_ARCH=i386 and CPUTYPE=i586. This enforces
>> `-march=i586' to be passed to the compiler, for all except the
>> bootloader (because it overwrites CFLAGS). With this, binaries
>> produced works fine (ie. /bin/sh no longer SIGILL when bringing up the
>> system). So I suspect that gcc default to i686 in this setup and
>> corrupt all the binaries, thus the attached patch.
>
> Wait.  You must have something wrong in your jail if you can't do a buildworld
> with CPUTYPE set to none and have it do the right thing.  You need to find
> your root problem.  Forcing CPUCFLAGS for the boot code is a band-aid, it's
> not the right solution to your problem.
>
Unless error of my part, I never mentioned it was using `buildworld',
which it is not. The system uses bare calls to make(1) in the
sys/boot/ directory. As the jail is 32bits, it was expected not to be
an issue, but the jail compiler uses /lib/libstand.a to link the
loader, and it obviously contains i686-only instructions, which
trigger a reset of an i586-only CPU.

The more broad issue with the setup is that gcc within that
environment, without being told -march=i586, produces i686
instructions which are incompatible with the target CPU.

 - Arnaud
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread John Baldwin
On Tuesday, May 24, 2011 5:30:37 pm Arnaud Lacombe wrote:
> Hi,
> 
> On Tue, May 24, 2011 at 4:41 PM, Dimitry Andric  wrote:
> > On 2011-05-24 22:09, Arnaud Lacombe wrote:
> >>
> >> Many Makefile (espectially under sys/boot/) overwrite the value of 
CFLAGS.
> >> This is an issue if you want to generate code for a specific CPU as 
before
> >> the
> >> Makefile is interpreted, CFLAGS might already have been set with CPU
> >> specific
> >> settings by, which is source from sys.mk.
> >
> > ...
> >>
> >> --- a/sys/boot/i386/boot2/Makefile
> >> +++ b/sys/boot/i386/boot2/Makefile
> >
> > ...
> >
> > The problem with this patch is that for some of the things you fixed,
> > stuff like boot-time programs, you NEVER want any CPU specific settings!
> > You must use the default, lowest common denominator setting instead, or
> > there is no guarantee the boot program will be correct.
> >
> To use your argument against you: with the default, the boot program
> is not correct (see below).

Thousands of machines succesfully booting FreeBSD would seem to contradict
your assertion.

> > So that is why these Makefiles purposefully overwrite CFLAGS. it is not
> > by accident.
> You just might be right, but unless the code say the overwrite is
> _on_purpose_, I would not assume the state of mind of the author, one
> way or another.

The boot code is certainly intended to be something that works across the
board.  Also, I doubt you will see any user-visible performance difference
from changing the optimization options for the boot code.

> > Besides, for space-constrained things like boot2, you
> > might not even be able to compile it when using non-standard settings,
> > since the code might grow too large.
> >
> or can shrink by using more optimized instructions.

Well, your test in a later e-mail is a bit flawed.  GCC tends to insert
a lot of padding for newer CPUs to align things on more optimal boundaries.
We run 'sed' over the assembly version of boot2 to strip all that out.
However, the more important point for the boot code is that it needs to
just work.

> The original trouble I met, is that building for an i586 target in a
> 32bits jail, on top of an amd64 system[0] (I do not have control over
> that setup) produces incorrect binaries. The current fix I've got is
> to define MACHINE_ARCH=i386 and CPUTYPE=i586. This enforces
> `-march=i586' to be passed to the compiler, for all except the
> bootloader (because it overwrites CFLAGS). With this, binaries
> produced works fine (ie. /bin/sh no longer SIGILL when bringing up the
> system). So I suspect that gcc default to i686 in this setup and
> corrupt all the binaries, thus the attached patch.

Wait.  You must have something wrong in your jail if you can't do a buildworld 
with CPUTYPE set to none and have it do the right thing.  You need to find 
your root problem.  Forcing CPUCFLAGS for the boot code is a band-aid, it's 
not the right solution to your problem.

It may be that your jail is not a pure 32-bit jail (some things like a 32-bit 
ps won't really work in with a 64-bit host for example).  Also, until 
recently, the machine_arch sysctl reported amd64 for 32-bit binaries.  That 
was recently changed and might also help your world build to be more correct.
You could also try doing 'make buildworld TARGET=i386' in a 64-bit jail.  
However, one question is what problem are you trying to solve?

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread Andriy Gapon
on 25/05/2011 15:25 Ruslan Ermilov said the following:
> On Tue, May 24, 2011 at 11:08:52PM -0400, Arnaud Lacombe wrote:
>> Hi,
>>
>> On Tue, May 24, 2011 at 10:54 PM, Arnaud Lacombe  wrote:
>>> ps: for static library and loader, I derived the total size as the sum
>>> of the size of the text/data/bss section of the member object using :
>>>
>>> size *.o | awk 'BEGIN {text=0; data=0; bss=0;}; {text+=$1; data+=$2;
>>> bss+=$3}; END {print text " " data " " bss " '$i'"}'
>>>
>>> where $i is the cpu type to test. make(1) is passed either CPUTYPE=$i
>>> for i in i[3456]86, or the empty string. The compiler used for the
>>> test is gcc, and it is the compiler build during a buildworld stage,
>>> in the tmp directory.
>>>
>> just to cut loose any question about my environment, additionally to
>> the original patch, I made the following modification to the tree to
>> try to isolate it from the host:
>>  - applied the following patch:
> 
> [...]
> 
>>  - manually created two symlinks:
>>  1) include/machine -> ../sys/i386/include/
>>  2) include/x86 -> ../sys/x86/include/
>>
>> The host is running a custom 8.2-STABLE/amd64 kernel (only config
>> change) on the following CPU:
>>
>> CPU: Intel(R) Xeon(R) CPU   X3430  @ 2.40GHz (2394.00-MHz K8-class 
>> CPU)
>>   Origin = "GenuineIntel"  Id = 0x106e5  Family = 6  Model = 1e  Stepping = 5
>>  
>> Features=0xbfebfbff
>>   
>> Features2=0x98e3fd
>>   AMD Features=0x28100800
>>   AMD Features2=0x1
>>   TSC: P-state invariant
>>
>> I am still not sure what is the default gcc target architecture on this 
>> machine.

Default target architecture should not depend on current machine.
It is in gcc manual page:
generic
Produce code optimized for the most common IA32/AMD64/EM64T
processors.  If you know the CPU on which your code will run,
then you should use the corresponding -mtune option instead of
-mtune=generic.  But, if you do not know exactly what CPU users
of your application will have, then you should use this option.

As new processors are deployed in the marketplace, the behavior
of this option will change.  Therefore, if you upgrade to a
newer version of GCC, the code generated option will change to
reflect the processors that were most common when that version
of GCC was released.

There is no -march=generic option because -march indicates the
instruction set the compiler can use, and there is no generic
instruction set applicable to all processors.  In contrast,
-mtune indicates the processor (or, in this case, collection of
processors) for which the code is optimized.
...
i686
Same as "generic", but when used as "march" option, PentiumPro
instruction set will be used, so the code will run on all i686
family chips.

> Why not go along a supported way, and do a cross-build?

There is nothing wrong about the day he does it.
And a classic cross-build won't help with setting i586 or lower that he needs:

i586, pentium
Intel Pentium CPU with no MMX support.

Just my 2 cents.
-- 
Andriy Gapon
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-25 Thread Ruslan Ermilov
On Tue, May 24, 2011 at 11:08:52PM -0400, Arnaud Lacombe wrote:
> Hi,
> 
> On Tue, May 24, 2011 at 10:54 PM, Arnaud Lacombe  wrote:
> > ps: for static library and loader, I derived the total size as the sum
> > of the size of the text/data/bss section of the member object using :
> >
> > size *.o | awk 'BEGIN {text=0; data=0; bss=0;}; {text+=$1; data+=$2;
> > bss+=$3}; END {print text " " data " " bss " '$i'"}'
> >
> > where $i is the cpu type to test. make(1) is passed either CPUTYPE=$i
> > for i in i[3456]86, or the empty string. The compiler used for the
> > test is gcc, and it is the compiler build during a buildworld stage,
> > in the tmp directory.
> >
> just to cut loose any question about my environment, additionally to
> the original patch, I made the following modification to the tree to
> try to isolate it from the host:
>  - applied the following patch:

[...]

>  - manually created two symlinks:
>  1) include/machine -> ../sys/i386/include/
>  2) include/x86 -> ../sys/x86/include/
> 
> The host is running a custom 8.2-STABLE/amd64 kernel (only config
> change) on the following CPU:
> 
> CPU: Intel(R) Xeon(R) CPU   X3430  @ 2.40GHz (2394.00-MHz K8-class 
> CPU)
>   Origin = "GenuineIntel"  Id = 0x106e5  Family = 6  Model = 1e  Stepping = 5
>  
> Features=0xbfebfbff
>   
> Features2=0x98e3fd
>   AMD Features=0x28100800
>   AMD Features2=0x1
>   TSC: P-state invariant
> 
> I am still not sure what is the default gcc target architecture on this 
> machine.

Why not go along a supported way, and do a cross-build?


-- 
Ruslan Ermilov
r...@freebsd.org
FreeBSD committer
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-24 Thread Arnaud Lacombe
Hi,

On Tue, May 24, 2011 at 10:54 PM, Arnaud Lacombe  wrote:
> ps: for static library and loader, I derived the total size as the sum
> of the size of the text/data/bss section of the member object using :
>
> size *.o | awk 'BEGIN {text=0; data=0; bss=0;}; {text+=$1; data+=$2;
> bss+=$3}; END {print text " " data " " bss " '$i'"}'
>
> where $i is the cpu type to test. make(1) is passed either CPUTYPE=$i
> for i in i[3456]86, or the empty string. The compiler used for the
> test is gcc, and it is the compiler build during a buildworld stage,
> in the tmp directory.
>
just to cut loose any question about my environment, additionally to
the original patch, I made the following modification to the tree to
try to isolate it from the host:
 - applied the following patch:

diff --git a/lib/libstand/Makefile b/lib/libstand/Makefile
index 05f0995..06592cd 100644
--- a/lib/libstand/Makefile
+++ b/lib/libstand/Makefile
@@ -19,14 +19,14 @@ WARNS?= 0

 CFLAGS+= -ffreestanding -Wformat
 CFLAGS+= -I${.CURDIR}
+CFLAGS+= -I${.CURDIR}/../../include
+CFLAGS+= -I${.CURDIR}/../../sys

 .if ${MACHINE_CPUARCH} == "i386" || ${MACHINE_CPUARCH} == "amd64"
 CFLAGS+=   -mpreferred-stack-boundary=2
 CFLAGS+=   -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -msoft-float
 .endif
-.if ${MACHINE} == "pc98"
-CFLAGS+=   -Os
-.endif
+#CFLAGS+=  -Os
 .if ${MACHINE_CPUARCH} == "powerpc"
 CFLAGS+=   -msoft-float -D_STANDALONE -DNETIF_DEBUG
 .endif
diff --git a/sys/boot/i386/libi386/Makefile b/sys/boot/i386/libi386/Makefile
index 7940471..1af3f44 100644
--- a/sys/boot/i386/libi386/Makefile
+++ b/sys/boot/i386/libi386/Makefile
@@ -51,6 +51,8 @@ CFLAGS+=  -I${.CURDIR}/../../common
-I${.CURDIR}/../btx/lib \
-I${.CURDIR}/../../../contrib/dev/acpica/include \
-I${.CURDIR}/../../.. -I.
 # the location of libstand
+CFLAGS+=   -I${.CURDIR}/../../../../include
+CFLAGS+=   -I${.CURDIR}/../../../../sys
 CFLAGS+=   -I${.CURDIR}/../../../../lib/libstand/

 .if ${CC:T:Mclang} == "clang"

 - manually created two symlinks:
 1) include/machine -> ../sys/i386/include/
 2) include/x86 -> ../sys/x86/include/

The host is running a custom 8.2-STABLE/amd64 kernel (only config
change) on the following CPU:

CPU: Intel(R) Xeon(R) CPU   X3430  @ 2.40GHz (2394.00-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x106e5  Family = 6  Model = 1e  Stepping = 5
 
Features=0xbfebfbff
  
Features2=0x98e3fd
  AMD Features=0x28100800
  AMD Features2=0x1
  TSC: P-state invariant

I am still not sure what is the default gcc target architecture on this machine.

 - Arnaud
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-24 Thread Arnaud Lacombe
Hi,

On Tue, May 24, 2011 at 4:41 PM, Dimitry Andric  wrote:
> Besides, for space-constrained things like boot2, you
> might not even be able to compile it when using non-standard settings,
> since the code might grow too large.
>
About the size, let's get some number

lib/libstand (library): (without -Os)
text data bss CPUTYPE
139696 4861 4160 i386
143212 4861 4160 i486
142491 4861 4160 i586
146893 4861 4160 i686
147145 4861 4160 

lib/libstand (library): (with -Os, only enabled on pc98)
90386 4472 4160 i386
90381 4472 4160 i486
90381 4472 4160 i586
90369 4472 4160 i686
90387 4472 4160 

sys/boot/i386/btx/lib/crt0.o is not affected size-wise by any of the
i[3456]86 CPUTYPE.

sys/boot/i386/boot2/boot2.out:
   text  databssdechex CPUTYPE
   5024  29178068331ab1 i386
   5024  29178068331ab1 i486
   5024  29178068331ab1 i586
   5004  29178068131a9d i686
   5024  29178068331ab1 

sys/boot/i386/libi386 (library):
text data bss CPUTYPE
36168 13970 18658 i386
37419 13970 18658 i486
37483 13970 18658 i586
40073 13970 18658 i686
40137 13970 18658 

sys/boot/ficl (library):
58245 4711 0 i386
61215 4711 0 i486
61183 4711 0 i586
70787 4711 0 i686
70851 4711 0 

and finally:

sys/boot/i386/loader:
57853 556 452 i386
58755 556 452 i486
58783 556 452 i586
63631 556 452 i686
63695 556 452 

So except for btx/crt0.o, the default setting generate the worst code size.

 - Arnaud

ps: for static library and loader, I derived the total size as the sum
of the size of the text/data/bss section of the member object using :

size *.o | awk 'BEGIN {text=0; data=0; bss=0;}; {text+=$1; data+=$2;
bss+=$3}; END {print text " " data " " bss " '$i'"}'

where $i is the cpu type to test. make(1) is passed either CPUTYPE=$i
for i in i[3456]86, or the empty string. The compiler used for the
test is gcc, and it is the compiler build during a buildworld stage,
in the tmp directory.

ps2: this was only a compile test. No code has been ran.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-24 Thread Arnaud Lacombe
Hi,

On Tue, May 24, 2011 at 4:41 PM, Dimitry Andric  wrote:
> On 2011-05-24 22:09, Arnaud Lacombe wrote:
>>
>> Many Makefile (espectially under sys/boot/) overwrite the value of CFLAGS.
>> This is an issue if you want to generate code for a specific CPU as before
>> the
>> Makefile is interpreted, CFLAGS might already have been set with CPU
>> specific
>> settings by, which is source from sys.mk.
>
> ...
>>
>> --- a/sys/boot/i386/boot2/Makefile
>> +++ b/sys/boot/i386/boot2/Makefile
>
> ...
>
> The problem with this patch is that for some of the things you fixed,
> stuff like boot-time programs, you NEVER want any CPU specific settings!
> You must use the default, lowest common denominator setting instead, or
> there is no guarantee the boot program will be correct.
>
To use your argument against you: with the default, the boot program
is not correct (see below).

> So that is why these Makefiles purposefully overwrite CFLAGS. it is not
> by accident.
You just might be right, but unless the code say the overwrite is
_on_purpose_, I would not assume the state of mind of the author, one
way or another.

> Besides, for space-constrained things like boot2, you
> might not even be able to compile it when using non-standard settings,
> since the code might grow too large.
>
or can shrink by using more optimized instructions.

The original trouble I met, is that building for an i586 target in a
32bits jail, on top of an amd64 system[0] (I do not have control over
that setup) produces incorrect binaries. The current fix I've got is
to define MACHINE_ARCH=i386 and CPUTYPE=i586. This enforces
`-march=i586' to be passed to the compiler, for all except the
bootloader (because it overwrites CFLAGS). With this, binaries
produced works fine (ie. /bin/sh no longer SIGILL when bringing up the
system). So I suspect that gcc default to i686 in this setup and
corrupt all the binaries, thus the attached patch.

 - Arnaud

[0]: to sum up, unless error of my part, this is a canadian setup
where HOST=amd64, BUILD=i386 and TARGET=i586, and all hell breaks
loose because of COMPAT_FREEBSD32.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-24 Thread Doug Barton

On 05/24/2011 13:41, Dimitry Andric wrote:

So that is why these Makefiles purposefully overwrite CFLAGS, it is not
by accident.


In those cases adding comments to the Makefile is probably appropriate, 
if they are not present already. Something to the effect of, "Purposely 
overwriting CFLAGS because ..."



Doug

--

Nothin' ever doesn't change, but nothin' changes much.
-- OK Go

Breadth of IT experience, and depth of knowledge in the DNS.
Yours for the right price.  :)  http://SupersetSolutions.com/

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [PATCH] Fix CFLAGS overwrite by Makefile

2011-05-24 Thread Dimitry Andric

On 2011-05-24 22:09, Arnaud Lacombe wrote:

Many Makefile (espectially under sys/boot/) overwrite the value of CFLAGS.
This is an issue if you want to generate code for a specific CPU as before the
Makefile is interpreted, CFLAGS might already have been set with CPU specific
settings by, which is source from sys.mk.

...

--- a/sys/boot/i386/boot2/Makefile
+++ b/sys/boot/i386/boot2/Makefile

...

The problem with this patch is that for some of the things you fixed,
stuff like boot-time programs, you NEVER want any CPU specific settings!
You must use the default, lowest common denominator setting instead, or
there is no guarantee the boot program will be correct.

So that is why these Makefiles purposefully overwrite CFLAGS, it is not
by accident.  Besides, for space-constrained things like boot2, you
might not even be able to compile it when using non-standard settings,
since the code might grow too large.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


[PATCH] Fix CFLAGS overwrite by Makefile

2011-05-24 Thread Arnaud Lacombe
Many Makefile (espectially under sys/boot/) overwrite the value of CFLAGS.
This is an issue if you want to generate code for a specific CPU as before the
Makefile is interpreted, CFLAGS might already have been set with CPU specific
settings by , which is source from sys.mk.

Signed-off-by: Arnaud Lacombe 
---
 sbin/gvinum/Makefile |2 +-
 sys/boot/arm/at91/Makefile.inc   |2 +-
 sys/boot/i386/boot2/Makefile |2 +-
 sys/boot/i386/gptboot/Makefile   |2 +-
 sys/boot/i386/gptzfsboot/Makefile|2 +-
 sys/boot/i386/kgzldr/Makefile|2 +-
 sys/boot/i386/zfsboot/Makefile   |2 +-
 sys/boot/pc98/boot2/Makefile |2 +-
 sys/boot/pc98/kgzldr/Makefile|2 +-
 sys/boot/powerpc/boot1.chrp/Makefile |2 +-
 sys/boot/sparc64/boot1/Makefile  |2 +-
 11 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/sbin/gvinum/Makefile b/sbin/gvinum/Makefile
index 8cccf56..465b617 100644
--- a/sbin/gvinum/Makefile
+++ b/sbin/gvinum/Makefile
@@ -5,7 +5,7 @@ SRCS=   gvinum.c gvinum.h geom_vinum_share.c
 MAN=   gvinum.8
 
 WARNS?=2
-CFLAGS=-I${.CURDIR}/../../sys -I${DESTDIR}/${INCLUDEDIR}/edit
+CFLAGS+=-I${.CURDIR}/../../sys -I${DESTDIR}/${INCLUDEDIR}/edit
 
 DPADD= ${LIBEDIT} ${LIBTERMCAP} ${LIBDEVSTAT} ${LIBKVM} ${LIBGEOM}
 LDADD= -ledit -ltermcap -ldevstat -lkvm -lgeom
diff --git a/sys/boot/arm/at91/Makefile.inc b/sys/boot/arm/at91/Makefile.inc
index 2f528e3..db37c7d 100644
--- a/sys/boot/arm/at91/Makefile.inc
+++ b/sys/boot/arm/at91/Makefile.inc
@@ -8,7 +8,7 @@ __at91_boot_Makefile.inc__:
 # tsc, bwct, kb920x, centipad are the supported flavors
 BOOT_FLAVOR?=kb920x
 
-CFLAGS=-Os -mcpu=arm9 -ffreestanding \
+CFLAGS+=-Os -mcpu=arm9 -ffreestanding \
-I${.CURDIR}/../libat91 \
-I${.CURDIR}/../../../.. \
-I${.CURDIR}/../../../../arm \
diff --git a/sys/boot/i386/boot2/Makefile b/sys/boot/i386/boot2/Makefile
index 9568c1c..fc231e6 100644
--- a/sys/boot/i386/boot2/Makefile
+++ b/sys/boot/i386/boot2/Makefile
@@ -22,7 +22,7 @@ BOOT2_UFS?=   UFS1_AND_UFS2
 #BOOT2_UFS?=   UFS2_ONLY
 #BOOT2_UFS?=   UFS1_ONLY
 
-CFLAGS=-Os \
+CFLAGS+=-Os \
-fno-guess-branch-probability \
-fomit-frame-pointer \
-fno-unit-at-a-time \
diff --git a/sys/boot/i386/gptboot/Makefile b/sys/boot/i386/gptboot/Makefile
index 5642220..18c0f35 100644
--- a/sys/boot/i386/gptboot/Makefile
+++ b/sys/boot/i386/gptboot/Makefile
@@ -19,7 +19,7 @@ GPTBOOT_UFS?= UFS1_AND_UFS2
 #GPTBOOT_UFS?= UFS2_ONLY
 #GPTBOOT_UFS?= UFS1_ONLY
 
-CFLAGS=-DBOOTPROG=\"gptboot\" \
+CFLAGS+=-DBOOTPROG=\"gptboot\" \
-Os \
-fno-guess-branch-probability \
-fomit-frame-pointer \
diff --git a/sys/boot/i386/gptzfsboot/Makefile 
b/sys/boot/i386/gptzfsboot/Makefile
index f0ee578..6e7d543 100644
--- a/sys/boot/i386/gptzfsboot/Makefile
+++ b/sys/boot/i386/gptzfsboot/Makefile
@@ -16,7 +16,7 @@ REL1= 0x700
 ORG1=  0x7c00
 ORG2=  0x0
 
-CFLAGS=-DBOOTPROG=\"gptzfsboot\" \
+CFLAGS+=-DBOOTPROG=\"gptzfsboot\" \
-Os \
-fno-guess-branch-probability \
-fomit-frame-pointer \
diff --git a/sys/boot/i386/kgzldr/Makefile b/sys/boot/i386/kgzldr/Makefile
index a124474..e8a4157 100644
--- a/sys/boot/i386/kgzldr/Makefile
+++ b/sys/boot/i386/kgzldr/Makefile
@@ -7,7 +7,7 @@ BINDIR= ${LIBDIR}
 NO_MAN=
 
 SRCS=  start.s boot.c inflate.c lib.c crt.s sio.s
-CFLAGS=-Os
+CFLAGS+=-Os
 CFLAGS+=-DKZIP
 NO_SHARED=
 LDFLAGS=-Wl,-r
diff --git a/sys/boot/i386/zfsboot/Makefile b/sys/boot/i386/zfsboot/Makefile
index 06ff863..aa1bfbe 100644
--- a/sys/boot/i386/zfsboot/Makefile
+++ b/sys/boot/i386/zfsboot/Makefile
@@ -14,7 +14,7 @@ REL1= 0x700
 ORG1=  0x7c00
 ORG2=  0x2000
 
-CFLAGS=-DBOOTPROG=\"zfsboot\" \
+CFLAGS+=-DBOOTPROG=\"zfsboot\" \
-Os -g \
-fno-guess-branch-probability \
-fomit-frame-pointer \
diff --git a/sys/boot/pc98/boot2/Makefile b/sys/boot/pc98/boot2/Makefile
index 18bf251..e9f1506 100644
--- a/sys/boot/pc98/boot2/Makefile
+++ b/sys/boot/pc98/boot2/Makefile
@@ -22,7 +22,7 @@ BOOT2_UFS?=   UFS1_AND_UFS2
 #BOOT2_UFS?=   UFS2_ONLY
 #BOOT2_UFS?=   UFS1_ONLY
 
-CFLAGS=-Os \
+CFLAGS+=-Os \
-fno-guess-branch-probability \
-fomit-frame-pointer \
-fno-unit-at-a-time \
diff --git a/sys/boot/pc98/kgzldr/Makefile b/sys/boot/pc98/kgzldr/Makefile
index 2303fd9..645cc58 100644
--- a/sys/boot/pc98/kgzldr/Makefile
+++ b/sys/boot/pc98/kgzldr/Makefile
@@ -7,7 +7,7 @@ BINDIR= ${LIBDIR}
 NO_MAN=
 
 SRCS=  start.s boot.c inflate.c lib.c crt.s sio.s
-CFLAGS=-Os
+CFLAGS+=-Os
 CFLAGS+=-DKZIP
 NO_SHARED=
 LDFLAGS=-Wl,-r
diff --git a/sys/boot/powerpc/boot1.chrp/Makefile 
b/sys/boot/powerpc/boot1.chrp/Makefile
index f3f7e4d..88358b5 100644
--- a/sys/boot/powerpc/boot1.chrp/Makefile
+++ b/sys/boot/powerpc/boot1.chrp/Makefile
@@ -13,7 +13,7 @@ SRCS= boot1.c ashldi3.c
 INTERNALPROG=
 NO_MAN=
 
-CFLAGS= -ffreestanding -msoft-float -Os \
+CFLAGS