Re: 11.0 -r301139: WITH_META_MODE=yes vs. "sh: ./make_keys: Exec format error"? [still true of -r301815]

[Bryan Drewery]

On 6/11/2016 9:55 PM, Mark Millard wrote:
> Doing cleanworld instead of "-j 5 buildworld buildkernel" and then retrying 
> "-j 5 buildworld buildkernel" resulted in the same sort of thing but for 
> maketab instead:

This is fixed after r301884.  You'll need to clean the objdir for
usr.bin/awk at the least to fix the problem.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: 11.0 -r301139: WITH_META_MODE=yes vs. "sh: ./make_keys: Exec format error"? [still true of -r301815]

[Simon J. Gerraty]

Bryan Drewery  wrote:
> > eg. in our internal tree - which cross builds fine:
> > 
> > make_keys: make_keys.c names.c ncurses_def.h ${HEADERS}
> > ${HOST_CC} -o $@ ${HOST_CFLAGS} 
> > ${NCURSES_DIR}/ncurses/tinfo/make_keys.c
> 
> I like this method but am going to put it off for a while.  The

The above is a reasonably non-intrusive way of dealing with targets like
that.

For tools which are more expensive to build, having a separate dir is
best, since (with DIRDEPS at least) it is then easier to build them
only once and at the right time.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: 11.0 -r301139: WITH_META_MODE=yes vs. "sh: ./make_keys: Exec format error"? [still true of -r301815]

[Bryan Drewery]

On 6/11/2016 9:55 PM, Mark Millard wrote:
> Doing cleanworld instead of "-j 5 buildworld buildkernel" and then retrying 
> "-j 5 buildworld buildkernel" resulted in the same sort of thing but for 
> maketab instead:

The cleanworld likely fixed an issue of a stale file sitting in the
objdir.  I was able to reproduce the maketab bug though and am fixing that.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: 11.0 -r301139: WITH_META_MODE=yes vs. "sh: ./make_keys: Exec format error"? [still true of -r301815]

[Bryan Drewery]

On 6/12/2016 5:39 AM, Simon J. Gerraty wrote:
> eg. in our internal tree - which cross builds fine:
> 
> make_keys: make_keys.c names.c ncurses_def.h ${HEADERS}
> ${HOST_CC} -o $@ ${HOST_CFLAGS} 
> ${NCURSES_DIR}/ncurses/tinfo/make_keys.c

I like this method but am going to put it off for a while.  The
confusion of CC vs XCC is bad enough without introducing a HOST_CC.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: 11.0 -r301139: WITH_META_MODE=yes vs. "sh: ./make_keys: Exec format error"? [still true of -r301815]

[Bryan Drewery]

On 6/12/2016 7:14 PM, Simon J. Gerraty wrote:
> Mark Millard  wrote:
>> Cross builds work just fine based on the FreeBSD tree when omitting 
>> WITH_META_MODE=.
>>
> 
> Hmm must do something odd then.
> 
>> As of -r301825 there is almost no use of HOST_CC at the upper levels or in 
>> share/mk/*:
> 
> Yes, which means if cross-building works it must be requring a separate
> walk of the tree or similar dance.

Yeah it's done in 2 walks with a shared objdir.  I am going to rework
these build-tools files to store their native binaries in BTOOLSDIR
(like DIRDEPS does for these for bootstrap-tools target actually).  Then
always use the one from the BTOOLSDIR, avoiding the chance of a target
one being built and ran from './'.

> It cannot work by building tools like make_keys in the same pass as the
> library, since that would require use of something other than CC.
> 
> 
>>> # grep HOST_CC /usr/src/Makefile*
>>> # grep HOST_CC /usr/src/share/mk/*
>>> /usr/src/share/mk/bsd.compiler.mk:.for var in CC CXX HOST_CC HOST_CXX
>>
>> where that last does:
> 
> Yes, that's for ccache, but I don't see HOST_CC used anywhere.
> 
> Sorry, guess I have no clue how cross-building currently works in
> FreeBSD, but as is it won't work with DIRDEPS - which pretty much
> expects to be able to build the tree in a single pass.
> 
> META_MODE should be orgthogonal, but that appears not to be so.
> 

It's mostly orthogonal. Since there is a shared objdir for both the
'build-tools' and 'lib' and 'everything' walks, the CFLAGS/CC does
change and cause rebuilds.  I have some hacks in there now that I'm not
happy with and am going to rework.


-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: 11.0 -r301139: WITH_META_MODE=yes vs. "sh: ./make_keys: Exec format error"? [still true of -r301815]

[Simon J. Gerraty]

Mark Millard  wrote:
> Cross builds work just fine based on the FreeBSD tree when omitting 
> WITH_META_MODE=.
> 

Hmm must do something odd then.

> As of -r301825 there is almost no use of HOST_CC at the upper levels or in 
> share/mk/*:

Yes, which means if cross-building works it must be requring a separate
walk of the tree or similar dance.
It cannot work by building tools like make_keys in the same pass as the
library, since that would require use of something other than CC.


> > # grep HOST_CC /usr/src/Makefile*
> > # grep HOST_CC /usr/src/share/mk/*
> > /usr/src/share/mk/bsd.compiler.mk:.for var in CC CXX HOST_CC HOST_CXX
> 
> where that last does:

Yes, that's for ccache, but I don't see HOST_CC used anywhere.

Sorry, guess I have no clue how cross-building currently works in
FreeBSD, but as is it won't work with DIRDEPS - which pretty much
expects to be able to build the tree in a single pass.

META_MODE should be orgthogonal, but that appears not to be so.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: 11.0 -r301139: WITH_META_MODE=yes vs. "sh: ./make_keys: Exec format error"? [still true of -r301815]

[Mark Millard]

On 2016-Jun-12, at 5:39 AM, Simon J. Gerraty  wrote:

> Mark Millard  wrote:
>>> --- build-tools_lib/ncurses/ncursesw ---
>>> Building /usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncursesw/make_keys
> 
> I must have been looking at on of our internal FreeBSD trees last
> time...
> 
> In FreeBSD lib/ncurses/ncursesw/Makefile and other places I checked
> just uses ${CC} for building make_keys - I don't see how that could work
> for cross-building. 
> 
> If you want cross-building to work, the simple solution is to ensure
> that you use HOST_CC rather than CC when building things that need to
> run on the build host.
> 
> eg. in our internal tree - which cross builds fine:
> 
> make_keys: make_keys.c names.c ncurses_def.h ${HEADERS}
>${HOST_CC} -o $@ ${HOST_CFLAGS} 
> ${NCURSES_DIR}/ncurses/tinfo/make_keys.c
> 
> make_hash: make_hash.c hashsize.h ncurses_def.h ${HEADERS}
>${HOST_CC} -o $@ ${HOST_CFLAGS} -DMAIN_PROGRAM \
>   ${NCURSES_DIR}/ncurses/tinfo/make_hash.c
> 
> and in share/mk/local.init.mk
> 
> HOST_CC?= /usr/bin/cc
> 
> should do.

Cross builds work just fine based on the FreeBSD tree when omitting 
WITH_META_MODE=.

It is only when using WITH_META_MODE= that the cross build does not work.

So the ${CC} use must mix with some other aspect of WITH_META_MODE= operation 
if ${CC} use contributes to the failure.

As of -r301825 there is almost no use of HOST_CC at the upper levels or in 
share/mk/*:

> # grep HOST_CC /usr/src/Makefile*
> # grep HOST_CC /usr/src/share/mk/*
> /usr/src/share/mk/bsd.compiler.mk:.for var in CC CXX HOST_CC HOST_CXX

where that last does:

> .for var in CC CXX HOST_CC HOST_CXX
> .if defined(${var}) && ${${var}:M${CCACHE_BIN}} == ""
> ${var}:=${CCACHE_BIN} ${${var}}
> .endif
> .endfor



Looking around more suggests that there is CC_FOR_BUILD and that HOST_CC is 
more historical:

> find /usr/src/ -name .svn -prune -o -exec grep HOST_CC {} ; -print | more
> # Historically, `CC_FOR_BUILD' used to be named `HOST_CC'. We still
> # use `HOST_CC' if defined, but it is deprecated.
> case $CC_FOR_BUILD,$HOST_CC,$CC in
>  ,*,*)  CC_FOR_BUILD=$HOST_CC ;;
> /usr/src/contrib/sqlite3/config.guess
> * Makefile.in: rename HOST_CC to CC_FOR_BUILD
> /usr/src/contrib/binutils/ld/ChangeLog-9197
> # Historically, `CC_FOR_BUILD' used to be named `HOST_CC'. We still
> # use `HOST_CC' if defined, but it is deprecated.
> case $CC_FOR_BUILD,$HOST_CC,$CC in
>  ,*,*)  CC_FOR_BUILD=$HOST_CC ;;
> /usr/src/contrib/binutils/config.guess
> (HOST_CC, HOST_PREFIX, HOST_PREFIX_1): Undefine, they should
> (HOST_CC, HOST_PREFIX, HOST_PREFIX_1): New variables.
> /usr/src/contrib/binutils/ChangeLog
> # Historically, `CC_FOR_BUILD' used to be named `HOST_CC'. We still
> # use `HOST_CC' if defined, but it is deprecated.
> case $CC_FOR_BUILD,$HOST_CC,$CC in
>  ,*,*)  CC_FOR_BUILD=$HOST_CC ;;
> /usr/src/contrib/ntp/config.guess
> # Historically, `CC_FOR_BUILD' used to be named `HOST_CC'. We still
> # use `HOST_CC' if defined, but it is deprecated.
> case $CC_FOR_BUILD,$HOST_CC,$CC in
>  ,*,*)  CC_FOR_BUILD=$HOST_CC ;;
> /usr/src/contrib/ntp/sntp/config.guess
> # Historically, `CC_FOR_BUILD' used to be named `HOST_CC'. We still
> # use `HOST_CC' if defined, but it is deprecated.
> case $CC_FOR_BUILD,$HOST_CC,$CC in
>  ,*,*)  CC_FOR_BUILD=$HOST_CC ;;
> /usr/src/contrib/ntp/sntp/libevent/build-aux/config.guess
> # Historically, `CC_FOR_BUILD' used to be named `HOST_CC'. We still
> # use `HOST_CC' if defined, but it is deprecated.
> case $CC_FOR_BUILD,$HOST_CC,$CC in
>  ,*,*)  CC_FOR_BUILD=$HOST_CC ;;
> /usr/src/contrib/ofed/librdmacm/config/config.guess
. . .


===
Mark Millard
markmi at dsl-only.net

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: 11.0 -r301139: WITH_META_MODE=yes vs. "sh: ./make_keys: Exec format error"? [still true of -r301815]

[Simon J. Gerraty]

Simon J. Gerraty  wrote:
> If you want cross-building to work, the simple solution is to ensure
> that you use HOST_CC rather than CC when building things that need to
> run on the build host.

FWIW there are not many makefiles to fix:

bin/csh/Makefile
gnu/usr.bin/cc/cc_tools/Makefile
lib/libmagic/Makefile
lib/ncurses/ncurses/Makefile
share/syscons/scrnmaps/Makefile
usr.bin/awk/Makefile

and bin/sh/Makefile which is a bit different for some reason.
also in Makefile.inc1

CC=${HOST_CC} is in the BMAKE command line:

BMAKE=  MAKEOBJDIRPREFIX=${WORLDTMP} \
${BMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc1 \
DESTDIR= \
BOOTSTRAPPING=${OSRELDATE} \
SSP_CFLAGS= \
HOSTPROG=yes \
CC=${HOST_CC} \
MK_HTML=no NO_LINT=yes MK_MAN=no \
-DNO_PIC MK_PROFILE=no -DNO_SHARED \
-DNO_CPU_CFLAGS MK_WARNS=no MK_CTF=no \
MK_CLANG_EXTRAS=no MK_CLANG_FULL=no \
MK_LLDB=no MK_TESTS=no \
MK_INCLUDES=yes 

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: 11.0 -r301139: WITH_META_MODE=yes vs. "sh: ./make_keys: Exec format error"? [still true of -r301815]

[Simon J. Gerraty]

Mark Millard  wrote:
> > --- build-tools_lib/ncurses/ncursesw ---
> > Building /usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncursesw/make_keys

I must have been looking at on of our internal FreeBSD trees last
time...

In FreeBSD lib/ncurses/ncursesw/Makefile and other places I checked
just uses ${CC} for building make_keys - I don't see how that could work
for cross-building. 

If you want cross-building to work, the simple solution is to ensure
that you use HOST_CC rather than CC when building things that need to
run on the build host.

eg. in our internal tree - which cross builds fine:

make_keys: make_keys.c names.c ncurses_def.h ${HEADERS}
${HOST_CC} -o $@ ${HOST_CFLAGS} ${NCURSES_DIR}/ncurses/tinfo/make_keys.c

make_hash: make_hash.c hashsize.h ncurses_def.h ${HEADERS}
${HOST_CC} -o $@ ${HOST_CFLAGS} -DMAIN_PROGRAM \
${NCURSES_DIR}/ncurses/tinfo/make_hash.c

and in share/mk/local.init.mk

HOST_CC?= /usr/bin/cc

should do.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: 11.0 -r301139: WITH_META_MODE=yes vs. "sh: ./make_keys: Exec format error"? [still true of -r301815]

[Mark Millard]

On 2016-Jun-2, at 11:11 AM, Bryan Drewery  wrote:

> On 6/2/2016 10:53 AM, Simon J. Gerraty wrote:
>> BTW Mark, thanks very much for testing this.
>> 
 # grep make_keys 
 ~/sys_typescripts/typescript_make_rpi2_nodebug_clang_bootstrap-amd64-host-2016-06-01:15:17:28
 Building /usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncursesw/make_keys
 Building /usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncurses/make_keys
 Building /usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncursesw/make_keys
 sh: ./make_keys: Exec format error
>>> 
>>> Note that ncursesw has two Building lines above with the same path listed.
>> 
>> This build is still using the normal orchestration (tree walks etc)
>> so it it not at all out of the question for directories to be visited
>> more than once.
>> 
>> If curious; you can add -dM to have make explain why it built it again
>> the 2nd time.
>> The output can be copious, so you might want to only enable it in
>> ncursesw
>> eg.
>> 
>> .MAKEFLAGS: -dM
>> 
> 
> r297997 was the original fix for it. It is visited twice (build-tools
> and everything).  The problem came when I added .META to all suffix
> rules to work around the lack of default .META (the patch we're discussing).
> 
> -- 
> Regards,
> Bryan Drewery

Summary: It looks like cross builds and WITH_META_MODE= still do not mix.

Example details follow.

I finally just updated my amd64 environment from -r301139 to . . .

> # uname -apKU
> FreeBSD FreeBSDx64 11.0-ALPHA3 FreeBSD 11.0-ALPHA3 #3 r301815M: Sat Jun 11 
> 12:03:13 PDT 2016 
> root@FreeBSDx64:/usr/obj/clang/amd64.amd64/usr/src/sys/GENERIC-NODEBUG  amd64 
> amd64 1100116 1100116

So I then tried to update my rpi2 targeted cross build:

> Script started on Sat Jun 11 18:58:11 2016
> Command: env __MAKE_CONF=/root/src.configs/make.conf 
> SRC_ENV_CONF=/root/src.configs/src.conf.rpi2-clang-bootstrap.amd64-host 
> WITH_META_MODE=yes MAKEOBJDIRPREFIX=/usr/obj/clang make -j 5 buildworld 
> buildkernel
. . .
> --- build-tools_lib/ncurses/ncurses ---
> Building /usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncurses/make_keys
> --- build-tools_rescue/rescue ---
> --- obj ---
> --- build-tools_lib/ncurses/ncursesw ---
> ** edit: BROKEN_LINKER 0
> --- build-tools_rescue/rescue ---
. . .
> --- build-tools_lib/ncurses/ncursesw ---
> Building /usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncursesw/make_keys

. . .
> Building /usr/obj/clang/arm.armv6/usr/src/lib/ncurses/ncursesw/init_keytry.h
> --- init_keytry.h ---
> sh: ./make_keys: Exec format error
> *** [init_keytry.h] Error code 126
> 
> make[4]: stopped in /usr/src/lib/ncurses/ncursesw
> 1 error
> 
> make[4]: stopped in /usr/src/lib/ncurses/ncursesw
> *** [lib/ncurses/ncursesw__L] Error code 2

Doing cleanworld instead of "-j 5 buildworld buildkernel" and then retrying "-j 
5 buildworld buildkernel" resulted in the same sort of thing but for maketab 
instead:

> Script started on Sat Jun 11 19:38:48 2016
> Command: env __MAKE_CONF=/root/src.configs/make.conf 
> SRC_ENV_CONF=/root/src.configs/src.conf.rpi2-clang-bootstrap.amd64-host 
> WITH_META_MODE=yes MAKEOBJDIRPREFIX=/usr/obj/clang make -j 5 buildworld 
> buildkernel
> --- buildworld ---
. . .
> --- ytab.h ---
> ln -sf awkgram.h ytab.h
> Building /usr/obj/clang/arm.armv6/usr/src/usr.bin/awk/maketab
. . .
> --- all_subdir_usr.bin ---
> Building /usr/obj/clang/arm.armv6/usr/src/usr.bin/awk/maketab

. . .
> ===> share/doc/usd (all)
> --- all_subdir_sys ---
> ln -sf /usr/src/sys/boot/efi/loader/../../../arm/include machine
> --- all_subdir_usr.bin ---
> --- proctab.c ---
> sh: ./maketab: Exec format error
> *** [proctab.c] Error code 126
> 
> make[4]: stopped in /usr/src/usr.bin/awk
> 1 error
> 
> make[4]: stopped in /usr/src/usr.bin/awk
> *** [all_subdir_usr.bin/awk] Error code 2


===
Mark Millard
markmi at dsl-only.net

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"