The new rules in src/Makefile.in change our native build process
somewhat. I traced through them, following along in my build log, and
came up with this Developers' Summary.
* Riastradh Build
The host syntaxes/compiles the new cref, then the new runtime and sf
(using the new cref). The target cold loads the new runtime and dumps
it, then loads the new sf and syntaxes the new compiler. The host
compiles the syntaxed compiler. The target loads and dumps the new
compiler band, which is used to compile the rest of the system(s)
(except that the host compiles the star-parser?).
The result is an all.com band and loadable options containing a
host-compiled cref, runtime and sf; everything else is compiled by the
new compiler (except star-parser?). This avoids compiling anything
twice, using Stage.sh, or tossing anything -- things that the
cross-compiles do with abandon.
The mix of host- and target-compiled subsystems assumes the new and
old machines are largely compatible. The current change in s-code
types means the host-compiled .bcis contain surprises for the target
machine.
* LIAR/C Build
I traced through a liarc build too, a full-on cross-compile, and came
up with the following summary.
The host syntaxes/compiles the new cref, runtime, sf and star-parser
(using the old cref!). The host cold loads the new runtime and dumps
an x-runtime band, then loads the new sf and syntaxes the new LIAR/C.
The host compiles the syntaxed compiler, then loads and dumps an
x-compiler band. The build tree is then cleaned, stashing all
compiler products in STAGEX subdirs, preparing to cross-compile. Just
the boot-dirs (cref, runtime, sf, compiler and star-parser) are
cross-compiled to C "bundles" (and the STAGEX subdirs tossed!).
Making distclean keeps the mit-scheme-c- distribution small.
The target (re)configures the build tree, c-compiles the machine and
bundles, installs the bundles in ./boot-root/lib/mit-scheme-c/lib, and
uses the new machine to cold load and dump a boot-compiler band.
The build tree is again distcleaned, tossing the cross-compiler's
products. The new target uses the bundles and bands in
boot-root/{bin,lib}/ to compile everything, and tosses boot-root/.
* LIAR/svm Build
The LIAR/svm cross-compile is similar.
The host syntaxes/compiles the new runtime, sf, cref and star-parser
(using the old cref). The host cold loads the new runtime and disk-
-saves an x-runtime band, then loads the new sf and syntaxes the new
LIAR/svm. The host compiles the syntaxed compiler, then loads and
dumps an x-compiler band. The build tree is then cleaned, stashing
all compiler products in STAGEX subdirs, preparing to cross-compile.
Just the boot-dirs are cross-compiled to svm1.
The target "finishes" the cross-compiled boot-dirs, then loads and
dumps a boot-compiler band.
The build tree is again cleaned, stashing the cross-compiler products
in STAGE0 subdirs. The new target uses the boot-compiler band to
compile (NOT cross-compile -- by procedure) everything.
* LIAR/native Build
There is a fourth build process, a cross-compile that uses a LIAR/C
host to build a native system. It is described in etc/make-native.sh.
It is the process used to create the first 64bit heap? It does not
seem to be working currently. I have some patches that get the
process through compile-boot-compiler.sh, but the new cross-compiler
craps soon after (because of a surprise in cref/make.bin?). I traced
through the build process anyway.
The host syntaxes/compiles the new runtime, sf, cref and star-parser
(using the old cref). The host cold loads the new runtime and disk-
saves an x-runtime band, then loads the new sf and syntaxes the new
LIAR/native, compiles it to C code, loads it and dumps an x-compiler
band. The build tree is then cleaned, stashing all compiler products
in STAGEX subdirs, preparing to cross-compile. Just the boot-dirs are
cross-compiled to native code (or svm1?).
The target "finishes" the cross-compiled boot-dirs, then cold loads
and disk-saves a boot-compiler machine and band.
The build tree is again cleaned, tossing the cross-compiler's products.
Now the build tree is (re)configured(?) and the machine (re)built?
The new target machine uses the boot-compiler band to compile (NOT
cross-compile -- by procedure) everything.
* Cross Compiling
The general pattern in the three cross-compiles is:
1. Build the cross-compiler on the host.
2. Cross-compile the boot-compiler on the host.
3. Build the boot-compiler on the target.
4. (Re)Compile everything with the boot-compiler.
Step 1 is the trick involving host-compiling the new runtime, sf, cref
and star-parser, then syntaxing the new compiler with the new runtime
(on the host!) and host-compiling the result. It would be more
straightforward if the native and cross compilers could exist in the
same world, separately.
Step 2 assumes the target can fasload anything the cross-compiler
fasdumps (with finish-cross-compilation), but this is not always
possible, as when the s-code types changed recently. The cross-
compiler needs a different (portickle? targeted?) fasdump. The LIAR/C
compiler accomplishes this by dumping Scheme constants as C code that
rebuilds them in the target heap -- a fasdump targeting C.
Steps 3 and 4 may happen on a physically different target machine.
The LIAR/C build stops after Step 2 and prepares an mit-scheme-c-
distribution. The other cross-compiling builds do not stop; they
assume the host and target Scheme machines are on the same physical
machine.
The Riastradh Build is very like Step 1, building a cross-compiler.
It adds the refinement of loading the new cref (but not star-parser?)
on the host before syntaxing and analyzing the new runtime and sf.
And it uses the target instead of the host to syntax the new compiler,
avoiding the need for a cross-runtime -- a new runtime on an old host.
* Scheme48ish Cross Compilation
The Scheme48 cool way to build would address the separation of host
(old) and target (new) compilers. Our cross-compilation achieves that
by syntaxing the new compiler without the old compiler present, and
hoping that the old compiler can compile the resulting syntax. The
new machine could interprete the resulting syntax, and thus compile
the compiler, but... that takes too long.
A Scheme48 cool AND cross build would separate old and new compilers,
loading the new compiler into the host as a separate cross-compiler --
into a (cross compiler) package linked to a (cross runtime) which
would, when necessary, contain a little training to simulate a new
heap in the old host. Cross-compilation would cross-fasdump .mqocs
that the new machine's fasload can handle without finishment. The
cross-fasdumper would be host code expecting simulated new-runtime
types and could bridge (be trained for) any change in heap format.
The resulting .mocs could be simply renamed .coms on the target. The
new machine would load them into (compiler), (runtime), etc. packages
and pretend they were compiled by (compiler) on the new machine -- not
by (cross compiler) on a host machine. Or they could continue to be
the mostly-native boot-compiler, tossed after it re-compiles
everything. Or we could just check that there is no difference
between the .mocs and the boot-compiler compiled .coms (and the .coms
compiled by the compiler built by the boot-compiler?!).
* Coming At You
I found star-parser being compiled late in the build. I think there
is a missing dependency that would get it pre-loaded in the host like
cref, before syntaxing the new runtime. Thus a new runtime could use
a new star-parser operator, a situation which recently caused
problems. Better yet: star-parser should be in runtime/ where it can
be loaded before it is used.
I don't know about converting compile-svm.sh to a set of
parallelizable makefile rules. The need for Stage.sh makes it... not
obvious to me. My only example is the pair of scripts
make-lairc-dist.sh and make-liarc.sh.
I would fix make-native.sh EXCEPT that I am not sure how useful it is
to build native systems on a LIAR/C host. LIAR/C does not seem to be
a general-purpose system. Its fasdump-to-C mechanism has limitations?
But it is currently our only way across the 32/64bit divide?
I would work on a cross-fasdumper, but I will need some really cool
heap format changes to motivate me... "really cool" like eliminating
SEQUENCE-3.
LIAR/C and separate compilation can stay limited, bootstrapping
techniques. I drool for a modern luser's "just load it" build
abstraction where compilation is just part of loading/evaluating.
Autoloading pre-compiled "options" is so... last millenium.
If there is any objection to moving the star-parser into runtime/, now
is the time to stop me. :-) And if there is any pressing need for a
working make-native.sh, now is the time to ask.
Yours In Scheme,
-Friar Puck
Just for reference, here are my tracings of the four build processes.
* Riastradh build, detailed trace
make
make lib/runtime.com
make compile-runtime
make syntax-runtime
make compile-cref
host: (compile-dir "cref")
host: re-load (new) cref, then (load "runtime.sf")
host: (load "runtime.cbf")
make microcode/scheme
make compile-microcode
(cd microcode && make all)
target: cold load new runtime, (disk-save "../lib/runtime.com")
make lib/all.com
make compile-compiler
make syntax-compiler
make compile-sf
make compile-cref, up-to-date
make syntax-runtime, up-to-date
host: re-load (new) cref, then (compile-dir "sf")
make lib/runtime.com, up-to-date
make microcode/scheme, up-to-date
target: load new sf, then (load "compiler.sf")
host: (load "compiler.cbf")
make compile-edwin
make bootstrap-toolchain
make lib/compiler.com
make compile-compiler, up-to-date
make lib/runtime.com, up-to-date
make microcode/scheme, up-to-date
target: (load-option 'compiler) (disk-save "compiler.com")
make microcode/scheme, up-to-date
make syntax-edwin
make bootstrap-toolchain, up-to-date
make compile-xml
make bootstrap-toolchain, up-to-date
make compile-sos
make bootstrap-toolchain, up-to-date
target: (compile-dir "sos")
make compile-star-parser
make compile-cref, up-to-date
make syntax-runtime, up-to-date
host: re-load (new) cref, then (compile-dir "star-parser")
target: (compile-dir "xml")
make syntax-win32
make bootstrap-toolchain, up-to-date
target: (load "win32.sf")
target: (load "edwin.sf")
target: (load "edwin.cbf")
make compile-sf, up-to-date
make lib/runtime.com, up-to-date
make microcode/scheme, up-to-date
target: (load-options '(COMPILER EDWIN)) (disk-save "all.com")
make compile-compiler, up-to-date
make compile-cref, up-to-date
make compile-ffi
make bootstrap-toolchain, up-to-date
make compile-cref, up-to-date
target: (compile-dir "ffi")
make compile-imail
make bootstrap-toolchain, up-to-date
make compile-sos, up-to-date
make compile-star-parser, up-to-date
make compile-edwin, up-to-date
target: (load-option 'edwin) (compile-dir "imail")
make compile-runtime, up-to-date
make compile-sf, up-to-date
make compile-sos, up-to-date
make compile-ssp
make bootstrap-toolchain, up-to-date
make compile-xml, up-to-date
target: (compile-dir "ssp")
make compile-star-parser, up-to-date
make compile-win32
make bootstrap-toolchain, up-to-date
make syntax-win32, up-to-date
target: (load "win32.cbf")
make compile-xml, up-to-date
Done.
* LIAR/C build, detailed trace
make-liarc-dist.sh
./Setup.sh
configure --enable-native-code=c
make liarc-dist
etc/compile-boot-compiler.sh
host: (for-each compile-dir '(cref runtime star-parser sf))
host: (disk-save "x-runtime.com")
host.x-runtime: (load "compiler.sf")
host: (load "compiler.cbf")
host.x-runtime: (disk-save "x-compiler.com")
Stage.sh make X
etc/c-prepare.sh
host.x-compiler:(for-each c-compile-dir
'(runtime star-parser sf cref compiler))
Stage.sh remove X
make distclean
make-liarc.sh
./configure --enable-native-code=c --prefix=./boot-root
make stamp_install-liarc-boot-compiler
make stamp_build-liarc-boot-compiler
make stamp_install-liarc-boot-bundles
make stamp_compile-liarc-boot-bundles
make stamp_install-microcode
make compile-microcode
(cd microcode && make all)
make install in microcode/
make compile-liarc-bundle foreach LIARC_BOOT_BUNDLES
make bundle.so
make module.o ...
microcode/liarc-cc module.o module.c -I ...
microcode/liarc-ld bundle.so module.o ...
make install-liarc-bundle foreach LIARC_BOOT_BUNDLES
make bundle.so, up-to-date
mkdir libdir/mit-scheme-c/lib
install bundle.so libdir/mit-scheme-c/lib
etc/build-boot-compiler.sh
target: cold load and disk-save lib/boot-compiler.com
make install-auxdir-top
install lib/*.com ./boot-root/lib/mit-scheme-c/
make c-clean
make distclean
./configure --enable-native-code=c
make stamp_compile-liarc-bundles
make stamp_liarc-compile-scheme
src/etc/c-compile.sh boot-compiler
boot: (c-compile)
(in-liarc... (compile-all-dirs c-compile-dir) ...)
make compile-liarc-bundles
make compile-microcode, up-to-date
make compile-liarc-bundle foreach LIARC_BUNDLES
make bundle.so
make module.o ...
microcode/liarc-cc module.o module.c -I ...
microcode/liarc-ld bundle.so module.o ...
make build-bands
etc/build-bands.sh
target: cold load and disk-save src/lib/runtime.com
target: load all and disk-save src/lib/all.com
make clean-boot-root
Done
* LIAR/svm build, detailed trace
make all-svm
make microcode/svm1-defns.h
make compile-microcode
(cd microcode && MAKE all)
etc/compile-svm.sh
host: load runtime.sf, sf.sf and cref.sf
host: load runtime.cbf, sf.cbf and cref.cbf
host: load star-parser/compile.scm
host: disk-save x-runtime.com
host.x-runtime: load compiler.sf, sf base/crsend
host: load "compiler.cbf"
host.x-runtime: disk-save x-compiler
./Stage.sh make X
host.x-compiler: boot-dirs
target: (finish-cross-compilation:directory ".")
target: cold load boot-dirs, disk-save boot-runtime, boot-compiler
./Stage.sh make-cross 0
target --band boot-compiler: (compile-everything)
make build-bands
target: cold load and disk-save src/lib/runtime.com
target: load all and disk-save src/lib/all.com
Done.
* LIAR/native[or svm] build on LIAR/C, detailed trace
etc/make-native.sh
./Setup.sh
./configure --prefix=`pwd`/boot-root
etc/compile-boot-compiler.sh
host: (for-each compile-dir '(cref runtime star-parser sf))
host: (disk-save "x-runtime.com")
host.x-runtime: (load "compiler.sf")
host: (load "compiler.cbf")
host.x-runtime: (disk-save "x-compiler.com")
./Stage.sh make X
cd compiler && make compile-liarc-bundle
# Error: no compiler-unx.c, needed by compiler-init.c
#
# Should have been produced by cbf-conditionally compiling
# (fasdumping to C) compiler-unx.pkd, e.g. by
# c-compile-dir in etc/compiler.scm.
etc/native-prepare.sh
host.x-compiler: (native-prepare)
compile-boot-dirs with compiler:cross-compiling? #t
make compile-microcode
target: finish-cross-compilation:directory
make stamp_install-native-boot-compiler
make stamp_build-native-boot-compiler
make stamp_install-microcode
make compile-microcode
cd microcode && make install
etc/build-boot-compiler.sh
target: cold load and disk-save lib/boot-compiler.com
make install-auxdir-top
install lib/*.com ./boot-root/lib/mit-scheme-*/
make c-clean
make distclean
./configure
make stamp_native-compile-scheme
make compile-microcode
etc/compile.sh boot-root/bin/mit-scheme --library boot-root/lib/...
boot-compiler: compile-cref, runtime, star-parser, sf
target: (disk-save "runtime.com")
target: (load "compiler.sf")
boot-compiler: (load "compiler.cbf")
target: compile-remaining-dirs
make build-bands
make clean-boot-root
_______________________________________________
MIT-Scheme-devel mailing list
MIT-Scheme-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/mit-scheme-devel
