Hi,
On 10/13/2011 01:30 PM, ext Carsten Munk wrote:
2011/10/12 Eero Tamminen<eero.tammi...@nokia.com>:
On 10/12/2011 10:56 AM, ext Carsten Munk wrote:
To avoid the trap of the static toolset, it was chosen to take the
existing x86-built binaries in OBS, build-dep on them and simply
(through OBS methods) export them from the X86 build to the ARM build
scheduler, replacing the ARM binaries in place with X86 binaries,
which had been rewritten to use a different rpath [4, also documents
cross toolchain method].
This is what I really don't like. Repository or rootstrap shouldn't
be "a random" mix of packages for different architectures.
>
Well, it's not really a random mix - but in truth, isn't it exactly
what SB1/2 does as well? Mixing binaries.
They don't have x86 & arm packages in a repository for ARM packages and
x86 accel packages claiming to be noarch like OBS does?
And such extra packages don't need to be installed / end up in
the target package database, they can be either bind mounted (SB1)
or path mapped (SB2).
First one is bit of purity thing (I don't know has / will it create
problems), second one may slowdown build speed slightly.
Keep in mind that the -x86 packages aren't the full packages of their
non-x86 counterparts.
carsten@ideapad:~/Downloads/f/ii$ rpm2cpio
../../file-x86-arm-5.04-1.1.Mer.armv7l.rpm | cpio -idv
./emul/ia32-linux/usr
./emul/ia32-linux/usr/bin
./emul/ia32-linux/usr/bin/file
./emul/ia32-linux/usr/share
./emul/ia32-linux/usr/share/applybinary
./emul/ia32-linux/usr/share/applybinary/file-x86
>
This is the 'file' binary - exactly same as in the "file" package's
x86 counterpart rpath'd to be using /emul/ia32-linux instead.
Ok, so this directory contains also the libs needed by these
binaries...
They are -NOT- ./configure --prefix'ed to another path
>
This meant that we were always sure, that the
accelerated X86 binary on ARM side would match the original ARM
binary's source version.
Ack.
This approach is no different from what is recommended for SB2
currently (on the mailing list, it's documentation is quite lacking).
Except that:
- you mix in the same place x86& ARM packages
Not exactly true, see above
- because of above, you need to rename acceleration packages
No, they're actually more like overlays - we actually install both ARM
package and package-x86
Yes, I meant that you have separate acceleration packages.
With SB2 those aren't needed, you can use your normal x86
packages, they don't need to be modified.
- your x86 binaries are in different path so that they don't
conflict with target ones
That's partly true - we have both ARM and accelerated binaries in the
resulting chroot, but as said before, they overlay.
Let me drag out the %post of such a resulting package:
carsten@ideapad:~/Downloads/f/ii$ rpm -q --scripts -p
../../file-x86-arm-5.04-1.1.Mer.armv7l.rpm
warning: ../../file-x86-arm-5.04-1.1.Mer.armv7l.rpm: Header V3
DSA/SHA1 Signature, key ID 0a94df2f: NOKEY
postinstall scriptlet (using /bin/sh):
/sbin/ldconfig
mv /usr/bin/file /usr/bin/file.orig-arm ; ln -s
/emul/ia32-linux/usr/bin/file /usr/bin/file
preuninstall scriptlet (using /bin/sh):
rm -f /usr/bin/file ; mv /usr/bin/file.orig-arm /usr/bin/file
>
As you can see, it basically takes the ia32 binary and dumps it in
place of the ARM binary.
I assume:
- above postinstall/preuninstall stuff is done only for binaries,
not for libraries needed by them (RPATH should handle that)
- before building a release image, you remove the packages
(small release image build slowdown) which reverts the changes.
?
-> you can leak wrong paths to generated scripts like SB1 does
>
I think this one is a bit misunderstood, we don't actually use PATH
tricks or similar, we replace the ARM binaries with their X86
counterparts, compiled to exactly same
With symlinks you can still get wrong paths if something uses
readlink(). Postinstall moving the file would be a bit safer than
symlink.
- your control is on the package level, you cannot have more
fine-grained control. That for example prevents accelerating
more packages
We're already replacing binaries on fine-grained level of filesystem,
I'd say (see above)
Ack.
- you cannot enforce/detect issues when acceleration packages cross
the lines they shouldn't[1].
?
[1] This is what newer SB2 versions have been paying some attention.
Latest added for example some support for detecting/controlling
whether builds access internet as some packages like to "call home"
when they're built...
>
You can do that with KVM if you really wanted - most people just opt
to block network access instead of the worker
Another, much earlier SB2 feature is preventing packages from writing
to files outside of their build directory. Some packages modify
on install files in the root file system...
Does OBS already have something for that?
If latest SB2 gets proper RPM support, would there be interest to
try that with OBS?
>
Of course, we're trying to avoid NIH and if there's better
technologies that is of course of interest - getting them evaluated in
an open manner.
What you have sounds actually much better than I though. About
minimal I can think of for getting builds accelerated.
Another issue that chroot based things have, is /etc contents.
For target you need it to correspond to what's on target, for
accessing host device files, doing networking etc, it needs to
correspond to what your host setup has. How OBS handles that?
Btw. SB2 has some downsides, mainly related to LD_PRELOAD use.
LD_PRELOAD means that:
- SB2 is somewhat dependent on the used C-library as different C-library
(versions) may require wrapping different functions to catch all file
accesses (also indirect ones).
- If SB2 is used only for ARM builds, as it should, above issue is
restricted just to file accessing C-library calls that are done by
dynamically linked Qemu i.e. fairly easily manageable. This way one
also avoids many other issues related to LD_PRELOAD usage like:
- handling static binaries
- bad C++ libraries with (static object) constructors that
may do file accesses before LD_PRELOAD takes an effect
- LD_PRELOAD usage may cause couple of percent slowdown compared to
a chroot approach not using LD_PRELOAD (SB1 used LD_PRELOAD for
/etc accesses and for fakeroot).
Maybe the main SB2 benefit would come from its debugging features.
You can get much better information about what happens in your
builds. For example:
- what's the call hierarchy,
- what takes time (actual names of scripts & ARM binaries, not just
names of Qemu and Perl like you would get with profiler),
- is something trying to write outside the build directory
etc.
It would be nice if OBS would have "pluggable" cross-compilation
support for things like that.
Btw. Does OBS support sbrsh in addition to Qemu, in case one would
want to do a port to an architecture not yet supported by Qemu?
How up to date the last two pages are e.g. in regards to what binaries
are accelerated? They mention armv5 toolchains...
Same method for toolchains is still in use, but has grown since that
documentation was written. We now also accelerate RPM installs and RPM
building - the most slow part of builds is the chroot setup.
If your build host has enough RAM, you could have the target rootfs
(where build dependency packages are installed) on ramfs, this way there
would be no disk writes for build deps and the data copied there would
in best case come from the kernel file cache.
Or are you already doing that?
http://monster.tspre.org:2080/project/packages?project=Core%3Aarmv7l -- find
packages with "-x86" and it gives you a good idea what's accelerated -
you'd have to look into binaries_to_prepare in each package to figure
out what binaries they specifically accelerate.
- Eero
_______________________________________________
MeeGo-dev mailing list
MeeGo-dev@meego.com
http://lists.meego.com/listinfo/meego-dev
http://wiki.meego.com/Mailing_list_guidelines
