Re: [OE-core] [PATCH 2/8] initscripts: add setup-commands.sh
On Mon, 2013-11-11 at 10:52 +0800, ChenQi wrote:
On 11/10/2013 07:00 AM, Phil Blundell wrote:
1. initscript doesn't obviously rdepend on busybox so it's not obvious
that the latter will always be available;
Yes. Initscript doesn't rdepend on busybox. But note it also doesn't
rdepend on sed or awk or grep.
So I think it's reasonable to assume the presence of busybox.
I think one could argue that it's also a bug that it doesn't rdepend on
the three things you mentioned (and indeed on /bin/sh itself, for
non-rpm systems). However, the usage of grep and sed in particular, and
perhaps to a slightly lesser extent awk, is so deeply ingrained into so
much software that I think it's probably fair to say those utilities
will almost always be present.
By contrast, it is perfectly feasible to build a system which doesn't
use busybox (by using the full GNU implementations of everything that
busybox provides) and indeed I think there might even be a task package
in oe-core which does exactly that. So it seems entirely possible
that /bin/busybox might not be installed unless you have a dependency on
it.
2. it should probably be using ${base_bindir} and ${bindir} rather than
hardcoding absolute paths.
In init scripts, we usually hardcode things, because these scripts
are destined to run on target.
In recipes we try not to hardcode things because the recipe may need
to extend to native or nativesdk.
I'm not quite sure I follow the logic here. Even if the script is
intended to run on the target it still ought to be respecting the values
of ${bindir}, ${sysconfdir} and such like.
3. the whole idea of creating a shadow /usr/bin underneath what's
meant to be a mountpoint seems rather dubious to me.
Agree.
The problem here is that the init scripts under /etc/rcS.d/ need to
execute commands like awk, dirname, and readlink which are from /usr.
As it's not appropriate to move these commands into /bin, basically
there are only two options I can see here. One is to modify these
scripts to use only commands from /bin or /sbin; the other is to make
use of busybox, as busybox is located under /bin as it provides these
commands.
I chose to the latter one because I thought that solution would have
the less impact.
What do you think? Do we need to modify the init scripts? Or any other
solution?
I thought that last time this topic came up on the mailing list, the
eventual conclusion was that Wind River (being more-or-less the only
people who seemed to feel strongly that supporting / and /usr on
different partitions was important) were going to come up with some sort
of overall strategy for solving the whole problem rather than just
fixing up individual bits in a piecemeal fashion. I think that strategy
would need to include a policy for which utilities initscripts can
legitimately expect to be available before /usr is mounted, and if this
set is different from what we have today then it would need to include a
plan for sorting that out.
To answer the particular question at hand it seems that someone would
need to do some analysis of things like:
- how many scripts actually need those commands
- how hard would it be to change them to not need them
- what would be the impact of moving them into ${base_bindir} (for
example, would this cause a whole load of libraries to get dragged into
${base_libdir} as well?)
Offhand I don't know the answer to any of those things.
4. this seems like distro policy and not something that really belongs
in oe-core at all. For systems where ${bindir} and ${base_bindir} are
on the same filesystem (or even are the same directory) this script will
just make bootup slower without achieving anything useful.
If /usr and / are on the same file system, this script has no real
effect because /usr will always be there. So I think it will not take
much time at boot.
Just spawning a new copy of the shell and reading the script does take a
finite (and measureable) time. If you can determine statically at build
time that the script isn't going to do anything useful then I don't
think it's appropriate to install it.
If ${base_bindir} and ${bindir} are the same, that means that there's
no /usr. In this case, there should no /usr/xxx entries
in /etc/busybox.links, so this script should also function correctly
and it will not take much time at boot.
(I just configured ${bindir} and ${base_bindir} to be the same and
performed a build, it failed. I'm not sure whether it's valid to make
such configurations in OE.)
It is a valid configuration, and this is the way that most of the images
I build are configured. It probably is true that there are still some
number of recipes in oe-core that don't support this properly, but I
don't want to see that situation get any worse.
p.
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
Re: [OE-core] [PATCH 2/8] initscripts: add setup-commands.sh
On 11 November 2013 02:52, ChenQi qi.c...@windriver.com wrote: The problem here is that the init scripts under /etc/rcS.d/ need to execute commands like awk, dirname, and readlink which are from /usr. Yes. So why do you really need to support split /usr, and why isn't an initramfs a suitable alternative? People seem to insist that split /usr is trivial but these patches are clearly showing it's not. Ross ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 2/8] initscripts: add setup-commands.sh
Hi Phil,
First of all, thank you for your careful review and explanation.
To conclude, you suggest modifying the init scripts and moving commands
around if needed, right?
And please see comments inline.
On 11/11/2013 07:53 PM, Phil Blundell wrote:
On Mon, 2013-11-11 at 10:52 +0800, ChenQi wrote:
On 11/10/2013 07:00 AM, Phil Blundell wrote:
1. initscript doesn't obviously rdepend on busybox so it's not obvious
that the latter will always be available;
Yes. Initscript doesn't rdepend on busybox. But note it also doesn't
rdepend on sed or awk or grep.
So I think it's reasonable to assume the presence of busybox.
I think one could argue that it's also a bug that it doesn't rdepend on
the three things you mentioned (and indeed on /bin/sh itself, for
non-rpm systems). However, the usage of grep and sed in particular, and
perhaps to a slightly lesser extent awk, is so deeply ingrained into so
much software that I think it's probably fair to say those utilities
will almost always be present.
By contrast, it is perfectly feasible to build a system which doesn't
use busybox (by using the full GNU implementations of everything that
busybox provides) and indeed I think there might even be a task package
in oe-core which does exactly that. So it seems entirely possible
that /bin/busybox might not be installed unless you have a dependency on
it.
Agree. I'm gonna drop this patch and try to come up with another solution :)
2. it should probably be using ${base_bindir} and ${bindir} rather than
hardcoding absolute paths.
In init scripts, we usually hardcode things, because these scripts
are destined to run on target.
In recipes we try not to hardcode things because the recipe may need
to extend to native or nativesdk.
I'm not quite sure I follow the logic here. Even if the script is
intended to run on the target it still ought to be respecting the values
of ${bindir}, ${sysconfdir} and such like.
3. the whole idea of creating a shadow /usr/bin underneath what's
meant to be a mountpoint seems rather dubious to me.
Agree.
The problem here is that the init scripts under /etc/rcS.d/ need to
execute commands like awk, dirname, and readlink which are from /usr.
As it's not appropriate to move these commands into /bin, basically
there are only two options I can see here. One is to modify these
scripts to use only commands from /bin or /sbin; the other is to make
use of busybox, as busybox is located under /bin as it provides these
commands.
I chose to the latter one because I thought that solution would have
the less impact.
What do you think? Do we need to modify the init scripts? Or any other
solution?
I thought that last time this topic came up on the mailing list, the
eventual conclusion was that Wind River (being more-or-less the only
people who seemed to feel strongly that supporting / and /usr on
different partitions was important) were going to come up with some sort
of overall strategy for solving the whole problem rather than just
fixing up individual bits in a piecemeal fashion. I think that strategy
would need to include a policy for which utilities initscripts can
legitimately expect to be available before /usr is mounted, and if this
set is different from what we have today then it would need to include a
plan for sorting that out.
Basically we don't have any problem if /usr is going to be mounted,
because the mountall.sh takes place really early.
What I want to achieve here is that we can boot into single user mode
for recovery or repair if /usr is not available (maybe just because the
drive's broken).
To answer the particular question at hand it seems that someone would
need to do some analysis of things like:
- how many scripts actually need those commands
- how hard would it be to change them to not need them
Only init scripts that have links under /etc/rcS.d/ need to be examined,
so that wouldn't be a lot of work.
I'm just not sure whether we should make such a restriction of which
commands could be used and which could not. After all, commands like
`readlink', `dirname' and `awk' are also very commonly used commands.
- what would be the impact of moving them into ${base_bindir} (for
example, would this cause a whole load of libraries to get dragged into
${base_libdir} as well?)
If I can easily modify the init scripts to make things work, I don't
want to move things around.
But I guess statements using awk are hard to be replaced unless I use a
block of codes
Offhand I don't know the answer to any of those things.
4. this seems like distro policy and not something that really belongs
in oe-core at all. For systems where ${bindir} and ${base_bindir} are
on the same filesystem (or even are the same directory) this script will
just make bootup slower without achieving anything useful.
If /usr and / are on the same file system, this script has no real
effect because /usr will always be there. So I think it will not take
much time at boot.
Just
Re: [OE-core] [PATCH 2/8] initscripts: add setup-commands.sh
On Mon, 2013-11-11 at 20:40 +0800, ChenQi wrote:
Hi Phil,
First of all, thank you for your careful review and explanation.
To conclude, you suggest modifying the init scripts and moving commands
around if needed, right?
Well, I'm not sure that I'm necessarily suggesting any particular
change. What I'm saying is that if we're going to try to solve the
split-/usr problem in oe-core (as opposed to Wind River just doing it in
their internal distro layers) then:
1. It needs to be done in a way that doesn't have any negative impact
for people who don't have a split /usr (or indeed who don't have /usr at
all)
2. The changes need to be presented as a coherent package with the
reasoning clearly explained.
3. We need to be clear that it's solving the whole problem (whatever
that might be) rather than just some part of it.
Could you please share your configuration so that when I make V2 I can
test the case of ${bindir} and ${base_bindir} being the same.
(Currently I really can't build out an image with such configuration ... )
I think it should basically just be a case of setting:
prefix =
exec_prefix =
prefix_native =
in your distro configuration.
p.
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 2/8] initscripts: add setup-commands.sh
On 11/11/13, 5:53 AM, Phil Blundell wrote:
On Mon, 2013-11-11 at 10:52 +0800, ChenQi wrote:
On 11/10/2013 07:00 AM, Phil Blundell wrote:
...
I thought that last time this topic came up on the mailing list, the
eventual conclusion was that Wind River (being more-or-less the only
people who seemed to feel strongly that supporting / and /usr on
different partitions was important) were going to come up with some sort
of overall strategy for solving the whole problem rather than just
fixing up individual bits in a piecemeal fashion. I think that strategy
would need to include a policy for which utilities initscripts can
legitimately expect to be available before /usr is mounted, and if this
set is different from what we have today then it would need to include a
plan for sorting that out.
There is a bug open for the Yocto Project 1.6 to explicitly do this. Come up
with an overall strategy for these kinds of things, as well as a test plan to
verify that whatever we end up doing works properly long term.
As you said, busybox is not required on a system -- just something that has a
reasonable set of software packages. Also a lot of this stuff is simply limited
to 'early boot'. At some point we do need to define 'early boot'. (Generally I
define it as until /usr would normally be mounted.)
And unfortunately you will see patches in piecemeal at this point. Until such a
strategy is generated, everything is being done reactively. We see a QA
error/warning and someone is assigned to solve it. Yocto Project Compliance
(and our own internal) guidelines then indicate since we've patched something it
has to go out to the community.
To answer the particular question at hand it seems that someone would
need to do some analysis of things like:
- how many scripts actually need those commands
- how hard would it be to change them to not need them
- what would be the impact of moving them into ${base_bindir} (for
example, would this cause a whole load of libraries to get dragged into
${base_libdir} as well?)
That was my thought exactly. Lets look at moving things -or- stop using them.
Whichever is more reasonable.
Offhand I don't know the answer to any of those things.
4. this seems like distro policy and not something that really belongs
in oe-core at all. For systems where ${bindir} and ${base_bindir} are
on the same filesystem (or even are the same directory) this script will
just make bootup slower without achieving anything useful.
If /usr and / are on the same file system, this script has no real
effect because /usr will always be there. So I think it will not take
much time at boot.
Just spawning a new copy of the shell and reading the script does take a
finite (and measureable) time. If you can determine statically at build
time that the script isn't going to do anything useful then I don't
think it's appropriate to install it.
If ${base_bindir} and ${bindir} are the same, that means that there's
no /usr. In this case, there should no /usr/xxx entries
in /etc/busybox.links, so this script should also function correctly
and it will not take much time at boot.
(I just configured ${bindir} and ${base_bindir} to be the same and
performed a build, it failed. I'm not sure whether it's valid to make
such configurations in OE.)
It is a valid configuration, and this is the way that most of the images
I build are configured. It probably is true that there are still some
number of recipes in oe-core that don't support this properly, but I
don't want to see that situation get any worse.
And yes, all scripts need to (at build time) bind to the bindir and base_bindir
of that distribution. Hardcoding those values into the scripts before build
time is wrong.
--Mark
p.
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 2/8] initscripts: add setup-commands.sh
On 11/11/13, 6:12 AM, Burton, Ross wrote: On 11 November 2013 02:52, ChenQi qi.c...@windriver.com wrote: The problem here is that the init scripts under /etc/rcS.d/ need to execute commands like awk, dirname, and readlink which are from /usr. Yes. So why do you really need to support split /usr, and why isn't an initramfs a suitable alternative? People seem to insist that split /usr is trivial but these patches are clearly showing it's not. Just to be clear, I've never said it was trivial -- but what I have said is that it -should- be trivial if people pay attention to this stuff early. The only thing that has to be done for the split is 'early boot'. (See my email response to Phil..) As for why, we still have people developing systems with small per-device memory that need to do the early boot from a small system, then mount the /usr partition from a shared location. (/usr is often times still mounted RO.. and is shared between multiple CPUs within a single device or chasis.) The usage of this is getting less and less, but we do still get issues from customers when things don't work right, so we know people are still doing it. --Mark Ross ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 2/8] initscripts: add setup-commands.sh
On 11/10/2013 07:00 AM, Phil Blundell wrote:
On Sat, 2013-11-09 at 13:28 +0800, qi.c...@windriver.com wrote:
+for dir in /usr/bin /usr/sbin; do
+if [ ! -e $dir ]; then
+ if [ $VERBOSE != no ]; then
+ echo WARN: $dir missing, setting up links to busybox
+ fi
+ mkdir -p $dir
+ for suffix in .nosuid .suid ; do
+ if [ ! -e /etc/busybox.links${suffix} ]; then
+ continue
+ fi
+ usr_commands=`grep $dir /etc/busybox.links${suffix}`
+ for command in $usr_commands; do
+ ln -sf /bin/busybox${suffix} $command
+ done
Hi Phil,
This script is an effort to try to make system boot up even if /usr is
not there.
It relies on busybox because in most cases, busybox is there in an OE
based system.
And please see comments below.
This seems slightly bogus for a number of reasons:
1. initscript doesn't obviously rdepend on busybox so it's not obvious
that the latter will always be available;
Yes. Initscript doesn't rdepend on busybox. But note it also doesn't
rdepend on sed or awk or grep.
So I think it's reasonable to assume the presence of busybox.
2. it should probably be using ${base_bindir} and ${bindir} rather than
hardcoding absolute paths.
In init scripts, we usually hardcode things, because these scripts are
destined to run on target.
In recipes we try not to hardcode things because the recipe may need to
extend to native or nativesdk.
3. the whole idea of creating a shadow /usr/bin underneath what's
meant to be a mountpoint seems rather dubious to me.
Agree.
The problem here is that the init scripts under /etc/rcS.d/ need to
execute commands like awk, dirname, and readlink which are from /usr.
As it's not appropriate to move these commands into /bin, basically
there are only two options I can see here. One is to modify these
scripts to use only commands from /bin or /sbin; the other is to make
use of busybox, as busybox is located under /bin as it provides these
commands.
I chose to the latter one because I thought that solution would have the
less impact.
What do you think? Do we need to modify the init scripts? Or any other
solution?
4. this seems like distro policy and not something that really belongs
in oe-core at all. For systems where ${bindir} and ${base_bindir} are
on the same filesystem (or even are the same directory) this script will
just make bootup slower without achieving anything useful.
p.
If /usr and / are on the same file system, this script has no real
effect because /usr will always be there. So I think it will not take
much time at boot.
If ${base_bindir} and ${bindir} are the same, that means that there's no
/usr. In this case, there should no /usr/xxx entries in
/etc/busybox.links, so this script should also function correctly and it
will not take much time at boot.
(I just configured ${bindir} and ${base_bindir} to be the same and
performed a build, it failed. I'm not sure whether it's valid to make
such configurations in OE.)
Best Regards,
Chen Qi
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 2/8] initscripts: add setup-commands.sh
On Sat, 2013-11-09 at 13:28 +0800, qi.c...@windriver.com wrote:
+for dir in /usr/bin /usr/sbin; do
+if [ ! -e $dir ]; then
+ if [ $VERBOSE != no ]; then
+ echo WARN: $dir missing, setting up links to busybox
+ fi
+ mkdir -p $dir
+ for suffix in .nosuid .suid ; do
+ if [ ! -e /etc/busybox.links${suffix} ]; then
+ continue
+ fi
+ usr_commands=`grep $dir /etc/busybox.links${suffix}`
+ for command in $usr_commands; do
+ ln -sf /bin/busybox${suffix} $command
+ done
This seems slightly bogus for a number of reasons:
1. initscript doesn't obviously rdepend on busybox so it's not obvious
that the latter will always be available;
2. it should probably be using ${base_bindir} and ${bindir} rather than
hardcoding absolute paths.
3. the whole idea of creating a shadow /usr/bin underneath what's
meant to be a mountpoint seems rather dubious to me.
4. this seems like distro policy and not something that really belongs
in oe-core at all. For systems where ${bindir} and ${base_bindir} are
on the same filesystem (or even are the same directory) this script will
just make bootup slower without achieving anything useful.
p.
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
