Re: [OE-core] sstate_clean() overzealous?
On Mon, 2011-10-03 at 17:47 +0100, Phil Blundell wrote:
The sstate_clean() function in sstate.bbclass is doing (inter alia):
stfile = d.getVar(STAMP, True) + .do_ + ss['task']
[...]
oe.path.remove(stfile + .*)
oe.path.remove(stfile + _setscene + .*)
which means that, for tasks which set the stamp-extra-info flag to
${MACHINE}, it ends up blowing away the stamps for all machines rather
than just the current one. The net effect of this seems to be that
there is no way to have the setscene stamps populated for more than one
MACHINE at any time, and hence all those tasks get rerun every time you
change MACHINE even if nothing else has been altered.
Is this behaviour deliberate?
No, its not intended to be wiping out all the different machines.
It is certainly a little bit annoying but
I don't understand the internals of sstate well enough to judge what
exactly that glob is trying to accomplish.
The trouble is we now support a variety of different stamp file formats
depending on what settings you have enabled (and metadata you're using).
Clean can be called from two sources, do_clean, or just before a sstate
controlled task is about to be run. In either case we want to clear out
any previous state including any stamp files that might have
represented this task since we're about to invalidate anything they did.
This means wiping out normal stamps and any setscene variants of those
stamps. We also want to remove any stamps representing other hashes for
this task since there is no version installed.
If the BasicHash signature generator code is enabled this adds hashes to
the stamp files. This turns the stamps into the form:
stampfile.hash
or in the case of machine specific tasks:
stampfile.hash.machine
I think we need to teach this clean function about stamp-extra-info and
instead of stfile + .*, remove stfile + .*. + stamp-extra-info if
stamp-extra-info is set. That would likely fix the problem you're
seeing.
Cheers,
Richard
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] sstate_clean() overzealous?
On Tue, 2011-10-04 at 11:54 +0100, Richard Purdie wrote: I think we need to teach this clean function about stamp-extra-info and instead of stfile + .*, remove stfile + .*. + stamp-extra-info if stamp-extra-info is set. That would likely fix the problem you're seeing. Okay, thanks. I applied this change locally: diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 91f209a..7f38800 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -258,10 +258,15 @@ def sstate_clean(ss, d): bb.utils.unlockfile(lock) stfile = d.getVar(STAMP, True) + .do_ + ss['task'] +extrainf = d.getVarFlag(do_ + ss['task'], 'stamp-extra-info') oe.path.remove(stfile) oe.path.remove(stfile + _setscene) -oe.path.remove(stfile + .*) -oe.path.remove(stfile + _setscene + .*) +if extrainf: +oe.path.remove(stfile + .*. + extrainf) + oe.path.remove(stfile + _setscene + .*. + extrainf) +else: +oe.path.remove(stfile + .*) + oe.path.remove(stfile + _setscene + .*) CLEANFUNCS += sstate_cleanall and it does indeed seem to resolve the problem I was seeing before. p. ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] sstate_clean() overzealous?
On Tue, 2011-10-04 at 16:03 +0100, Phil Blundell wrote: On Tue, 2011-10-04 at 11:54 +0100, Richard Purdie wrote: I think we need to teach this clean function about stamp-extra-info and instead of stfile + .*, remove stfile + .*. + stamp-extra-info if stamp-extra-info is set. That would likely fix the problem you're seeing. Okay, thanks. I applied this change locally: diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 91f209a..7f38800 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -258,10 +258,15 @@ def sstate_clean(ss, d): bb.utils.unlockfile(lock) stfile = d.getVar(STAMP, True) + .do_ + ss['task'] +extrainf = d.getVarFlag(do_ + ss['task'], 'stamp-extra-info') oe.path.remove(stfile) oe.path.remove(stfile + _setscene) -oe.path.remove(stfile + .*) -oe.path.remove(stfile + _setscene + .*) +if extrainf: +oe.path.remove(stfile + .*. + extrainf) + oe.path.remove(stfile + _setscene + .*. + extrainf) Thinking further about this it may need to be .* instead of .*.. I suspect with the above it wouldn't be wiping out some of the stamps it should be with OE-Core and the Basic siggen code. BasicHash would work though. Cheers, Richard ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] sstate_clean() overzealous?
On Mon, 2011-10-03 at 22:08 +, McClintock Matthew-B29882 wrote:
On Mon, Oct 3, 2011 at 4:29 PM, Matthew McClintock m...@freescale.com wrote:
On Mon, Oct 3, 2011 at 4:24 PM, Saul Wold saul.w...@intel.com wrote:
I am not sure that's correct, since we use sstate with the Autobuilder and
it's shared across different hosts building different MACHINEs, we don't
see
the -natives getting build, do you have any examples?
I'm building with targets that are not upstream, so I can't comment on
the upstream variant. There are different TUNES though. I'm actually
currently in the process of looking at why sstate-cache is not being
reused.
Can you point at the autobuilder stuff? Is it going from
fsl-mpc8315e-rdb to and x86_64 target? Or between two x86_64 targets?
For example autoconf-native:
$ bitbake-diffsigs
../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo
[mattsm@right build_p5020ds_release (master *$)]$ bitbake-diffsigs
../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo
[snip]
basehash changed from bab02dfa833aca4810b0fae8c4ab48f9 to
e156f3fb86236bfc0262e6287bd73784
List of dependencies for variable baselib changed from
set(['BASELIB']) to set(['DEFAULTTUNE'])
Dependency on variable BASELIB was added
Dependency on Variable DEFAULTTUNE was removed
Variable baselib value changed from ${BASELIB} to
${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) or
'INVALID'), True) or 'lib'}
Hash for dependent task
virtual:native:/local/home/mattsm/git/poky/meta/recipes-devtools/autoconf/autoconf_2.68.bb.do_install
changed from e74616029727b6e61b00273425128f1d to
478e7900a2310f2669708e6909883865
Back tracking this appears to be a multilib issue since I have that
enabled on one of the -native packages:
../meta/conf/bitbake.conf:baselib = ${BASELIB}
../meta/conf/multilib.conf:baselib = ${@d.getVar('BASE_LIB_tune-' +
(d.getVar('DEFAULTTUNE', True) or 'INVALID'), True) or 'lib'}
This is bad and shouldn't be happening, particularly for the native
packages. I did accidentally commit part of a work in progress in this
commit:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=ea0b7440ec5a8663b66f56a19f0b38e803b523d3
I didn't revert it since I thought it was a good idea and fixed some
problems I was seeing.
I suspect a baselib = lib in native.bbclass might be a good idea
too...
Cheers,
Richard
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
[OE-core] sstate_clean() overzealous?
The sstate_clean() function in sstate.bbclass is doing (inter alia):
stfile = d.getVar(STAMP, True) + .do_ + ss['task']
[...]
oe.path.remove(stfile + .*)
oe.path.remove(stfile + _setscene + .*)
which means that, for tasks which set the stamp-extra-info flag to
${MACHINE}, it ends up blowing away the stamps for all machines rather
than just the current one. The net effect of this seems to be that
there is no way to have the setscene stamps populated for more than one
MACHINE at any time, and hence all those tasks get rerun every time you
change MACHINE even if nothing else has been altered.
Is this behaviour deliberate? It is certainly a little bit annoying but
I don't understand the internals of sstate well enough to judge what
exactly that glob is trying to accomplish.
thanks
p.
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] sstate_clean() overzealous?
On 10/3/2011 2:11 PM, McClintock Matthew-B29882 wrote:
On Mon, Oct 3, 2011 at 11:47 AM, Phil Blundellph...@gnu.org wrote:
The sstate_clean() function in sstate.bbclass is doing (inter alia):
stfile = d.getVar(STAMP, True) + .do_ + ss['task']
[...]
oe.path.remove(stfile + .*)
oe.path.remove(stfile + _setscene + .*)
which means that, for tasks which set the stamp-extra-info flag to
${MACHINE}, it ends up blowing away the stamps for all machines rather
than just the current one. The net effect of this seems to be that
there is no way to have the setscene stamps populated for more than one
MACHINE at any time, and hence all those tasks get rerun every time you
change MACHINE even if nothing else has been altered.
Is this behaviour deliberate? It is certainly a little bit annoying but
I don't understand the internals of sstate well enough to judge what
exactly that glob is trying to accomplish.
I've noticed that changing the MACHINE will also invalidate the
-native stuff as well.
hmmm so we can not share sstate parts for two machines in same tmpdir ?
-M
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] sstate_clean() overzealous?
On 10/03/2011 02:11 PM, McClintock Matthew-B29882 wrote:
On Mon, Oct 3, 2011 at 11:47 AM, Phil Blundellph...@gnu.org wrote:
The sstate_clean() function in sstate.bbclass is doing (inter alia):
stfile = d.getVar(STAMP, True) + .do_ + ss['task']
[...]
oe.path.remove(stfile + .*)
oe.path.remove(stfile + _setscene + .*)
which means that, for tasks which set the stamp-extra-info flag to
${MACHINE}, it ends up blowing away the stamps for all machines rather
than just the current one. The net effect of this seems to be that
there is no way to have the setscene stamps populated for more than one
MACHINE at any time, and hence all those tasks get rerun every time you
change MACHINE even if nothing else has been altered.
Is this behaviour deliberate? It is certainly a little bit annoying but
I don't understand the internals of sstate well enough to judge what
exactly that glob is trying to accomplish.
I've noticed that changing the MACHINE will also invalidate the
-native stuff as well.
I am not sure that's correct, since we use sstate with the Autobuilder
and it's shared across different hosts building different MACHINEs, we
don't see the -natives getting build, do you have any examples?
Sau!
-M
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] sstate_clean() overzealous?
On Mon, Oct 3, 2011 at 4:24 PM, Saul Wold saul.w...@intel.com wrote: I am not sure that's correct, since we use sstate with the Autobuilder and it's shared across different hosts building different MACHINEs, we don't see the -natives getting build, do you have any examples? I'm building with targets that are not upstream, so I can't comment on the upstream variant. There are different TUNES though. I'm actually currently in the process of looking at why sstate-cache is not being reused. Can you point at the autobuilder stuff? Is it going from fsl-mpc8315e-rdb to and x86_64 target? Or between two x86_64 targets? -M ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] sstate_clean() overzealous?
On Mon, Oct 3, 2011 at 4:18 PM, Khem Raj raj.k...@gmail.com wrote: hmmm so we can not share sstate parts for two machines in same tmpdir ? I'm not 100% sure I'm doing things properly but this is what I did: $ bitbake pseudo-native $ cp sstate-cache/* ~/sstate-cache $ rm -rf tmp/ $ bitbake pseudo-native Now sstate-cache is verified to be working then I changed machines and ran $ cp ~/sstate-cache/* sstate-cache/ $ bitbake pseudo-native bitbake starts rebuilding everything, and added -DDD I see SState cache is missing. -M ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] sstate_clean() overzealous?
On Mon, Oct 3, 2011 at 4:29 PM, Matthew McClintock m...@freescale.com wrote:
On Mon, Oct 3, 2011 at 4:24 PM, Saul Wold saul.w...@intel.com wrote:
I am not sure that's correct, since we use sstate with the Autobuilder and
it's shared across different hosts building different MACHINEs, we don't see
the -natives getting build, do you have any examples?
I'm building with targets that are not upstream, so I can't comment on
the upstream variant. There are different TUNES though. I'm actually
currently in the process of looking at why sstate-cache is not being
reused.
Can you point at the autobuilder stuff? Is it going from
fsl-mpc8315e-rdb to and x86_64 target? Or between two x86_64 targets?
For example autoconf-native:
$ bitbake-diffsigs
../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo
[mattsm@right build_p5020ds_release (master *$)]$ bitbake-diffsigs
../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo
[snip]
basehash changed from bab02dfa833aca4810b0fae8c4ab48f9 to
e156f3fb86236bfc0262e6287bd73784
List of dependencies for variable baselib changed from
set(['BASELIB']) to set(['DEFAULTTUNE'])
Dependency on variable BASELIB was added
Dependency on Variable DEFAULTTUNE was removed
Variable baselib value changed from ${BASELIB} to
${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) or
'INVALID'), True) or 'lib'}
Hash for dependent task
virtual:native:/local/home/mattsm/git/poky/meta/recipes-devtools/autoconf/autoconf_2.68.bb.do_install
changed from e74616029727b6e61b00273425128f1d to
478e7900a2310f2669708e6909883865
Back tracking this appears to be a multilib issue since I have that
enabled on one of the -native packages:
../meta/conf/bitbake.conf:baselib = ${BASELIB}
../meta/conf/multilib.conf:baselib = ${@d.getVar('BASE_LIB_tune-' +
(d.getVar('DEFAULTTUNE', True) or 'INVALID'), True) or 'lib'}
Which causes the other differences
-M
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
