Moin,
Am Sun, 01 Jul 2007 01:03:25 +0100 schrieb Richard Purdie:
> Due to the fact the changes impact a fairly wide spread of areas I
> think we need to work on committing this in stages.
Agreed.
> As an example, I've just looked at your changes to the svn fetcher
> and noticed the problems you had with the existing code making it
> difficult to add an "svn info" command. I've just committed a patch
> to bitbake trunk to do that without the duplication of code you have.
Ok, great.
> For reference, I'm thinking we should get this working in
> bitbake-trunk and then we can backport the changes needed to bitbake
> 1.8 in one lump.
Ok.
> What does refname default to? It always has to be set?
No default, yet. I figured since this would only be needed by very few
packages it would be ok to force all of them to set the parameter. But
see below.
> > In order to not have to go to the network for each
> > revision_identifier lookup I initially just cached it in the
> > bb.data object that is always passed around. Didn't work. So I then
> > tried to cache it in the bb.fetch module (e.g. like the already
> > existing urldata). That did work, somewhat, but left me wondering
> > for hours why it sometimes just did not work. With no indication to
> > the problem whatsoever. I then found out that you sneaked an os.fork
> > () into bb.runqueue which of course makes all caching in memory
> > impossible.
>
> "sneaked" isn't quite the right word, bitbake 1.8 was widely
> advertised as multithreaded so it had to fork somewhere! It forks
> even in the single thread case so people can't make bad
> assumptions ;-).
Well, when I hear "multithreaded" I usually assume threads to be
used and not processes :-)
Though I know that there are some issues with Python threading (namely
the Global Interpreter Lock), this should not pose a problem since
bitbake doesn't do extensive calculations in Python code (in the childs
at least).
> I'd be interested to know which cases this didn't work in. At parse
> time bitbake is single threaded (and always will be effectively) so
> if you calculate SRCREV at parse time, you should be able to save it.
Hmm, I can't pinpoint the exact necessary circumstances under which my
code is called too late, but I think it just always happens when the
normal cache is intact (e.g. not every .bb file is parsed on startup).
I then clearly see lots of different results for os.getpid(), e.g. am
always in a new forked child.
It's true that this might also be solved by cleaner integration into
the dependency calculation stage.
> It does throw the data away after parsing so we'd perhaps need some
> kind of shared memory store to put it into but that should be
> entirely possible.
>
> > So I need to do my revision caching on disk, even though I actually
> > only want to cache for the duration of a bitbake run. I therefore
> > instituted something I call "runcache" (actually "per-run cache",
> > but that's just too long) that is just a pickle'd cache file, which
> > is supposed to be cleared at the beginning of each run. With proper
> > locking of course.
>
> Ok, I think some kind of on disk caching will be needed since
> different users are going to have different needs and hence different
> "cache policies". Personally, I'd want to be able to manually reset
> the cache when I wanted updates rather than having the system always
> go for them. I think it would be better to have things in memory once
> bitbake is running in a given session though so perhaps we can create
> some kind of hybrid here with policy controlled by a variable?
Oh, that's a great idea and I entirely support it. There need to be
some more changes though: All fetchers must create some consistent
image of the current remote revision and then be modified to actually
check out and use that revision. Then it should be no problem to write
that current state to disk and reuse it at a later time.
> That workaround is truly hideous ;-). Perhaps we should add an
> optional option to bb.fetch.init() which allows it to only run
> against remote sources (marking the fetchers as local/remote). If
> might be worth just adding a special version of the function...
Two things:
One: I just thought about a better way to reference the revision
information which virtually eliminates the problem (and might have some
nice properties): Use Python's __getattr__. I would define a magical
object (maybe "bb.fetch.rev" or something like that) with a __getattr__
() method. Then you would use something like [EMAIL PROTECTED]
instead of ${SRCREV_svn1}.
This way, code is being run when the revision information is requested
and the code can do arbitrary things. For starters it could just return
a fixed string when the information is not yet available ("" or "now").
It could also assign some default refname for URLs that don't have a
refname set.
The other thing: Yes, bb.fetch IMHO needs some attention. I entirely
dislike the approach to keep data in the module global namespace. I
also don't like the way fetcher objects are created/used. IMHO a clean
object-oriented design would be:
+ Create one fetcher object of the correct class for each URL that is
to be fetched. (Or maybe one object per scheme per FILE.) These can
then be constructed and used in a local namespace without disturbing
anything anywhere.
+ (Maybe: do away with the methods list, that can be constructed by
introspection.)
+ If need be (looks like it does), create one object to manage the
fetch process, e.g. one manager object that you associate several
fetcher objects with. Then calling the fetch method on the manager
object will call the fetch methods on the fetcher objects in order.
Basically what the module does now, but with the advantage that you
can create manager objects in a local namespace that are independent
from the module.
--
Henryk Plötz
Grüße aus Berlin
~ Help Microsoft fight software piracy: Give Linux to a friend today! ~
_______________________________________________
Bitbake-dev mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/bitbake-dev
