Paul's comments in #oe have highlighted the fact that whilst I and perhaps some others know where things are going (and have come from), a lot of things are in my head which isn't perhaps the best place for them.
History/background ================== Once upon a time, we didn't do anything with the runtime variables RDEPENDS, RRECOMMENDS etc. This lead to a lot of duplication within the metadata because of the two separate namespaces - you'd have have to add BOOTSTRAP_EXTRA_DEPENDS = "virtual/kernel" and BOOTSTRAP_EXTRA_RDEPENDS = "kernel-module-uinput". I complained about this and was told by Rene to shutup, or fix it if it bothered me so much. So I did :) (well, have attempted to anyway) Circa bitbake 1.3, I added handling for the RDEPENDS variables so bitbake could convert RDEPENDS into a set of DEPENDS. This was quite a painful process, I changed a lot of the way .dev worked and I resolved 95% of the problems encountered. I'm 100% certain this process was essential to allow OE to evolve. The end result was the modern neat looking task packages and the removal of all the add to RDEPENDS and make sure something equivalent is added to DEPENDS logic in .dev. I don't think anyone can look at what was there before, look at what there is now and not say it was an improvement. Things like task-base were simply not realistically possible under the old system. One of the big issues that was highlighted was should you RDEPEND on fontconfig or libfontconfig? That would depend on whether you inherit debian.bbclass or not. We don't require people to use debian.bbclass but most just happen to. There are also other classes that do subtle renaming (such as the kernel modules). Solution: We now always use unrenamed package names in the runtime variables and debian.bbclass handles renaming just as the packages are created. The problem is this isn't 100% foolproof. Consider: RDEPENDS_A = "B" RDEPENDS_B = "A" Which do we build first? We can't package A without packaging B first and vice versa. In bitbake 1.3 there were certain types of circular dependencies we errored on and shouldn't have done. In fixing those, I managed to let errors like the above slip through the net. This problem still exists in bitbake 1.6 as things like: DEPENDS_A = "B" RDEPENDS_B = "A" doesn't error and should (assuming you use debian.bbclass). Thankfully, these kind of dependencies very rarely happen. As the RDEPENDS handling was added, the code in bitbake went out of control and became unmaintainable. Arguably I understand it better than anyone and I couldn't fix the above problem in the existing codebase. Another related issue is that in bitbake 1.6, bitbake only handles things as packages with no real understanding of the tasks that make up the package which is really a dependency problem. I had problems trying to express the task dependencies required by debian.bbclass. I also had to make a choice about whether a multithreaded bitbake would operate at the package or the task level. I chose the task level as it offers more interesting possibilities in the future and addresses a lot more issues. This did mean the core of bitbake would need rewriting as it was previously package based. At the time, nothing in the metadata properly explained the dependencies at a task level. See these mails: http://www.handhelds.org/hypermail/oe/102/10285.html http://www.handhelds.org/hypermail/oe/102/10299.html As you can see, I threw out the old dependency notation (bbdepcmd) and replaced it with the items detailed above. I also carefully considered what our expectations of things like "-c fetch" are. For a long time, I've been getting confused as users want "-c fetch" to fetch all dependencies for something like opie-image, yet nothing in the metadata gives that dependency. After much thought, I concluded that "-c fetch" should only run the fetch task for the packages specified. To make "-c fetch" fetch all its dependencies, you could add something like this to the metadata for bitbake trunk: do_fetch[recrdeptask] = "do_fetch" I plan to either add an option "-c fetch -xyz" which simply injects the above metadata fragment, or we add a fetchall task in OE which does this and hence give users what they want/need in a maintainable way. Note, we also lose the horrible BUILD_ALL_DEPS variable I added as a mistake when working on bitbake 1.3 - whilst I hate the thing now, it did buy us time to step back and fix all this properly though! The final problem is that bitbake prior to 1.7 computes provider decisions "on the fly". When it starts building, it has no idea what it will end up doing. If/when a package fails, it would rethink what it was doing and only then try and compute a way out. It would only find the next alternative. This is the final barrier to multithreading - no predetermined paths but all computed dynamically. Again, this cries out for consideration as part of the rewrite. I'm acutally sad to lose the dynamic approach but it has too many drawbacks (see usability below). What did we do? =============== Hopefully the above tells you why we needed to throw away some of the old code. My plan was simple - implement a multithreading capable bitbake and at the same time, rewrite the dependency code with sanity in mind. I threw away the old code and added a new two level system: taskdata.py - handles the metadata, works out the dependencies and parses into a list of tasks runqueue.py - takes the list of tasks, filters them, decides on the optimal execution order and then runs them. Some of the handling is quite subtle. taskdata's data structures are reusable, runqueue's are not. When a package fails, we throw away the runqueue and ask taskdata for a new list of tasks (if possible). It is now hopefully structured code we can maintain with a semblance of an API (not stable yet though!). I'll also note that bitbake is actually *much* faster now. Compare the speed of a bitbake opie-image --dry-run under bitbake 1.6 with the new one. This is despite doing all the computation in advance! :) Where are we now? ================= The position in trunk is that these two modules basically work. They are not hooked into the rest of bitbake in a way which exposes all their power. New commandline options might help with the and/or improvements to the metadata as mentioned above. Of course there are also the inevitable corner cases in the metadata the new code highlights. One is the circular dependency example mentioned above which affects opie-mediaplayer1. I'm certain its not a bitbake issue but debian.bbclass being flawed. The solution is to split packaging into two stages: 1. packaging-prepare - split the files into the packages and calculate the renaming 2. packaging - actually make the packages and write in the RDEPENDS lines. This means we can write: do_package[rdeptask] = "do_package_prepare" (instead of do_package[rdeptask] = "do_package") and the whole circular dependency problem goes away. I've already covered the "-c fetch" problem and the two potential solutions. Another current issue is handling of the "-f" option. taskdata and runqueue need to be taught about that and currently it sometimes works but is unpredictable. The final issue I'm aware of is a usability one. I'm convinced taskdata/runqueue work but their output to the user is even less helpful than bitbake was previously (which was bad enough). I have see bitbake stop dead seemingly with no output at the moment which is no good. I have split things into logging domains as a first step to rectify this but it still needs a ton of work. The problem is knowing when a given message is important and the user needs to see it/do something about it and when a message should just appear as low level debug. I'm fixing cases of bad messages when I find them but I'm really going to need help with this in both identifying the cases and improving the existing messages. The point to keep in mind is that it was near impossible to improve the logging in the old dependency code due to its on the fly computation nature. Under the new code, we actually have the dependency data in advance and I'm therefore convinced that we stand a much better chance of improving usability against this new backdrop. I suspect I've rambled enough now :) If anyone wants to discuss any of this or better still help with the code, please do so! Richard _______________________________________________ Bitbake-dev mailing list [email protected] https://lists.berlios.de/mailman/listinfo/bitbake-dev
