Re: [U-Boot] Merging device trees at runtime for module-based systems
On Wed, Oct 31, 2012 at 11:00 PM, Daniel Mack zon...@gmail.com wrote: cc devicetree-discuss. Here's a reference to the full thread: http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/ Interesting. I only just was made aware of this thread. There is a similar discussion going on kicked off by the BeagleBone folks where they want to insert additional DT data from Linux userspace. Whether the data is merged at U-Boot time or kernel time, I expect that the required data format will be very similar. https://lkml.org/lkml/2012/10/31/502 https://lkml.org/lkml/2012/11/5/615 I'm trying to draft up a document that captures the requirements and lay out what needs to be done to the tools, U-Boot and the kernel. g. ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
On Thu, Nov 01, 2012 at 10:24:06AM +0100, Daniel Mack wrote: On 01.11.2012 04:26, David Gibson wrote: On Fri, Oct 26, 2012 at 09:24:11AM +0200, Daniel Mack wrote: I would especially like to know where such a new functionality should live, which data types it should operate on and what would be an appropriate name for it. So.. the first thought I have reading the original mail in the thread is that it's arguable that you really want a more heavyweight firmware for this setup, that actively maintains a live device tree as OF does, rather than u-boot which is pretty oriented towards a close-to-static device setup. That's just a thought though, I'm not saying that at least some of this functionality doesn't belong in libfdt. So, my thought would be that stuff for manipulating big chunks of tree should go in a new .c file inside the libfdt tree. We already have del_node and nop_node of course, which can remove whole subtrees. I guess the big extra function you'd want would be something like: fdt_graft(void *fdt, int offset, void *subtree); Which would graft the tree blob give by subtree into the master tree (fdt) at node 'offset'. Actually that might need to take a name for the top-level of the subtree to take in the new tree too. I called the function fdt_overlay, but I guess the implementation is similar to what you thought of. I pushed it here, see the topmost 3 commits: https://github.com/zonque/dtc/commits/overlay Interesting. So, it seems to me that fdt_graft() and fdt_overlay() are different operations - both could be potentially useful. fdt_graft() would attach a new subtree somewhere within the master tree, with the assumption that the root of the subtree would become a new node in the resulting tree. Overwriting an existing subtree with a new one would be an error for a graft. fdt_overlay, as you've implemented, can either add new nodes or modify existing ones by replacing or adding new properties. So, some notes on the actual implementation: The in-place modification of the given path (which should really be const char *) in your fdt_add_subnode_r() is nasty, nasty, nasty. And it's unnecessary because you can use the existing fdt_add_subnode_namelen() to work with subsections of the path without needing to either have a temporary buffer or do in-place modification. ...except, I don't think you actually need fdt_add_subnoode_r() for your overlay implementation in any case. AFAICT in your fdt_overlay() implementation you're only adding nodes from the second tree if they contain properties (the fdt_add_subnode_r() call is under the FDT_PROP case). I'm not sure if that was a deliberate policy decision - if so I really can't see a reason for it. If instead you *always* add subnodes when they exist in the second tree, you'll be doing your add nodes from the FDT_BEGIN_NODE tag case. And you always get BEGIN_NODE tags for parents before subnodes, so you can naturally add your new subnode path component by component without having to walk down the path again in fdt_add_subnode_r(). As an added bonus you no longer need pathbuf and it's arbitrary size limit. Hrm.. wait... I guess you need a stack so you can handle FDT_END_NODE correctly. I suspect a recursive solution (effectively using the machine stack) would still take less (machine) stack space than pathbuf. Especially if pathbuf was increased up to PATH_MAX, which is my usual rule of thumb when I can't avoid an arbitrary buffer size. On a tangent, note that fdt_graft() as defined above, unlike fdt_overlay() would allow a considerably optimized implementation. Instead of doing lots of individual inserts into the tree (and therefore a lot of memmove()s), you could do one big _fdt_splice(), copy in the grafted tree's structure block, then run through it correcting property name offsets. -- David Gibson| I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
On 03.11.2012 16:25, David Gibson wrote: On Thu, Nov 01, 2012 at 10:24:06AM +0100, Daniel Mack wrote: On 01.11.2012 04:26, David Gibson wrote: On Fri, Oct 26, 2012 at 09:24:11AM +0200, Daniel Mack wrote: I would especially like to know where such a new functionality should live, which data types it should operate on and what would be an appropriate name for it. So.. the first thought I have reading the original mail in the thread is that it's arguable that you really want a more heavyweight firmware for this setup, that actively maintains a live device tree as OF does, rather than u-boot which is pretty oriented towards a close-to-static device setup. That's just a thought though, I'm not saying that at least some of this functionality doesn't belong in libfdt. So, my thought would be that stuff for manipulating big chunks of tree should go in a new .c file inside the libfdt tree. We already have del_node and nop_node of course, which can remove whole subtrees. I guess the big extra function you'd want would be something like: fdt_graft(void *fdt, int offset, void *subtree); Which would graft the tree blob give by subtree into the master tree (fdt) at node 'offset'. Actually that might need to take a name for the top-level of the subtree to take in the new tree too. I called the function fdt_overlay, but I guess the implementation is similar to what you thought of. I pushed it here, see the topmost 3 commits: https://github.com/zonque/dtc/commits/overlay Interesting. So, it seems to me that fdt_graft() and fdt_overlay() are different operations - both could be potentially useful. fdt_graft() would attach a new subtree somewhere within the master tree, with the assumption that the root of the subtree would become a new node in the resulting tree. Overwriting an existing subtree with a new one would be an error for a graft. fdt_overlay, as you've implemented, can either add new nodes or modify existing ones by replacing or adding new properties. So, some notes on the actual implementation: The in-place modification of the given path (which should really be const char *) in your fdt_add_subnode_r() is nasty, nasty, nasty. And it's unnecessary because you can use the existing fdt_add_subnode_namelen() to work with subsections of the path without needing to either have a temporary buffer or do in-place modification. Ok - thanks for the hint. I was looking for a nicer way as I also was really unhappy about the implementation, but without access to heap operations and avoid eating up too much stack, I thought this is a nasty but working solution. After all, that whole hacking was just meant as a proof of concept for the result, which failed ;) ...except, I don't think you actually need fdt_add_subnoode_r() for your overlay implementation in any case. AFAICT in your fdt_overlay() implementation you're only adding nodes from the second tree if they contain properties (the fdt_add_subnode_r() call is under the FDT_PROP case). I'm not sure if that was a deliberate policy decision - if so I really can't see a reason for it. That was merely to spare some cycles, as fdt_add_subnode_r() turned out not to be free of costs. If instead you *always* add subnodes when they exist in the second tree, you'll be doing your add nodes from the FDT_BEGIN_NODE tag case. And you always get BEGIN_NODE tags for parents before subnodes, so you can naturally add your new subnode path component by component without having to walk down the path again in fdt_add_subnode_r(). As an added bonus you no longer need pathbuf and it's arbitrary size limit. Yes, I had something like that in mind, but that requires keeping the state of both trees around and care for the stacking etc. Hrm.. wait... I guess you need a stack so you can handle FDT_END_NODE correctly. I suspect a recursive solution (effectively using the machine stack) would still take less (machine) stack space than pathbuf. Especially if pathbuf was increased up to PATH_MAX, which is my usual rule of thumb when I can't avoid an arbitrary buffer size. On a tangent, note that fdt_graft() as defined above, unlike fdt_overlay() would allow a considerably optimized implementation. Instead of doing lots of individual inserts into the tree (and therefore a lot of memmove()s), you could do one big _fdt_splice(), copy in the grafted tree's structure block, then run through it correcting property name offsets. Given the fact that this implementation turns out to be inappropriate as solution to my actual problem, I'm uncertain whether it should be merged at all. Dunno of how much use that would be for others after all. Many thanks for you input, Daniel ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
On Wed, Oct 31, 2012 at 10:36:08PM -0600, Stephen Warren wrote: On 10/31/2012 05:56 PM, Mitch Bradley wrote: On 10/31/2012 1:00 PM, Daniel Mack wrote: cc devicetree-discuss. Here's a reference to the full thread: http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/ On 26.10.2012 20:39, Stephen Warren wrote: On 10/24/2012 03:47 AM, Daniel Mack wrote: Hi, a project I'm involved in uses a module/baseboard combo, and components on either board are described in DT. I'm currently using separate dts files which build upon each other with include statements, which works fine for development. In production though, we will certainly have running changes (and hence different versions) over the lifetime of the product for both the baseboard and the module, and the hardware has support for identifying the versions of both sides at runtime. ... I start to believe that the cleanest solution to this would be to have full DTC functionality in U-Boot and compile the tree ... which is exactly the way that Open Firmware does it, since the invention of the device tree. The model is that the boot firmware, which needs to know the system configuration to do its job anyway, exports that configuration via the device tree. Doesn't OF generate the DT from internal data structures (although I don't know where those come from...), Well.. in OF the device tree *is* a core live data structure. It will be constructed as devices are probed, firmware level device drivers are attached to it and it might be modified by client interface operations. Traditionally there is no tree in the flattened format, only the live tree which clients will query with OF calls. Some recent OF implementations do use the flat tree in some ways - we've had some systems where a flattened tree is provided by an early boot to the full OF; it acts as a skeleton that is then extended in OF's live tree. We've also had some OF implementations that for compatiblity with kernels that expect a flattened tree flatten their internal tree structure into the FDT format as the last thing before entering the OS. whereas what Daniel mentions above is more like the bootloader having access to a bunch of .dts fragments, selecting the appropriate subset of those to use, parsing them into an internal data structure (i.e. running dtc), and then generating a DTB from it. The overall result is that the bootloader causes a DTB to be generated at run-time, so at that level it's the same, but the implementation seems pretty different. How OF constructs its internal tree can vary a lot with the OF implementation, and I'm not terribly clear on what you had in mind, so I don't think either side is really sufficiently well defined to really say if they're similar or not. -- David Gibson| I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
On 10/31/2012 1:00 PM, Daniel Mack wrote:
cc devicetree-discuss. Here's a reference to the full thread:
http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/
On 26.10.2012 20:39, Stephen Warren wrote:
On 10/24/2012 03:47 AM, Daniel Mack wrote:
Hi,
a project I'm involved in uses a module/baseboard combo, and components
on either board are described in DT. I'm currently using separate dts
files which build upon each other with include statements, which works
fine for development.
In production though, we will certainly have running changes (and hence
different versions) over the lifetime of the product for both the
baseboard and the module, and the hardware has support for identifying
the versions of both sides at runtime.
So let's say we have n versions of the baseboard and m versions of the
module, we would much like to only prepare n + m files, instead of n * m
by pre-compiling every possible combination (some of which may actually
never occur 'in the wild').
So my question is: is it possible to do that kind of assembly of a
number of files at runtime in U-Boot? I guess all it takes is merging a
number of trees together, right? I browsed through the APIs but couldn't
yet find an clear approach to that kind of problem. If not, what would
it take to add that functionality? I can probably help with the
implementation if someone tells me what would be the right way.
Yes, solving this would be very useful; it's a wide-spread problem.
Some thoughts though:
Simply overlaying two DTBs on top of each-other (in the same fashion
that dtc's /include/ statement would do at compile-time) might not be
fully general enough, although perhaps it would be sufficient for your
immediate needs.
For example, lets say that a GPIO is routed from a device on the main
board to a device on a daughter board, or even from one daughter board
into the main board and back out to a different daughter board. Now,
consider that the different board(s) that are the source of the GPIO
might use completely different SoCs or versions of the SoC, which might
require using a different GPIO specifier to represent the signal. That
means you need to change the .dtb file for the client of the GPIO
depending on the HW or .dtb that provides the GPIO. That's certainly not
a simple matter of merging multiple .dtb blobs together.
Hmm. After implementing a very simple overlay approach, I can now see
your point :) Yes in fact, that's a real problem.
The same issue could easily apply to I2C or SPI buses, chip selects, etc.
One solution would be to explicitly represent a connector or
connection-point in DT, such that the connector can implement the naming
of all signals that pass through it, and provide a translation point for
hooking the two DT fragments together. This seems within the spirit of DT.
Yes, but you still can't handle references that way.
Let me try and conclude this for others. Say the module tree A looks
something like this:
/ {
multi-regulator {
vcc1v8: regulator@0 {
/* ... */
};
};
};
... and the baseboard (B), that makes use of (and hence depends on)
the module, has something like this:
/ {
consumer {
main-supply = vcc1v8;
};
};
Now, let's say in a subsequent version of the module, we change whatever
provides that supply for 1.8 volts, but the consumer on the baseboard
shouldn't care much of course, thanks to all the abstraction layers that
we have now in the kernel.
However, the problem here is that I can't just compile trees A and B
individually into .dtbs that get merged later, because dtc will bail on
the unresolved reference of vcc1v8 of course. And cases like this are
the whole reason why I started to think about modularization of trees in
the first place.
So the simple overlay method doesn't help here at all, even though I can
share the code if anyone's interested.
Another solution might be some form of variables/macros/code in the DTB
that can be used to parameterize other DTBs that get merged with it.
This is probably an enormous can of worms.
Yes, exactly, a can of worms and most probably unmaintainble in real
life. I start to believe that the cleanest solution to this would be to
have full DTC functionality in U-Boot and compile the tree
... which is exactly the way that Open Firmware does it, since the
invention of the device tree. The model is that the boot firmware,
which needs to know the system configuration to do its job anyway,
exports that configuration via the device tree.
from dts, but
then again I have no clue on how to handle the file lookups that arise
from includes. Do you think it would it be worth going that way?
If not, I guess we're down to n*m files eventually, which is really sad
as they might even
Re: [U-Boot] Merging device trees at runtime for module-based systems
On Fri, Oct 26, 2012 at 09:24:11AM +0200, Daniel Mack wrote: On 26.10.2012 02:53, David Gibson wrote: On Thu, Oct 25, 2012 at 10:46:32PM +0200, Wolfgang Denk wrote: Dear Daniel, In message 50893633.6070...@gmail.com you wrote: Overwrites must be addressed in the first place. The most common example is that a more generic part (the module tree) registers all details about a peripheral up-front but then sets its status to 'disabled'. That way, the more specific part (the base board tree) can overwrite this property to 'okay' at wish to enable it and not care for the pre-defined details. This is also how we do things in our device-trees. Agreed. I definitely can see the benefit of such a feature and would be happy if you could go forward and implement it. Ok then. I guess this should be something that can eventually be merged back into libfdt? I can't speak for the FDT custodian, but I think this makes a lot of sense. As a rule I'm happy to see more functionality for libfdt. I've only seen bits and pieces of this thread, though, so I'd need to see a summary of what exactly is being proposed. That's strange, as I copied you from the very first posting. Anyway, here's the archive: Oh I probably got them somewhere in my mail. But it's only recently that I really noticed - I get a fair bit of mail. http://lists.denx.de/pipermail/u-boot/2012-October/138227.html I would especially like to know where such a new functionality should live, which data types it should operate on and what would be an appropriate name for it. So.. the first thought I have reading the original mail in the thread is that it's arguable that you really want a more heavyweight firmware for this setup, that actively maintains a live device tree as OF does, rather than u-boot which is pretty oriented towards a close-to-static device setup. That's just a thought though, I'm not saying that at least some of this functionality doesn't belong in libfdt. So, my thought would be that stuff for manipulating big chunks of tree should go in a new .c file inside the libfdt tree. We already have del_node and nop_node of course, which can remove whole subtrees. I guess the big extra function you'd want would be something like: fdt_graft(void *fdt, int offset, void *subtree); Which would graft the tree blob give by subtree into the master tree (fdt) at node 'offset'. Actually that might need to take a name for the top-level of the subtree to take in the new tree too. Things get trickier when you consider what might need to be tweaked in the subtree to make it fit into the master tree. If it requires widespread alterations through the subtree that's going to get really ugly and I think you would be better off with a firmware with a fuller handling of a live device tree. But I think that can probably be avoided with proper design of the bindings. To get that to work you'll need to make sure you use some sort of local addressing within the subtree. Then it should only be necessary to insert/alter a ranges property at the top level of the subtree (or possibly its parent) to map that correctly into the global address space. Likewise interrupts within the subtree probably shouldn't address an external interrupt controller but rather the root of the tree. You can then insert an interrupt-map property which will wire those into the global interrupt tree. -- David Gibson| I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
On 10/31/2012 6:36 PM, Stephen Warren wrote: On 10/31/2012 05:56 PM, Mitch Bradley wrote: On 10/31/2012 1:00 PM, Daniel Mack wrote: cc devicetree-discuss. Here's a reference to the full thread: http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/ On 26.10.2012 20:39, Stephen Warren wrote: On 10/24/2012 03:47 AM, Daniel Mack wrote: Hi, a project I'm involved in uses a module/baseboard combo, and components on either board are described in DT. I'm currently using separate dts files which build upon each other with include statements, which works fine for development. In production though, we will certainly have running changes (and hence different versions) over the lifetime of the product for both the baseboard and the module, and the hardware has support for identifying the versions of both sides at runtime. ... I start to believe that the cleanest solution to this would be to have full DTC functionality in U-Boot and compile the tree ... which is exactly the way that Open Firmware does it, since the invention of the device tree. The model is that the boot firmware, which needs to know the system configuration to do its job anyway, exports that configuration via the device tree. Doesn't OF generate the DT from internal data structures (although I don't know where those come from...), whereas what Daniel mentions above is more like the bootloader having access to a bunch of .dts fragments, selecting the appropriate subset of those to use, parsing them into an internal data structure (i.e. running dtc), and then generating a DTB from it. The overall result is that the bootloader causes a DTB to be generated at run-time, so at that level it's the same, but the implementation seems pretty different. Yes, which is why I cut the cited sentence at the place I did (before the part about dts fragments). ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
On 01.11.2012 04:26, David Gibson wrote: On Fri, Oct 26, 2012 at 09:24:11AM +0200, Daniel Mack wrote: I would especially like to know where such a new functionality should live, which data types it should operate on and what would be an appropriate name for it. So.. the first thought I have reading the original mail in the thread is that it's arguable that you really want a more heavyweight firmware for this setup, that actively maintains a live device tree as OF does, rather than u-boot which is pretty oriented towards a close-to-static device setup. That's just a thought though, I'm not saying that at least some of this functionality doesn't belong in libfdt. So, my thought would be that stuff for manipulating big chunks of tree should go in a new .c file inside the libfdt tree. We already have del_node and nop_node of course, which can remove whole subtrees. I guess the big extra function you'd want would be something like: fdt_graft(void *fdt, int offset, void *subtree); Which would graft the tree blob give by subtree into the master tree (fdt) at node 'offset'. Actually that might need to take a name for the top-level of the subtree to take in the new tree too. I called the function fdt_overlay, but I guess the implementation is similar to what you thought of. I pushed it here, see the topmost 3 commits: https://github.com/zonque/dtc/commits/overlay Things get trickier when you consider what might need to be tweaked in the subtree to make it fit into the master tree. If it requires widespread alterations through the subtree that's going to get really ugly and I think you would be better off with a firmware with a fuller handling of a live device tree. But I think that can probably be avoided with proper design of the bindings. To get that to work you'll need to make sure you use some sort of local addressing within the subtree. Then it should only be necessary to insert/alter a ranges property at the top level of the subtree (or possibly its parent) to map that correctly into the global address space. Likewise interrupts within the subtree probably shouldn't address an external interrupt controller but rather the root of the tree. You can then insert an interrupt-map property which will wire those into the global interrupt tree. As pointed out on another end of this thread, the use of my simple implementation is rather limited. I need to think about something more sophisticated, or abadon the idea alltogether. Thanks, Daniel ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
cc devicetree-discuss. Here's a reference to the full thread:
http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/
On 26.10.2012 20:39, Stephen Warren wrote:
On 10/24/2012 03:47 AM, Daniel Mack wrote:
Hi,
a project I'm involved in uses a module/baseboard combo, and components
on either board are described in DT. I'm currently using separate dts
files which build upon each other with include statements, which works
fine for development.
In production though, we will certainly have running changes (and hence
different versions) over the lifetime of the product for both the
baseboard and the module, and the hardware has support for identifying
the versions of both sides at runtime.
So let's say we have n versions of the baseboard and m versions of the
module, we would much like to only prepare n + m files, instead of n * m
by pre-compiling every possible combination (some of which may actually
never occur 'in the wild').
So my question is: is it possible to do that kind of assembly of a
number of files at runtime in U-Boot? I guess all it takes is merging a
number of trees together, right? I browsed through the APIs but couldn't
yet find an clear approach to that kind of problem. If not, what would
it take to add that functionality? I can probably help with the
implementation if someone tells me what would be the right way.
Yes, solving this would be very useful; it's a wide-spread problem.
Some thoughts though:
Simply overlaying two DTBs on top of each-other (in the same fashion
that dtc's /include/ statement would do at compile-time) might not be
fully general enough, although perhaps it would be sufficient for your
immediate needs.
For example, lets say that a GPIO is routed from a device on the main
board to a device on a daughter board, or even from one daughter board
into the main board and back out to a different daughter board. Now,
consider that the different board(s) that are the source of the GPIO
might use completely different SoCs or versions of the SoC, which might
require using a different GPIO specifier to represent the signal. That
means you need to change the .dtb file for the client of the GPIO
depending on the HW or .dtb that provides the GPIO. That's certainly not
a simple matter of merging multiple .dtb blobs together.
Hmm. After implementing a very simple overlay approach, I can now see
your point :) Yes in fact, that's a real problem.
The same issue could easily apply to I2C or SPI buses, chip selects, etc.
One solution would be to explicitly represent a connector or
connection-point in DT, such that the connector can implement the naming
of all signals that pass through it, and provide a translation point for
hooking the two DT fragments together. This seems within the spirit of DT.
Yes, but you still can't handle references that way.
Let me try and conclude this for others. Say the module tree A looks
something like this:
/ {
multi-regulator {
vcc1v8: regulator@0 {
/* ... */
};
};
};
... and the baseboard (B), that makes use of (and hence depends on)
the module, has something like this:
/ {
consumer {
main-supply = vcc1v8;
};
};
Now, let's say in a subsequent version of the module, we change whatever
provides that supply for 1.8 volts, but the consumer on the baseboard
shouldn't care much of course, thanks to all the abstraction layers that
we have now in the kernel.
However, the problem here is that I can't just compile trees A and B
individually into .dtbs that get merged later, because dtc will bail on
the unresolved reference of vcc1v8 of course. And cases like this are
the whole reason why I started to think about modularization of trees in
the first place.
So the simple overlay method doesn't help here at all, even though I can
share the code if anyone's interested.
Another solution might be some form of variables/macros/code in the DTB
that can be used to parameterize other DTBs that get merged with it.
This is probably an enormous can of worms.
Yes, exactly, a can of worms and most probably unmaintainble in real
life. I start to believe that the cleanest solution to this would be to
have full DTC functionality in U-Boot and compile the tree from dts, but
then again I have no clue on how to handle the file lookups that arise
from includes. Do you think it would it be worth going that way?
If not, I guess we're down to n*m files eventually, which is really sad
as they might even become a storage problem at some point.
Thanks for your input,
Daniel
___
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
On 10/31/2012 05:00 PM, Daniel Mack wrote:
cc devicetree-discuss. Here's a reference to the full thread:
http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/
On 26.10.2012 20:39, Stephen Warren wrote:
On 10/24/2012 03:47 AM, Daniel Mack wrote:
Hi,
a project I'm involved in uses a module/baseboard combo, and components
on either board are described in DT. I'm currently using separate dts
files which build upon each other with include statements, which works
fine for development.
In production though, we will certainly have running changes (and hence
different versions) over the lifetime of the product for both the
baseboard and the module, and the hardware has support for identifying
the versions of both sides at runtime.
So let's say we have n versions of the baseboard and m versions of the
module, we would much like to only prepare n + m files, instead of n * m
by pre-compiling every possible combination (some of which may actually
never occur 'in the wild').
So my question is: is it possible to do that kind of assembly of a
number of files at runtime in U-Boot? I guess all it takes is merging a
number of trees together, right? I browsed through the APIs but couldn't
yet find an clear approach to that kind of problem. If not, what would
it take to add that functionality? I can probably help with the
implementation if someone tells me what would be the right way.
Yes, solving this would be very useful; it's a wide-spread problem.
Some thoughts though:
Simply overlaying two DTBs on top of each-other (in the same fashion
that dtc's /include/ statement would do at compile-time) might not be
fully general enough, although perhaps it would be sufficient for your
immediate needs.
For example, lets say that a GPIO is routed from a device on the main
board to a device on a daughter board, or even from one daughter board
into the main board and back out to a different daughter board. Now,
consider that the different board(s) that are the source of the GPIO
might use completely different SoCs or versions of the SoC, which might
require using a different GPIO specifier to represent the signal. That
means you need to change the .dtb file for the client of the GPIO
depending on the HW or .dtb that provides the GPIO. That's certainly not
a simple matter of merging multiple .dtb blobs together.
Hmm. After implementing a very simple overlay approach, I can now see
your point :) Yes in fact, that's a real problem.
The same issue could easily apply to I2C or SPI buses, chip selects, etc.
One solution would be to explicitly represent a connector or
connection-point in DT, such that the connector can implement the naming
of all signals that pass through it, and provide a translation point for
hooking the two DT fragments together. This seems within the spirit of DT.
Yes, but you still can't handle references that way.
Let me try and conclude this for others. Say the module tree A looks
something like this:
/ {
multi-regulator {
vcc1v8: regulator@0 {
/* ... */
};
};
};
... and the baseboard (B), that makes use of (and hence depends on)
the module, has something like this:
/ {
consumer {
main-supply = vcc1v8;
};
};
Now, let's say in a subsequent version of the module, we change whatever
provides that supply for 1.8 volts, but the consumer on the baseboard
shouldn't care much of course, thanks to all the abstraction layers that
we have now in the kernel.
However, the problem here is that I can't just compile trees A and B
individually into .dtbs that get merged later, because dtc will bail on
the unresolved reference of vcc1v8 of course. And cases like this are
the whole reason why I started to think about modularization of trees in
the first place.
So the simple overlay method doesn't help here at all, even though I can
share the code if anyone's interested.
Yes, you've understood me exactly.
The connector-base approach I was thinking about might look (very very)
roughly as follows:
main board:
/ {
multi-regulator {
vcc1v8: regulator@0 {
/* ... */
};
};
connector {
compatible = vendor,board-socket-a;
vcc1v8-supply = vcc1v8;
};
};
child board:
/ {
connector {
compatible = vendor,board-plug-a;
vcc1v8: regulator {
};
};
consumer {
main-supply = vcc1v8;
};
};
... plus some logic so that the drivers for the two connector nodes
get linked together, such that
Re: [U-Boot] Merging device trees at runtime for module-based systems
On 01.11.2012 00:13, Stephen Warren wrote:
On 10/31/2012 05:00 PM, Daniel Mack wrote:
cc devicetree-discuss. Here's a reference to the full thread:
http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/
On 26.10.2012 20:39, Stephen Warren wrote:
On 10/24/2012 03:47 AM, Daniel Mack wrote:
Hi,
a project I'm involved in uses a module/baseboard combo, and components
on either board are described in DT. I'm currently using separate dts
files which build upon each other with include statements, which works
fine for development.
In production though, we will certainly have running changes (and hence
different versions) over the lifetime of the product for both the
baseboard and the module, and the hardware has support for identifying
the versions of both sides at runtime.
So let's say we have n versions of the baseboard and m versions of the
module, we would much like to only prepare n + m files, instead of n * m
by pre-compiling every possible combination (some of which may actually
never occur 'in the wild').
So my question is: is it possible to do that kind of assembly of a
number of files at runtime in U-Boot? I guess all it takes is merging a
number of trees together, right? I browsed through the APIs but couldn't
yet find an clear approach to that kind of problem. If not, what would
it take to add that functionality? I can probably help with the
implementation if someone tells me what would be the right way.
Yes, solving this would be very useful; it's a wide-spread problem.
Some thoughts though:
Simply overlaying two DTBs on top of each-other (in the same fashion
that dtc's /include/ statement would do at compile-time) might not be
fully general enough, although perhaps it would be sufficient for your
immediate needs.
For example, lets say that a GPIO is routed from a device on the main
board to a device on a daughter board, or even from one daughter board
into the main board and back out to a different daughter board. Now,
consider that the different board(s) that are the source of the GPIO
might use completely different SoCs or versions of the SoC, which might
require using a different GPIO specifier to represent the signal. That
means you need to change the .dtb file for the client of the GPIO
depending on the HW or .dtb that provides the GPIO. That's certainly not
a simple matter of merging multiple .dtb blobs together.
Hmm. After implementing a very simple overlay approach, I can now see
your point :) Yes in fact, that's a real problem.
The same issue could easily apply to I2C or SPI buses, chip selects, etc.
One solution would be to explicitly represent a connector or
connection-point in DT, such that the connector can implement the naming
of all signals that pass through it, and provide a translation point for
hooking the two DT fragments together. This seems within the spirit of DT.
Yes, but you still can't handle references that way.
Let me try and conclude this for others. Say the module tree A looks
something like this:
/ {
multi-regulator {
vcc1v8: regulator@0 {
/* ... */
};
};
};
... and the baseboard (B), that makes use of (and hence depends on)
the module, has something like this:
/ {
consumer {
main-supply = vcc1v8;
};
};
Now, let's say in a subsequent version of the module, we change whatever
provides that supply for 1.8 volts, but the consumer on the baseboard
shouldn't care much of course, thanks to all the abstraction layers that
we have now in the kernel.
However, the problem here is that I can't just compile trees A and B
individually into .dtbs that get merged later, because dtc will bail on
the unresolved reference of vcc1v8 of course. And cases like this are
the whole reason why I started to think about modularization of trees in
the first place.
So the simple overlay method doesn't help here at all, even though I can
share the code if anyone's interested.
Yes, you've understood me exactly.
The connector-base approach I was thinking about might look (very very)
roughly as follows:
main board:
/ {
multi-regulator {
vcc1v8: regulator@0 {
/* ... */
};
};
connector {
compatible = vendor,board-socket-a;
vcc1v8-supply = vcc1v8;
};
};
child board:
/ {
connector {
compatible = vendor,board-plug-a;
vcc1v8: regulator {
};
};
consumer {
main-supply = vcc1v8;
};
};
... which doesn't eally make the individual bits more readable.
... plus some logic so
Re: [U-Boot] Merging device trees at runtime for module-based systems
On 10/31/2012 05:56 PM, Mitch Bradley wrote: On 10/31/2012 1:00 PM, Daniel Mack wrote: cc devicetree-discuss. Here's a reference to the full thread: http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/ On 26.10.2012 20:39, Stephen Warren wrote: On 10/24/2012 03:47 AM, Daniel Mack wrote: Hi, a project I'm involved in uses a module/baseboard combo, and components on either board are described in DT. I'm currently using separate dts files which build upon each other with include statements, which works fine for development. In production though, we will certainly have running changes (and hence different versions) over the lifetime of the product for both the baseboard and the module, and the hardware has support for identifying the versions of both sides at runtime. ... I start to believe that the cleanest solution to this would be to have full DTC functionality in U-Boot and compile the tree ... which is exactly the way that Open Firmware does it, since the invention of the device tree. The model is that the boot firmware, which needs to know the system configuration to do its job anyway, exports that configuration via the device tree. Doesn't OF generate the DT from internal data structures (although I don't know where those come from...), whereas what Daniel mentions above is more like the bootloader having access to a bunch of .dts fragments, selecting the appropriate subset of those to use, parsing them into an internal data structure (i.e. running dtc), and then generating a DTB from it. The overall result is that the bootloader causes a DTB to be generated at run-time, so at that level it's the same, but the implementation seems pretty different. ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
On 26.10.2012 02:53, David Gibson wrote: On Thu, Oct 25, 2012 at 10:46:32PM +0200, Wolfgang Denk wrote: Dear Daniel, In message 50893633.6070...@gmail.com you wrote: Overwrites must be addressed in the first place. The most common example is that a more generic part (the module tree) registers all details about a peripheral up-front but then sets its status to 'disabled'. That way, the more specific part (the base board tree) can overwrite this property to 'okay' at wish to enable it and not care for the pre-defined details. This is also how we do things in our device-trees. Agreed. I definitely can see the benefit of such a feature and would be happy if you could go forward and implement it. Ok then. I guess this should be something that can eventually be merged back into libfdt? I can't speak for the FDT custodian, but I think this makes a lot of sense. As a rule I'm happy to see more functionality for libfdt. I've only seen bits and pieces of this thread, though, so I'd need to see a summary of what exactly is being proposed. That's strange, as I copied you from the very first posting. Anyway, here's the archive: http://lists.denx.de/pipermail/u-boot/2012-October/138227.html I would especially like to know where such a new functionality should live, which data types it should operate on and what would be an appropriate name for it. Many thanks, Daniel ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
Hi Daniel, On Fri, Oct 26, 2012 at 12:24 AM, Daniel Mack zon...@gmail.com wrote: On 26.10.2012 02:53, David Gibson wrote: On Thu, Oct 25, 2012 at 10:46:32PM +0200, Wolfgang Denk wrote: Dear Daniel, In message 50893633.6070...@gmail.com you wrote: Overwrites must be addressed in the first place. The most common example is that a more generic part (the module tree) registers all details about a peripheral up-front but then sets its status to 'disabled'. That way, the more specific part (the base board tree) can overwrite this property to 'okay' at wish to enable it and not care for the pre-defined details. This is also how we do things in our device-trees. Agreed. I definitely can see the benefit of such a feature and would be happy if you could go forward and implement it. Ok then. I guess this should be something that can eventually be merged back into libfdt? I can't speak for the FDT custodian, but I think this makes a lot of sense. As a rule I'm happy to see more functionality for libfdt. I've only seen bits and pieces of this thread, though, so I'd need to see a summary of what exactly is being proposed. That's strange, as I copied you from the very first posting. Anyway, here's the archive: http://lists.denx.de/pipermail/u-boot/2012-October/138227.html I would especially like to know where such a new functionality should live, which data types it should operate on and what would be an appropriate name for it. It seems like this functionality should live in libfdt with its own tests there. If you are going to merge then you could perhaps also 'unmerge', i.e. given a base fdt and a derivative, create the 'diff' device tree. It should probably deal with all nodes/properties - after all the fdt doesn't really understand data types. Regards, Simon Many thanks, Daniel ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
On 10/24/2012 03:47 AM, Daniel Mack wrote: Hi, a project I'm involved in uses a module/baseboard combo, and components on either board are described in DT. I'm currently using separate dts files which build upon each other with include statements, which works fine for development. In production though, we will certainly have running changes (and hence different versions) over the lifetime of the product for both the baseboard and the module, and the hardware has support for identifying the versions of both sides at runtime. So let's say we have n versions of the baseboard and m versions of the module, we would much like to only prepare n + m files, instead of n * m by pre-compiling every possible combination (some of which may actually never occur 'in the wild'). So my question is: is it possible to do that kind of assembly of a number of files at runtime in U-Boot? I guess all it takes is merging a number of trees together, right? I browsed through the APIs but couldn't yet find an clear approach to that kind of problem. If not, what would it take to add that functionality? I can probably help with the implementation if someone tells me what would be the right way. Yes, solving this would be very useful; it's a wide-spread problem. Some thoughts though: Simply overlaying two DTBs on top of each-other (in the same fashion that dtc's /include/ statement would do at compile-time) might not be fully general enough, although perhaps it would be sufficient for your immediate needs. For example, lets say that a GPIO is routed from a device on the main board to a device on a daughter board, or even from one daughter board into the main board and back out to a different daughter board. Now, consider that the different board(s) that are the source of the GPIO might use completely different SoCs or versions of the SoC, which might require using a different GPIO specifier to represent the signal. That means you need to change the .dtb file for the client of the GPIO depending on the HW or .dtb that provides the GPIO. That's certainly not a simple matter of merging multiple .dtb blobs together. The same issue could easily apply to I2C or SPI buses, chip selects, etc. One solution would be to explicitly represent a connector or connection-point in DT, such that the connector can implement the naming of all signals that pass through it, and provide a translation point for hooking the two DT fragments together. This seems within the spirit of DT. Another solution might be some form of variables/macros/code in the DTB that can be used to parameterize other DTBs that get merged with it. This is probably an enormous can of worms. I wonder if similar yet more subtle issues might arise, such as some motherboards requiring an active-low IRQ signal yet others requiring an active-high IRQ signal, thus requiring a daughter-board to program its IRQ source differently. Similarly, what about different drive strength requirements for a signal source, depending on what board version receives the signal? ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
Dear Stephen Warren, In message 508ad8f9.8030...@wwwdotorg.org you wrote: Simply overlaying two DTBs on top of each-other (in the same fashion that dtc's /include/ statement would do at compile-time) might not be fully general enough, although perhaps it would be sufficient for your immediate needs. I think it should be sufficient for the overwhelming majority of use cases. When designing and implementing this feature, I suggest to start small with the most common use cases in mind only. For example, lets say that a GPIO is routed from a device on the main board to a device on a daughter board, or even from one daughter board into the main board and back out to a different daughter board. Now, consider that the different board(s) that are the source of the GPIO might use completely different SoCs or versions of the SoC, which might require using a different GPIO specifier to represent the signal. That means you need to change the .dtb file for the client of the GPIO depending on the HW or .dtb that provides the GPIO. That's certainly not a simple matter of merging multiple .dtb blobs together. Yes, one can construct arbitrarily complicated situations. But I think it is perfectly reasonable to ignore these, at least for the initial implementation. I'm not even convinced that we should try to come up with a solution that is capable of dealing automtically with any situation of such complexity. In reality, we can probably combine a simple overly mechanism with additional fixup though some shell script running FDT manipulation commands directly. I wonder if similar yet more subtle issues might arise, such as some motherboards requiring an active-low IRQ signal yet others requiring an active-high IRQ signal, thus requiring a daughter-board to program its IRQ source differently. Similarly, what about different drive strength requirements for a signal source, depending on what board version receives the signal? I suggest to try to ignore such situations for now, and get started with a working, simple implementation. If we actually run into a situation where handling such situations is needed, we can then discuss about solutions based on a much beter understanding and experience with the - then - existing simple code. In short: let's do a simple, working thing first, and add bells and whistles later. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de A dog always bit deepest on the veterinary hand. - Terry Pratchett, _Wyrd Sisters_ ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
Dear Daniel, In message 5087b919.2010...@gmail.com you wrote: So let's say we have n versions of the baseboard and m versions of the module, we would much like to only prepare n + m files, instead of n * m by pre-compiling every possible combination (some of which may actually never occur 'in the wild'). What you are facing is a situation that 1) appears to be pretty common, and 2) has (AFAICT) not been truely satisfactory solved yet. So my question is: is it possible to do that kind of assembly of a number of files at runtime in U-Boot? I guess all it takes is merging a number of trees together, right? I browsed through the APIs but couldn't yet find an clear approach to that kind of problem. If not, what would it take to add that functionality? I can probably help with the implementation if someone tells me what would be the right way. I think it should be possible to overlay several DT images; ideally these would be strictly orthogonal, i. e. the newly loaded one would only add new properties, but I think it should be also no problem to define some latest-wins policy, i. e. in case of already existing properties the newly loaded ones would just overwrite the previous settings. I definitely can see the benefit of such a feature and would be happy if you could go forward and implement it. Thanks in advance. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de While money can't buy happiness, it certainly lets you choose your own form of misery. ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
Hi Wolfgang, On 25.10.2012 14:44, Wolfgang Denk wrote: In message 5087b919.2010...@gmail.com you wrote: So let's say we have n versions of the baseboard and m versions of the module, we would much like to only prepare n + m files, instead of n * m by pre-compiling every possible combination (some of which may actually never occur 'in the wild'). What you are facing is a situation that 1) appears to be pretty common, and 2) has (AFAICT) not been truely satisfactory solved yet. So my question is: is it possible to do that kind of assembly of a number of files at runtime in U-Boot? I guess all it takes is merging a number of trees together, right? I browsed through the APIs but couldn't yet find an clear approach to that kind of problem. If not, what would it take to add that functionality? I can probably help with the implementation if someone tells me what would be the right way. I think it should be possible to overlay several DT images; ideally these would be strictly orthogonal, i. e. the newly loaded one would only add new properties, but I think it should be also no problem to define some latest-wins policy, i. e. in case of already existing properties the newly loaded ones would just overwrite the previous settings. Overwrites must be addressed in the first place. The most common example is that a more generic part (the module tree) registers all details about a peripheral up-front but then sets its status to 'disabled'. That way, the more specific part (the base board tree) can overwrite this property to 'okay' at wish to enable it and not care for the pre-defined details. This is also how we do things in our device-trees. I definitely can see the benefit of such a feature and would be happy if you could go forward and implement it. Ok then. I guess this should be something that can eventually be merged back into libfdt? Thanks, Daniel ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
Dear Daniel, In message 50893633.6070...@gmail.com you wrote: Overwrites must be addressed in the first place. The most common example is that a more generic part (the module tree) registers all details about a peripheral up-front but then sets its status to 'disabled'. That way, the more specific part (the base board tree) can overwrite this property to 'okay' at wish to enable it and not care for the pre-defined details. This is also how we do things in our device-trees. Agreed. I definitely can see the benefit of such a feature and would be happy if you could go forward and implement it. Ok then. I guess this should be something that can eventually be merged back into libfdt? I can't speak for the FDT custodian, but I think this makes a lot of sense. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de When the bosses talk about improving productivity, they are never talking about themselves. ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] Merging device trees at runtime for module-based systems
On Thu, Oct 25, 2012 at 10:46:32PM +0200, Wolfgang Denk wrote: Dear Daniel, In message 50893633.6070...@gmail.com you wrote: Overwrites must be addressed in the first place. The most common example is that a more generic part (the module tree) registers all details about a peripheral up-front but then sets its status to 'disabled'. That way, the more specific part (the base board tree) can overwrite this property to 'okay' at wish to enable it and not care for the pre-defined details. This is also how we do things in our device-trees. Agreed. I definitely can see the benefit of such a feature and would be happy if you could go forward and implement it. Ok then. I guess this should be something that can eventually be merged back into libfdt? I can't speak for the FDT custodian, but I think this makes a lot of sense. As a rule I'm happy to see more functionality for libfdt. I've only seen bits and pieces of this thread, though, so I'd need to see a summary of what exactly is being proposed. -- David Gibson| I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
[U-Boot] Merging device trees at runtime for module-based systems
Hi, a project I'm involved in uses a module/baseboard combo, and components on either board are described in DT. I'm currently using separate dts files which build upon each other with include statements, which works fine for development. In production though, we will certainly have running changes (and hence different versions) over the lifetime of the product for both the baseboard and the module, and the hardware has support for identifying the versions of both sides at runtime. So let's say we have n versions of the baseboard and m versions of the module, we would much like to only prepare n + m files, instead of n * m by pre-compiling every possible combination (some of which may actually never occur 'in the wild'). So my question is: is it possible to do that kind of assembly of a number of files at runtime in U-Boot? I guess all it takes is merging a number of trees together, right? I browsed through the APIs but couldn't yet find an clear approach to that kind of problem. If not, what would it take to add that functionality? I can probably help with the implementation if someone tells me what would be the right way. Any pointer greatly appreciated. Thanks, Daniel ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
