Re: [RESEND PATCH v1 0/5] Solve postboot supplier cleanup and optimize probe ordering
On Wed, Jun 26, 2019 at 2:31 PM Rob Herring wrote: > > On Mon, Jun 24, 2019 at 9:54 PM Greg Kroah-Hartman > wrote: > > > > On Mon, Jun 24, 2019 at 03:37:07PM -0700, Sandeep Patil wrote: > > > We are trying to make sure that all (most) drivers in an Aarch64 system > > > can > > > be kernel modules for Android, like any other desktop system for > > > example. There are a number of problems we need to fix before that happens > > > ofcourse. > > > > I will argue that this is NOT an android-specific issue. If the goal of > > creating an arm64 kernel that will "just work" for a wide range of > > hardware configurations without rebuilding is going to happen, we need > > to solve this problem with DT. This goal was one of the original wishes > > of the arm64 development effort, let's not loose sight of it as > > obviously, this is not working properly just yet. > > I fail to see how the different Linux behavior between drivers > built-in and as modules has anything whatsoever to do with DT. You are right, built-in vs module problem is not a DT issue. But this is not so much a built-in vs module issue. It's just that built-in has a hack available that works sometimes. But really, both are broken. > Fix the > problems in Linux and use the dependencies that are already expressed > in DT and *then* we can talk about using DT to provide *hints* for > solving any remaining problems. Done. Sent v2 patch series that uses existing bindings. -Saravana
Re: [RESEND PATCH v1 0/5] Solve postboot supplier cleanup and optimize probe ordering
On Mon, Jun 24, 2019 at 9:54 PM Greg Kroah-Hartman wrote: > > On Mon, Jun 24, 2019 at 03:37:07PM -0700, Sandeep Patil wrote: > > We are trying to make sure that all (most) drivers in an Aarch64 system can > > be kernel modules for Android, like any other desktop system for > > example. There are a number of problems we need to fix before that happens > > ofcourse. > > I will argue that this is NOT an android-specific issue. If the goal of > creating an arm64 kernel that will "just work" for a wide range of > hardware configurations without rebuilding is going to happen, we need > to solve this problem with DT. This goal was one of the original wishes > of the arm64 development effort, let's not loose sight of it as > obviously, this is not working properly just yet. I fail to see how the different Linux behavior between drivers built-in and as modules has anything whatsoever to do with DT. Fix the problems in Linux and use the dependencies that are already expressed in DT and *then* we can talk about using DT to provide *hints* for solving any remaining problems. Rob
Re: [RESEND PATCH v1 0/5] Solve postboot supplier cleanup and optimize probe ordering
On 6/25/19 9:30 PM, Sandeep Patil wrote: > On Tue, Jun 25, 2019 at 11:53:13AM +0800, Greg Kroah-Hartman wrote: >> On Mon, Jun 24, 2019 at 03:37:07PM -0700, Sandeep Patil wrote: >>> We are trying to make sure that all (most) drivers in an Aarch64 system can >>> be kernel modules for Android, like any other desktop system for >>> example. There are a number of problems we need to fix before that happens >>> ofcourse. >> >> I will argue that this is NOT an android-specific issue. If the goal of >> creating an arm64 kernel that will "just work" for a wide range of >> hardware configurations without rebuilding is going to happen, we need >> to solve this problem with DT. This goal was one of the original wishes >> of the arm64 development effort, let's not loose sight of it as >> obviously, this is not working properly just yet. > > I believe the proposed solution in this patch series is just that. I am not > sure what the alternatives are. The alternative suggested was to reuse Look at the responses from myself and from Rob to patch 0. No one responded to our comments. -Frank > pre-existing dt-bindings for dependency based probe re-ordering and > resolution. > > However, it seems we had no way to *really* check if these dependencies are > the real. So, a device may or may not actually depend on the other device for > probe / initialization when the dependency is mentioned in it's dt node. From > DT's point of view, there is no way to tell this .. > > I don't know how this is handled in x86. With DT, I don't see how we can do > this unless DT dependencies are _really_ tied with runtime dependencies (The > cycles would have been apparent if that was the case. > > Honestly, the "depends-on" property suggested here just piles on to the > existing state. So, it is somewhat doubling the exiting bindings. It says, > you must use depends-on property to define probe / initialization dependency. > The existing bindings like 'clock', 'interrupt', '*-supply' do not enforce > that right now, so you will have device nodes that have these bindings right > now but don't necessarily need them for successful probe for example. > >> >> It just seems that Android is the first one to actually try and >> implement that goal :) > > I guess :) > > - ssp > >> >> thanks, >> >> greg k-h >
Re: [RESEND PATCH v1 0/5] Solve postboot supplier cleanup and optimize probe ordering
On Tue, Jun 25, 2019 at 11:53:13AM +0800, Greg Kroah-Hartman wrote: > On Mon, Jun 24, 2019 at 03:37:07PM -0700, Sandeep Patil wrote: > > We are trying to make sure that all (most) drivers in an Aarch64 system can > > be kernel modules for Android, like any other desktop system for > > example. There are a number of problems we need to fix before that happens > > ofcourse. > > I will argue that this is NOT an android-specific issue. If the goal of > creating an arm64 kernel that will "just work" for a wide range of > hardware configurations without rebuilding is going to happen, we need > to solve this problem with DT. This goal was one of the original wishes > of the arm64 development effort, let's not loose sight of it as > obviously, this is not working properly just yet. I believe the proposed solution in this patch series is just that. I am not sure what the alternatives are. The alternative suggested was to reuse pre-existing dt-bindings for dependency based probe re-ordering and resolution. However, it seems we had no way to *really* check if these dependencies are the real. So, a device may or may not actually depend on the other device for probe / initialization when the dependency is mentioned in it's dt node. From DT's point of view, there is no way to tell this .. I don't know how this is handled in x86. With DT, I don't see how we can do this unless DT dependencies are _really_ tied with runtime dependencies (The cycles would have been apparent if that was the case. Honestly, the "depends-on" property suggested here just piles on to the existing state. So, it is somewhat doubling the exiting bindings. It says, you must use depends-on property to define probe / initialization dependency. The existing bindings like 'clock', 'interrupt', '*-supply' do not enforce that right now, so you will have device nodes that have these bindings right now but don't necessarily need them for successful probe for example. > > It just seems that Android is the first one to actually try and > implement that goal :) I guess :) - ssp > > thanks, > > greg k-h
Re: [RESEND PATCH v1 0/5] Solve postboot supplier cleanup and optimize probe ordering
On Mon, Jun 24, 2019 at 03:37:07PM -0700, Sandeep Patil wrote: > We are trying to make sure that all (most) drivers in an Aarch64 system can > be kernel modules for Android, like any other desktop system for > example. There are a number of problems we need to fix before that happens > ofcourse. I will argue that this is NOT an android-specific issue. If the goal of creating an arm64 kernel that will "just work" for a wide range of hardware configurations without rebuilding is going to happen, we need to solve this problem with DT. This goal was one of the original wishes of the arm64 development effort, let's not loose sight of it as obviously, this is not working properly just yet. It just seems that Android is the first one to actually try and implement that goal :) thanks, greg k-h
Re: [RESEND PATCH v1 0/5] Solve postboot supplier cleanup and optimize probe ordering
(Responding to the first email in the series to summarize the current situation and choices we have.) On Mon, Jun 03, 2019 at 05:32:13PM -0700, 'Saravana Kannan' via kernel-team wrote: > Add a generic "depends-on" property that allows specifying mandatory > functional dependencies between devices. Add device-links after the > devices are created (but before they are probed) by looking at this > "depends-on" property. > > This property is used instead of existing DT properties that specify > phandles of other devices (Eg: clocks, pinctrl, regulators, etc). This > is because not all resources referred to by existing DT properties are > mandatory functional dependencies. Some devices/drivers might be able > to operate with reduced functionality when some of the resources > aren't available. For example, a device could operate in polling mode > if no IRQ is available, a device could skip doing power management if > clock or voltage control isn't available and they are left on, etc. > > So, adding mandatory functional dependency links between devices by > looking at referred phandles in DT properties won't work as it would > prevent probing devices that could be probed. By having an explicit > depends-on property, we can handle these cases correctly. > > Having functional dependencies explicitly called out in DT and > automatically added before the devices are probed, provides the > following benefits: > > - Optimizes device probe order and avoids the useless work of > attempting probes of devices that will not probe successfully > (because their suppliers aren't present or haven't probed yet). > > For example, in a commonly available mobile SoC, registering just > one consumer device's driver at an initcall level earlier than the > supplier device's driver causes 11 failed probe attempts before the > consumer device probes successfully. This was with a kernel with all > the drivers statically compiled in. This problem gets a lot worse if > all the drivers are loaded as modules without direct symbol > dependencies. > > - Supplier devices like clock providers, regulators providers, etc > need to keep the resources they provide active and at a particular > state(s) during boot up even if their current set of consumers don't > request the resource to be active. This is because the rest of the > consumers might not have probed yet and turning off the resource > before all the consumers have probed could lead to a hang or > undesired user experience. > > Some frameworks (Eg: regulator) handle this today by turning off > "unused" resources at late_initcall_sync and hoping all the devices > have probed by then. This is not a valid assumption for systems with > loadable modules. Other frameworks (Eg: clock) just don't handle > this due to the lack of a clear signal for when they can turn off > resources. This leads to downstream hacks to handle cases like this > that can easily be solved in the upstream kernel. > > By linking devices before they are probed, we give suppliers a clear > count of the number of dependent consumers. Once all of the > consumers are active, the suppliers can turn off the unused > resources without making assumptions about the number of consumers. > > By default we just add device-links to track "driver presence" (probe > succeeded) of the supplier device. If any other functionality provided > by device-links are needed, it is left to the consumer/supplier > devices to change the link when they probe. > We are trying to make sure that all (most) drivers in an Aarch64 system can be kernel modules for Android, like any other desktop system for example. There are a number of problems we need to fix before that happens ofcourse. That patch series does the following - 1. Resolve the consumer<->supplier relationship in order for subsystems to turn off resources to save power is #1 on our list. This is because it will define how driver and DT writers define and write their code for Android devices. 2. Resolve cyclic dependency between devices nodes as a side-effect. That will make sure Android systems do not suffer from deferred probing and we have a generic way of serialize driver probes. (This can also be mitigated somewhat by module dependencies outside of the kernel.) Subsystems like regulator, interconnect can immediately benefit from #1 and it makes the module/driver loading possible for Android systems without regressing power horribly. After thinking about Rob's suggestion to loop though dependencies, I think it doesn't work because we don't know what to do when there are cycles. Drivers sometimes need to have access to phandles in order to retrieve resources from that phandle. We can't assume that is an implied "probe dependency". Saravana had several such examples already and it is a _real world_ scenario. I think the current binding are insufficient to describe the runtime hardware dependencies. IOW, the current
Re: [RESEND PATCH v1 0/5] Solve postboot supplier cleanup and optimize probe ordering
Adding cc: David Collins Plus my comments below. On 6/3/19 5:32 PM, Saravana Kannan wrote: > Add a generic "depends-on" property that allows specifying mandatory > functional dependencies between devices. Add device-links after the > devices are created (but before they are probed) by looking at this > "depends-on" property. > > This property is used instead of existing DT properties that specify > phandles of other devices (Eg: clocks, pinctrl, regulators, etc). This > is because not all resources referred to by existing DT properties are > mandatory functional dependencies. Some devices/drivers might be able > to operate with reduced functionality when some of the resources > aren't available. For example, a device could operate in polling mode > if no IRQ is available, a device could skip doing power management if > clock or voltage control isn't available and they are left on, etc. > > So, adding mandatory functional dependency links between devices by > looking at referred phandles in DT properties won't work as it would > prevent probing devices that could be probed. By having an explicit > depends-on property, we can handle these cases correctly. > > Having functional dependencies explicitly called out in DT and > automatically added before the devices are probed, provides the > following benefits: > > - Optimizes device probe order and avoids the useless work of > attempting probes of devices that will not probe successfully > (because their suppliers aren't present or haven't probed yet). > > For example, in a commonly available mobile SoC, registering just > one consumer device's driver at an initcall level earlier than the > supplier device's driver causes 11 failed probe attempts before the > consumer device probes successfully. This was with a kernel with all > the drivers statically compiled in. This problem gets a lot worse if > all the drivers are loaded as modules without direct symbol > dependencies. > > - Supplier devices like clock providers, regulators providers, etc > need to keep the resources they provide active and at a particular > state(s) during boot up even if their current set of consumers don't > request the resource to be active. This is because the rest of the > consumers might not have probed yet and turning off the resource > before all the consumers have probed could lead to a hang or > undesired user experience. > > Some frameworks (Eg: regulator) handle this today by turning off > "unused" resources at late_initcall_sync and hoping all the devices > have probed by then. This is not a valid assumption for systems with > loadable modules. Other frameworks (Eg: clock) just don't handle > this due to the lack of a clear signal for when they can turn off > resources. This leads to downstream hacks to handle cases like this > that can easily be solved in the upstream kernel. > > By linking devices before they are probed, we give suppliers a clear > count of the number of dependent consumers. Once all of the > consumers are active, the suppliers can turn off the unused > resources without making assumptions about the number of consumers. > > By default we just add device-links to track "driver presence" (probe > succeeded) of the supplier device. If any other functionality provided > by device-links are needed, it is left to the consumer/supplier > devices to change the link when they probe. > > > Saravana Kannan (5): > of/platform: Speed up of_find_device_by_node() > driver core: Add device links support for pending links to suppliers > dt-bindings: Add depends-on property > of/platform: Add functional dependency link from "depends-on" property > driver core: Add sync_state driver/bus callback > > .../devicetree/bindings/depends-on.txt| 26 + > drivers/base/core.c | 106 ++ > drivers/of/platform.c | 75 - > include/linux/device.h| 24 > include/linux/of.h| 3 + > 5 files changed, 233 insertions(+), 1 deletion(-) > create mode 100644 Documentation/devicetree/bindings/depends-on.txt > I don't think the above description adequately describes one key problem that the patch set addresses. David Collins described the problem in an email late in the thread of the first submission of this series. Instead of providing a link to that email, I am going to fully copy it here: On 5/31/19 4:27 PM, David Collins wrote: > Hello Saravana, > > On 5/23/19 6:01 PM, Saravana Kannan wrote: > ... >> Having functional dependencies explicitly called out in DT and >> automatically added before the devices are probed, provides the >> following benefits: > ... >> - Supplier devices like clock providers, regulators providers, etc >> need to keep the resources they provide active and at a particular >> state(s) during boot up even if their current set of consumers
Re: [RESEND PATCH v1 0/5] Solve postboot supplier cleanup and optimize probe ordering
On Wed, Jun 12, 2019 at 3:21 PM Frank Rowand wrote: > > Adding cc: David Collins > > Plus my comments below. > > On 6/3/19 5:32 PM, Saravana Kannan wrote: > > Add a generic "depends-on" property that allows specifying mandatory > > functional dependencies between devices. Add device-links after the > > devices are created (but before they are probed) by looking at this > > "depends-on" property. > > > > This property is used instead of existing DT properties that specify > > phandles of other devices (Eg: clocks, pinctrl, regulators, etc). This > > is because not all resources referred to by existing DT properties are > > mandatory functional dependencies. Some devices/drivers might be able > > to operate with reduced functionality when some of the resources > > aren't available. For example, a device could operate in polling mode > > if no IRQ is available, a device could skip doing power management if > > clock or voltage control isn't available and they are left on, etc. > > > > So, adding mandatory functional dependency links between devices by > > looking at referred phandles in DT properties won't work as it would > > prevent probing devices that could be probed. By having an explicit > > depends-on property, we can handle these cases correctly. > > > > Having functional dependencies explicitly called out in DT and > > automatically added before the devices are probed, provides the > > following benefits: > > > > - Optimizes device probe order and avoids the useless work of > > attempting probes of devices that will not probe successfully > > (because their suppliers aren't present or haven't probed yet). > > > > For example, in a commonly available mobile SoC, registering just > > one consumer device's driver at an initcall level earlier than the > > supplier device's driver causes 11 failed probe attempts before the > > consumer device probes successfully. This was with a kernel with all > > the drivers statically compiled in. This problem gets a lot worse if > > all the drivers are loaded as modules without direct symbol > > dependencies. > > > > - Supplier devices like clock providers, regulators providers, etc > > need to keep the resources they provide active and at a particular > > state(s) during boot up even if their current set of consumers don't > > request the resource to be active. This is because the rest of the > > consumers might not have probed yet and turning off the resource > > before all the consumers have probed could lead to a hang or > > undesired user experience. > > > > Some frameworks (Eg: regulator) handle this today by turning off > > "unused" resources at late_initcall_sync and hoping all the devices > > have probed by then. This is not a valid assumption for systems with > > loadable modules. Other frameworks (Eg: clock) just don't handle > > this due to the lack of a clear signal for when they can turn off > > resources. This leads to downstream hacks to handle cases like this > > that can easily be solved in the upstream kernel. > > > > By linking devices before they are probed, we give suppliers a clear > > count of the number of dependent consumers. Once all of the > > consumers are active, the suppliers can turn off the unused > > resources without making assumptions about the number of consumers. > > > > By default we just add device-links to track "driver presence" (probe > > succeeded) of the supplier device. If any other functionality provided > > by device-links are needed, it is left to the consumer/supplier > > devices to change the link when they probe. > > > > > > Saravana Kannan (5): > > of/platform: Speed up of_find_device_by_node() > > driver core: Add device links support for pending links to suppliers > > dt-bindings: Add depends-on property > > of/platform: Add functional dependency link from "depends-on" property > > driver core: Add sync_state driver/bus callback > > > > .../devicetree/bindings/depends-on.txt| 26 + > > drivers/base/core.c | 106 ++ > > drivers/of/platform.c | 75 - > > include/linux/device.h| 24 > > include/linux/of.h| 3 + > > 5 files changed, 233 insertions(+), 1 deletion(-) > > create mode 100644 Documentation/devicetree/bindings/depends-on.txt > > > > > I don't think the above description adequately describes one key problem > that the patch set addresses. > > David Collins described the problem in an email late in the thread of > the first submission of this series. Instead of providing a link to > that email, I am going to fully copy it here: > > On 5/31/19 4:27 PM, David Collins wrote: > > Hello Saravana, > > > > On 5/23/19 6:01 PM, Saravana Kannan wrote: > > ... > >> Having functional dependencies explicitly called out in DT and > >> automatically added before the devices are probed, provides the > >>
