On Tue, Aug 25, 2026 at 5:13 AM Christian König <[email protected]> wrote: > > Hi Ravi, > > Essentially you can have two different modes of operation: > > 1. Preallocate resources on work submission to the HW. This is what the > dma_fence framework implements. > > The restriction here is that you don't have any dependency to userspace or > other device drivers, e.g. no page faulting, waiting for memory values etc... > > Background is that the core memory management depends so hard on the > execution properties of the dma_fence that none of those advanced features > are possible without breaking core assumptions. > > Inter device operation is only allowed if the device drivers "know" about > each other, for example between rendering and displaying pipelines. > > But we have intentionally rejected any approach of generalizing device to > device signaling in a framework because it makes dma_fence practically a > recipe for trouble and we have more than enough evidence to confirm this now. > > > 2. On demand resource allocation using PCIe functionalities like > IOMMUv2/ATC/ATS/PRI or device specific page fault handling. > > The point is that you can do things like generating commands on the fly, long > running jobs, device to device signaling etc... > > The only restriction is that you can't create a dma_fence with this as > output, you can only consume them as input to the operation. > > There was also an absolutely great presentation on the topic from Faith on > LPC a few years ago. > > Regards, > Christian. >
Christian and I discussed offline whether a fixed, kernel-controlled pipeline could support this safely. The contracts and enforcement remain open, and input from others working on these use cases would be welcome. Mode 1 (preallocated): The kernel must guarantee completion or safe termination so resources can be reclaimed when subsystems stop making progress. Arbitrary cross-device dependencies can break that guarantee. We discussed restricting the allowed pipelines as one possible approach, but still need to figure out how to enforce the dependencies and recover resources when a stage hangs. Mode 2 (on-demand): Standalone signaling is fine but very few accelerators fall under this category. (Bringing Rob's September 27 reply from the parallel thread CACSVV00oBPN=rw1yvbjnm2tmhuq+3iv84jkp4djqcnkrxcv...@mail.gmail.com/) Rob Clark <[email protected]> wrote: > So I'm just starting to think a bit about this, since it will be > potentially relevant when we implement fwsched for adreno.. > > Do we really need anything more than some sort of dma_fence signaling > extension? Ie. signaling on the CPU remains exactly the same as > today, anything waiting on the CPU would see no change. But if both > exporter and importer support the same extension, that can be used to > extract extra information to pass to fw. > > The fw might observe the fence getting signaled with lower latency, > but that's fine, there isn't really any guarantee (other than timeline > ordering) about the order that multiple waiters observe that the fence > is signaled. We've just pushed one of the waiters down to firmware, > that is all. Ie. something like > > rcu_read_lock(); > if (fence->ops->signaling_ext && > do_I_understand_this(fence->ops->signaling_ext)) { > fence->ops->get_magic_signaling_foo(fence, &magic); > ... pass magic to fw ... > } else { > ... same as before ... > } > rcu_read_unlock(); > > If either the importer or exporter don't understand the same signaling > extension, everything works as before. For ex, external eGPU/dGPU > sharing with say the on-chip ISP. But if both do support it, the > signaling extension provides a way to extract whatever magic thing is > needed to do the wait in fw. > > BR, > -R > Hi Rob, For operations that already satisfy the dma_fence completion contract, this looks worth exploring. Christian, does this fit the case you described about known drivers, provided the participating drivers preserve the resource-lifetime and recovery guarantees? Rob, could we have the shared signaling mechanism in an independent and generic synx_global.c layer? dma-fence can tap into this layer for the signaling extension you mentioned. The reason for keeping this layer independent is that we have clients who use reusable synx objects (signal same handle multiple times) which don't interop with dma-fence. And we continue to get more feature requirements from performance-critical clients around reusable objects with timelines. Assuming this is where global synchronization is headed, these clients should be able to tap into reusability via a route other than dma-fence. For existing dma-fence operations, your extension could expose the information firmware needs for its wait, preserving CPU signaling and the fallback you described. Other clients could use the synx global layer under a separately defined resource-management contract. Would such layering address your use case while also supporting clients that need reusable remote synchronization? Thanks, Pravin
