[Mesa-dev] [PATCH] dma-buf: Document dma-buf implicit fencing/resv fencing rules
Docs for struct dma_resv are fairly clear: "A reservation object can have attached one exclusive fence (normally associated with write operations) or N shared fences (read operations)." https://dri.freedesktop.org/docs/drm/driver-api/dma-buf.html#reservation-objects Furthermore a review across all of upstream. First of render drivers and how they set implicit fences: - nouveau follows this contract, see in validate_fini_no_ticket() nouveau_bo_fence(nvbo, fence, !!b->write_domains); and that last boolean controls whether the exclusive or shared fence slot is used. - radeon follows this contract by setting p->relocs[i].tv.num_shared = !r->write_domain; in radeon_cs_parser_relocs(), which ensures that the call to ttm_eu_fence_buffer_objects() in radeon_cs_parser_fini() will do the right thing. - vmwgfx seems to follow this contract with the shotgun approach of always setting ttm_val_buf->num_shared = 0, which means ttm_eu_fence_buffer_objects() will only use the exclusive slot. - etnaviv follows this contract, as can be trivially seen by looking at submit_attach_object_fences() - i915 is a bit a convoluted maze with multiple paths leading to i915_vma_move_to_active(). Which sets the exclusive flag if EXEC_OBJECT_WRITE is set. This can either come as a buffer flag for softpin mode, or through the write_domain when using relocations. It follows this contract. - lima follows this contract, see lima_gem_submit() which sets the exclusive fence when the LIMA_SUBMIT_BO_WRITE flag is set for that bo - msm follows this contract, see msm_gpu_submit() which sets the exclusive flag when the MSM_SUBMIT_BO_WRITE is set for that buffer - panfrost follows this contract with the shotgun approach of just always setting the exclusive fence, see panfrost_attach_object_fences(). Benefits of a single engine I guess - v3d follows this contract with the same shotgun approach in v3d_attach_fences_and_unlock_reservation(), but it has at least an XXX comment that maybe this should be improved - v4c uses the same shotgun approach of always setting an exclusive fence, see vc4_update_bo_seqnos() - vgem also follows this contract, see vgem_fence_attach_ioctl() and the VGEM_FENCE_WRITE. This is used in some igts to validate prime sharing with i915.ko without the need of a 2nd gpu - vritio follows this contract again with the shotgun approach of always setting an exclusive fence, see virtio_gpu_array_add_fence() This covers the setting of the exclusive fences when writing. Synchronizing against the exclusive fence is a lot more tricky, and I only spot checked a few: - i915 does it, with the optional EXEC_OBJECT_ASYNC to skip all implicit dependencies (which is used by vulkan) - etnaviv does this. Implicit dependencies are collected in submit_fence_sync(), again with an opt-out flag ETNA_SUBMIT_NO_IMPLICIT. These are then picked up in etnaviv_sched_dependency which is the drm_sched_backend_ops->dependency callback. - v4c seems to not do much here, maybe gets away with it by not having a scheduler and only a single engine. Since all newer broadcom chips than the OG vc4 use v3d for rendering, which follows this contract, the impact of this issue is fairly small. - v3d does this using the drm_gem_fence_array_add_implicit() helper, which then it's drm_sched_backend_ops->dependency callback v3d_job_dependency() picks up. - panfrost is nice here and tracks the implicit fences in panfrost_job->implicit_fences, which again the drm_sched_backend_ops->dependency callback panfrost_job_dependency() picks up. It is mildly questionable though since it only picks up exclusive fences in panfrost_acquire_object_fences(), but not buggy in practice because it also always sets the exclusive fence. It should pick up both sets of fences, just in case there's ever going to be a 2nd gpu in a SoC with a mali gpu. Or maybe a mali SoC with a pcie port and a real gpu, which might actually happen eventually. A bug, but easy to fix. Should probably use the drm_gem_fence_array_add_implicit() helper. - lima is nice an easy, uses drm_gem_fence_array_add_implicit() and the same schema as v3d. - msm is mildly entertaining. It also supports MSM_SUBMIT_NO_IMPLICIT, but because it doesn't use the drm/scheduler it handles fences from the wrong context with a synchronous dma_fence_wait. See submit_fence_sync() leading to msm_gem_sync_object(). Investing into a scheduler might be a good idea. - all the remaining drivers are ttm based, where I hope they do appropriately obey implicit fences already. I didn't do the full audit there because a) not follow the contract would confuse ttm quite well and b) reading non-standard scheduler and submit code which isn't based on drm/scheduler is a pain. Onwards to the display side. - Any driver using the drm_gem_plane_helper_prepare_fb() helper will correctly.
Re: [Mesa-dev] [PATCH] dma-buf: Document dma-buf implicit fencing/resv fencing rules
Hi, On Wed, 23 Jun 2021 at 17:20, Daniel Vetter wrote: > +* > +* IMPLICIT SYNCHRONIZATION RULES: > +* > +* Drivers which support implicit synchronization of buffer access as > +* e.g. exposed in `Implicit Fence Poll Support`_ should follow the > +* below rules. 'Should' ... ? Must. > +* - Drivers should add a shared fence through > +* dma_resv_add_shared_fence() for anything the userspace API > +* considers a read access. This highly depends upon the API and > +* window system: E.g. OpenGL is generally implicitly synchronized > on > +* Linux, but explicitly synchronized on Android. Whereas Vulkan is > +* generally explicitly synchronized for everything, and window > system > +* buffers have explicit API calls (which then need to make sure the > +* implicit fences store here in @resv are updated correctly). > +* > +* - [...] Mmm, I think this is all right, but it could be worded much more clearly. Right now it's a bunch of points all smashed into one, and there's a lot of room for misinterpretation. Here's a strawman, starting with most basic and restrictive, working through to when you're allowed to wriggle your way out: Rule 1: Drivers must add a shared fence through dma_resv_add_shared_fence() for any read accesses against that buffer. This appends a fence to the shared array, ensuring that any future non-read access will be synchronised against this operation to only begin after it has completed. Rule 2: Drivers must add an exclusive fence through dma_resv_add_excl_fence() for any write accesses against that buffer. This replaces the exclusive fence with the new operation, ensuring that all future access will be synchronised against this operation to only begin after it has completed. Rule 3: Drivers must synchronise all accesses to buffers against existing implicit fences. Read accesses must synchronise against the exclusive fence (read-after-write), and write accesses must synchronise against both the exclusive (write-after-write) and shared (write-after-read) fences. Note 1: Users like OpenGL and window systems on non-Android userspace are generally implicitly synchronised. An implicitly-synchronised userspace is unaware of fences from prior operations, so the kernel mediates scheduling to create the illusion that GPU work is FIFO. For example, an application will flush and schedule GPU write work to render its image, then immediately tell the window system to display that image; the window system may immediately flush and schedule GPU read work to display that image, with neither waiting for the write to have completed. The kernel provides coherence by synchronising the read access against the write fence in the exclusive slot, so that the image displayed is correct. Note 2: Users like Vulkan and Android window system are generally explicitly synchronised. An explicitly-synchronised userspace is responsible for tracking its own read and write access and providing the kernel with synchronisation barriers. For instance, a Vulkan application rendering to a buffer and subsequently using it as a read texture, must annotate the read operation with a read-after-write synchronisation barrier. Note 3: Implicit and explicit userspace can coexist. For instance, an explicitly-synchronised Vulkan application may be running as a client of an implicitly-synchronised window system which uses OpenGL for composition; an implicitly-synchronised OpenGL application may be running as a client of a window system which uses Vulkan for composition. Note 4: Some subsystems, for example V4L2, do not pipeline operations, and instead only return to userspace when the scheduled work against a buffer has fully retired. Exemption 1: Fully self-coherent userspace may skip implicit synchronisation barriers. For instance, accesses between two Vulkan-internal buffers allocated by a single application do not need to synchronise against each other's implicit fences, as the client is responsible for explicitly providing barriers for access. A self-contained OpenGL userspace also has no need to implicitly synchronise its access if the driver instead tracks all access and inserts the appropriate synchronisation barriers. Exemption 2: When implicit and explicit userspace coexist, the explicit side may skip intermediate synchronisation, and only place synchronisation barriers at transition points. For example, a Vulkan compositor displaying a buffer from an OpenGL application would need to synchronise its first access against the fence placed in the exclusive implicit-synchronisation slot. Once this read has fully retired, the compositor has no need to participate in implicit synchronisation until it is ready to return the buffer to the application, at which point it must insert all its non-retired accesses into the shared slot, which the application will then synchronise future write
Re: [Mesa-dev] [PATCH] dma-buf: Document dma-buf implicit fencing/resv fencing rules
On Thu, Jun 24, 2021 at 1:08 PM Daniel Stone wrote: > > Hi, > > On Wed, 23 Jun 2021 at 17:20, Daniel Vetter wrote: > > +* > > +* IMPLICIT SYNCHRONIZATION RULES: > > +* > > +* Drivers which support implicit synchronization of buffer access > > as > > +* e.g. exposed in `Implicit Fence Poll Support`_ should follow the > > +* below rules. > > 'Should' ... ? Must. Yeah I guess I can upgrade a bunch of them. > > +* - Drivers should add a shared fence through > > +* dma_resv_add_shared_fence() for anything the userspace API > > +* considers a read access. This highly depends upon the API and > > +* window system: E.g. OpenGL is generally implicitly > > synchronized on > > +* Linux, but explicitly synchronized on Android. Whereas Vulkan > > is > > +* generally explicitly synchronized for everything, and window > > system > > +* buffers have explicit API calls (which then need to make sure > > the > > +* implicit fences store here in @resv are updated correctly). > > +* > > +* - [...] > > Mmm, I think this is all right, but it could be worded much more > clearly. Right now it's a bunch of points all smashed into one, and > there's a lot of room for misinterpretation. > > Here's a strawman, starting with most basic and restrictive, working > through to when you're allowed to wriggle your way out: > > Rule 1: Drivers must add a shared fence through > dma_resv_add_shared_fence() for any read accesses against that buffer. > This appends a fence to the shared array, ensuring that any future > non-read access will be synchronised against this operation to only > begin after it has completed. > > Rule 2: Drivers must add an exclusive fence through > dma_resv_add_excl_fence() for any write accesses against that buffer. > This replaces the exclusive fence with the new operation, ensuring > that all future access will be synchronised against this operation to > only begin after it has completed. > > Rule 3: Drivers must synchronise all accesses to buffers against > existing implicit fences. Read accesses must synchronise against the > exclusive fence (read-after-write), and write accesses must > synchronise against both the exclusive (write-after-write) and shared > (write-after-read) fences. > > Note 1: Users like OpenGL and window systems on non-Android userspace > are generally implicitly synchronised. An implicitly-synchronised > userspace is unaware of fences from prior operations, so the kernel > mediates scheduling to create the illusion that GPU work is FIFO. For > example, an application will flush and schedule GPU write work to > render its image, then immediately tell the window system to display > that image; the window system may immediately flush and schedule GPU > read work to display that image, with neither waiting for the write to > have completed. The kernel provides coherence by synchronising the > read access against the write fence in the exclusive slot, so that the > image displayed is correct. > > Note 2: Users like Vulkan and Android window system are generally > explicitly synchronised. An explicitly-synchronised userspace is > responsible for tracking its own read and write access and providing > the kernel with synchronisation barriers. For instance, a Vulkan > application rendering to a buffer and subsequently using it as a read > texture, must annotate the read operation with a read-after-write > synchronisation barrier. > > Note 3: Implicit and explicit userspace can coexist. For instance, an > explicitly-synchronised Vulkan application may be running as a client > of an implicitly-synchronised window system which uses OpenGL for > composition; an implicitly-synchronised OpenGL application may be > running as a client of a window system which uses Vulkan for > composition. > > Note 4: Some subsystems, for example V4L2, do not pipeline operations, > and instead only return to userspace when the scheduled work against a > buffer has fully retired. > > Exemption 1: Fully self-coherent userspace may skip implicit > synchronisation barriers. For instance, accesses between two > Vulkan-internal buffers allocated by a single application do not need > to synchronise against each other's implicit fences, as the client is > responsible for explicitly providing barriers for access. A > self-contained OpenGL userspace also has no need to implicitly > synchronise its access if the driver instead tracks all access and > inserts the appropriate synchronisation barriers. > > Exemption 2: When implicit and explicit userspace coexist, the > explicit side may skip intermediate synchronisation, and only place > synchronisation barriers at transition points. For example, a Vulkan > compositor displaying a buffer from an OpenGL application would need > to synchronise its first access against the fence placed in the > exclusive implicit-synchronisation slot. Once
[Mesa-dev] [PATCH] dma-buf: Document dma-buf implicit fencing/resv fencing rules
Docs for struct dma_resv are fairly clear: "A reservation object can have attached one exclusive fence (normally associated with write operations) or N shared fences (read operations)." https://dri.freedesktop.org/docs/drm/driver-api/dma-buf.html#reservation-objects Furthermore a review across all of upstream. First of render drivers and how they set implicit fences: - nouveau follows this contract, see in validate_fini_no_ticket() nouveau_bo_fence(nvbo, fence, !!b->write_domains); and that last boolean controls whether the exclusive or shared fence slot is used. - radeon follows this contract by setting p->relocs[i].tv.num_shared = !r->write_domain; in radeon_cs_parser_relocs(), which ensures that the call to ttm_eu_fence_buffer_objects() in radeon_cs_parser_fini() will do the right thing. - vmwgfx seems to follow this contract with the shotgun approach of always setting ttm_val_buf->num_shared = 0, which means ttm_eu_fence_buffer_objects() will only use the exclusive slot. - etnaviv follows this contract, as can be trivially seen by looking at submit_attach_object_fences() - i915 is a bit a convoluted maze with multiple paths leading to i915_vma_move_to_active(). Which sets the exclusive flag if EXEC_OBJECT_WRITE is set. This can either come as a buffer flag for softpin mode, or through the write_domain when using relocations. It follows this contract. - lima follows this contract, see lima_gem_submit() which sets the exclusive fence when the LIMA_SUBMIT_BO_WRITE flag is set for that bo - msm follows this contract, see msm_gpu_submit() which sets the exclusive flag when the MSM_SUBMIT_BO_WRITE is set for that buffer - panfrost follows this contract with the shotgun approach of just always setting the exclusive fence, see panfrost_attach_object_fences(). Benefits of a single engine I guess - v3d follows this contract with the same shotgun approach in v3d_attach_fences_and_unlock_reservation(), but it has at least an XXX comment that maybe this should be improved - v4c uses the same shotgun approach of always setting an exclusive fence, see vc4_update_bo_seqnos() - vgem also follows this contract, see vgem_fence_attach_ioctl() and the VGEM_FENCE_WRITE. This is used in some igts to validate prime sharing with i915.ko without the need of a 2nd gpu - vritio follows this contract again with the shotgun approach of always setting an exclusive fence, see virtio_gpu_array_add_fence() This covers the setting of the exclusive fences when writing. Synchronizing against the exclusive fence is a lot more tricky, and I only spot checked a few: - i915 does it, with the optional EXEC_OBJECT_ASYNC to skip all implicit dependencies (which is used by vulkan) - etnaviv does this. Implicit dependencies are collected in submit_fence_sync(), again with an opt-out flag ETNA_SUBMIT_NO_IMPLICIT. These are then picked up in etnaviv_sched_dependency which is the drm_sched_backend_ops->dependency callback. - v4c seems to not do much here, maybe gets away with it by not having a scheduler and only a single engine. Since all newer broadcom chips than the OG vc4 use v3d for rendering, which follows this contract, the impact of this issue is fairly small. - v3d does this using the drm_gem_fence_array_add_implicit() helper, which then it's drm_sched_backend_ops->dependency callback v3d_job_dependency() picks up. - panfrost is nice here and tracks the implicit fences in panfrost_job->implicit_fences, which again the drm_sched_backend_ops->dependency callback panfrost_job_dependency() picks up. It is mildly questionable though since it only picks up exclusive fences in panfrost_acquire_object_fences(), but not buggy in practice because it also always sets the exclusive fence. It should pick up both sets of fences, just in case there's ever going to be a 2nd gpu in a SoC with a mali gpu. Or maybe a mali SoC with a pcie port and a real gpu, which might actually happen eventually. A bug, but easy to fix. Should probably use the drm_gem_fence_array_add_implicit() helper. - lima is nice an easy, uses drm_gem_fence_array_add_implicit() and the same schema as v3d. - msm is mildly entertaining. It also supports MSM_SUBMIT_NO_IMPLICIT, but because it doesn't use the drm/scheduler it handles fences from the wrong context with a synchronous dma_fence_wait. See submit_fence_sync() leading to msm_gem_sync_object(). Investing into a scheduler might be a good idea. - all the remaining drivers are ttm based, where I hope they do appropriately obey implicit fences already. I didn't do the full audit there because a) not follow the contract would confuse ttm quite well and b) reading non-standard scheduler and submit code which isn't based on drm/scheduler is a pain. Onwards to the display side. - Any driver using the drm_gem_plane_helper_prepare_fb() helper will correctly.
[Mesa-dev] [ANNOUNCE] glu 9.0.2
Big thanks to Dylan for the Meson build system! Dylan Baker (3): editorconfig: Add configuration for Meson Add Meson build system Makefile: Distribute Meson files as part of the dist tarball Kevin Bowling (1): build: Add support for libglvnd Lucas Stach (1): build: fix the debug parameter to properly handle --disable-debug Matt Turner (2): Remove glu_mangle.h glu 9.0.2 Nicolas Caramelli (1): Check the definition instead of the extension to which it belongs git tag: glu-9.0.2 https://mesa.freedesktop.org/archive/glu/glu-9.0.2.tar.gz SHA256: 24effdfb952453cc00e275e1c82ca9787506aba0282145fff054498e60e19a65 glu-9.0.2.tar.gz SHA512: 5653ae7be7c580cbcd589061667e35b611bc7c43fccf7dd3f6c9c286a19052c1fbc8ade3ef435bda6e9ef6304248a12cc67c0603015692b5158ed4d6327be7d5 glu-9.0.2.tar.gz PGP: https://mesa.freedesktop.org/archive/glu/glu-9.0.2.tar.gz.sig https://mesa.freedesktop.org/archive/glu/glu-9.0.2.tar.xz SHA256: 6e7280ff585c6a1d9dfcdf2fca489251634b3377bfc33c29e4002466a38d02d4 glu-9.0.2.tar.xz SHA512: 2517d7406bb643d12c017a95dcb5d8716f307344332638bcbdf274a90752a7c22165d34745f1b082ed916bb07d40e62d1d1d67d96426225be63166f3480d6f64 glu-9.0.2.tar.xz PGP: https://mesa.freedesktop.org/archive/glu/glu-9.0.2.tar.xz.sig signature.asc Description: PGP signature ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [PATCH] dma-buf: Document dma-buf implicit fencing/resv fencing rules
On Thu, 24 Jun 2021 at 02:20, Daniel Vetter wrote: > > Docs for struct dma_resv are fairly clear: > > "A reservation object can have attached one exclusive fence (normally > associated with write operations) or N shared fences (read > operations)." > > https://dri.freedesktop.org/docs/drm/driver-api/dma-buf.html#reservation-objects > > Furthermore a review across all of upstream. > > First of render drivers and how they set implicit fences: > > - nouveau follows this contract, see in validate_fini_no_ticket() > > nouveau_bo_fence(nvbo, fence, !!b->write_domains); > > and that last boolean controls whether the exclusive or shared fence > slot is used. > > - radeon follows this contract by setting > > p->relocs[i].tv.num_shared = !r->write_domain; > > in radeon_cs_parser_relocs(), which ensures that the call to > ttm_eu_fence_buffer_objects() in radeon_cs_parser_fini() will do the > right thing. > > - vmwgfx seems to follow this contract with the shotgun approach of > always setting ttm_val_buf->num_shared = 0, which means > ttm_eu_fence_buffer_objects() will only use the exclusive slot. > > - etnaviv follows this contract, as can be trivially seen by looking > at submit_attach_object_fences() > > - i915 is a bit a convoluted maze with multiple paths leading to > i915_vma_move_to_active(). Which sets the exclusive flag if > EXEC_OBJECT_WRITE is set. This can either come as a buffer flag for > softpin mode, or through the write_domain when using relocations. It > follows this contract. > > - lima follows this contract, see lima_gem_submit() which sets the > exclusive fence when the LIMA_SUBMIT_BO_WRITE flag is set for that > bo > > - msm follows this contract, see msm_gpu_submit() which sets the > exclusive flag when the MSM_SUBMIT_BO_WRITE is set for that buffer > > - panfrost follows this contract with the shotgun approach of just > always setting the exclusive fence, see > panfrost_attach_object_fences(). Benefits of a single engine I guess > > - v3d follows this contract with the same shotgun approach in > v3d_attach_fences_and_unlock_reservation(), but it has at least an > XXX comment that maybe this should be improved > > - v4c uses the same shotgun approach of always setting an exclusive > fence, see vc4_update_bo_seqnos() > > - vgem also follows this contract, see vgem_fence_attach_ioctl() and > the VGEM_FENCE_WRITE. This is used in some igts to validate prime > sharing with i915.ko without the need of a 2nd gpu > > - vritio follows this contract again with the shotgun approach of > always setting an exclusive fence, see virtio_gpu_array_add_fence() > > This covers the setting of the exclusive fences when writing. > > Synchronizing against the exclusive fence is a lot more tricky, and I > only spot checked a few: > > - i915 does it, with the optional EXEC_OBJECT_ASYNC to skip all > implicit dependencies (which is used by vulkan) > > - etnaviv does this. Implicit dependencies are collected in > submit_fence_sync(), again with an opt-out flag > ETNA_SUBMIT_NO_IMPLICIT. These are then picked up in > etnaviv_sched_dependency which is the > drm_sched_backend_ops->dependency callback. > > - v4c seems to not do much here, maybe gets away with it by not having > a scheduler and only a single engine. Since all newer broadcom chips than > the OG vc4 use v3d for rendering, which follows this contract, the > impact of this issue is fairly small. > > - v3d does this using the drm_gem_fence_array_add_implicit() helper, > which then it's drm_sched_backend_ops->dependency callback > v3d_job_dependency() picks up. > > - panfrost is nice here and tracks the implicit fences in > panfrost_job->implicit_fences, which again the > drm_sched_backend_ops->dependency callback panfrost_job_dependency() > picks up. It is mildly questionable though since it only picks up > exclusive fences in panfrost_acquire_object_fences(), but not buggy > in practice because it also always sets the exclusive fence. It > should pick up both sets of fences, just in case there's ever going > to be a 2nd gpu in a SoC with a mali gpu. Or maybe a mali SoC with a > pcie port and a real gpu, which might actually happen eventually. A > bug, but easy to fix. Should probably use the > drm_gem_fence_array_add_implicit() helper. > > - lima is nice an easy, uses drm_gem_fence_array_add_implicit() and > the same schema as v3d. > > - msm is mildly entertaining. It also supports MSM_SUBMIT_NO_IMPLICIT, > but because it doesn't use the drm/scheduler it handles fences from > the wrong context with a synchronous dma_fence_wait. See > submit_fence_sync() leading to msm_gem_sync_object(). Investing into > a scheduler might be a good idea. > > - all the remaining drivers are ttm based, where I hope they do > appropriately obey implicit fences already. I didn't do the full > audit there because a) not follow the contract would
