Re: [PATCH] [Draft]: media: videobuf2-dma-heap: add a vendor defined memory runtine
Hi Randy, Sorry for the late reply, I went on vacation last week. On Sun, Aug 7, 2022 at 12:23 AM Hsia-Jun Li wrote: > > > > On 8/5/22 18:09, Tomasz Figa wrote: > > CAUTION: Email originated externally, do not click links or open > > attachments unless you recognize the sender and know the content is safe. > > > > > > On Tue, Aug 2, 2022 at 9:21 PM ayaka wrote: > >> > >> Sorry, the previous one contains html data. > >> > >>> On Aug 2, 2022, at 3:33 PM, Tomasz Figa wrote: > >>> > >>> On Mon, Aug 1, 2022 at 8:43 PM ayaka wrote: > Sent from my iPad > >> On Aug 1, 2022, at 5:46 PM, Tomasz Figa wrote: > > CAUTION: Email originated externally, do not click links or open > > attachments unless you recognize the sender and know the content is > > safe. > >> On Mon, Aug 1, 2022 at 3:44 PM Hsia-Jun Li > >> wrote: > >>> On 8/1/22 14:19, Tomasz Figa wrote: > >> Hello Tomasz > >>> ?Hi Randy, > >>> On Mon, Aug 1, 2022 at 5:21 AM wrote: > From: Randy Li > This module is still at a early stage, I wrote this for showing what > APIs we need here. > Let me explain why we need such a module here. > If you won't allocate buffers from a V4L2 M2M device, this module > may not be very useful. I am sure the most of users won't know a > device would require them allocate buffers from a DMA-Heap then > import those buffers into a V4L2's queue. > Then the question goes back to why DMA-Heap. From the Android's > description, we know it is about the copyright's DRM. > When we allocate a buffer in a DMA-Heap, it may register that buffer > in the trusted execution environment so the firmware which is running > or could only be acccesed from there could use that buffer later. > The answer above leads to another thing which is not done in this > version, the DMA mapping. Although in some platforms, a DMA-Heap > responses a IOMMU device as well. For the genernal purpose, we would > be better assuming the device mapping should be done for each device > itself. The problem here we only know alloc_devs in those DMAbuf > methods, which are DMA-heaps in my design, the device from the queue > is not enough, a plane may requests another IOMMU device or table > for mapping. > Signed-off-by: Randy Li > --- > drivers/media/common/videobuf2/Kconfig| 6 + > drivers/media/common/videobuf2/Makefile | 1 + > .../common/videobuf2/videobuf2-dma-heap.c | 350 > ++ > include/media/videobuf2-dma-heap.h| 30 ++ > 4 files changed, 387 insertions(+) > create mode 100644 > drivers/media/common/videobuf2/videobuf2-dma-heap.c > create mode 100644 include/media/videobuf2-dma-heap.h > >>> First of all, thanks for the series. > >>> Possibly a stupid question, but why not just allocate the DMA-bufs > >>> directly from the DMA-buf heap device in the userspace and just import > >>> the buffers to the V4L2 device using V4L2_MEMORY_DMABUF? > >> Sometimes the allocation policy could be very complex, let's suppose a > >> multiple planes pixel format enabling with frame buffer compression. > >> Its luma, chroma data could be allocated from a pool which is delegated > >> for large buffers while its metadata would come from a pool which many > >> users could take some few slices from it(likes system pool). > >> Then when we have a new users knowing nothing about this platform, if > >> we > >> just configure the alloc_devs in each queues well. The user won't need > >> to know those complex rules. > >> The real situation could be more complex, Samsung MFC's left and right > >> banks could be regarded as two pools, many devices would benefit from > >> this either from the allocation times or the security buffers policy. > >> In our design, when we need to do some security decoding(DRM video), > >> codecs2 would allocate buffers from the pool delegated for that. While > >> the non-DRM video, users could not care about this. > > I'm a little bit surprised about this, because on Android all the > > graphics buffers are allocated from the system IAllocator and imported > > to the specific devices. > In the non-tunnel mode, yes it is. While the tunnel mode is completely > vendor defined. Neither HWC nor codec2 cares about where the buffers > coming from, you could do what ever you want. > Besides there are DRM video in GNU Linux platform, I heard the webkit > has made huge effort here and Playready is one could work in non-Android > Linux. > > Would it make sense to instead extend the UAPI to expose enough > > information about the allocation requirements to the
Re: [PATCH] [Draft]: media: videobuf2-dma-heap: add a vendor defined memory runtine
On 8/18/22 13:50, Tomasz Figa wrote: CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Randy, Sorry for the late reply, I went on vacation last week. On Sun, Aug 7, 2022 at 12:23 AM Hsia-Jun Li wrote: On 8/5/22 18:09, Tomasz Figa wrote: CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. On Tue, Aug 2, 2022 at 9:21 PM ayaka wrote: Sorry, the previous one contains html data. On Aug 2, 2022, at 3:33 PM, Tomasz Figa wrote: On Mon, Aug 1, 2022 at 8:43 PM ayaka wrote: Sent from my iPad On Aug 1, 2022, at 5:46 PM, Tomasz Figa wrote: CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. On Mon, Aug 1, 2022 at 3:44 PM Hsia-Jun Li wrote: On 8/1/22 14:19, Tomasz Figa wrote: Hello Tomasz ?Hi Randy, On Mon, Aug 1, 2022 at 5:21 AM wrote: From: Randy Li This module is still at a early stage, I wrote this for showing what APIs we need here. Let me explain why we need such a module here. If you won't allocate buffers from a V4L2 M2M device, this module may not be very useful. I am sure the most of users won't know a device would require them allocate buffers from a DMA-Heap then import those buffers into a V4L2's queue. Then the question goes back to why DMA-Heap. From the Android's description, we know it is about the copyright's DRM. When we allocate a buffer in a DMA-Heap, it may register that buffer in the trusted execution environment so the firmware which is running or could only be acccesed from there could use that buffer later. The answer above leads to another thing which is not done in this version, the DMA mapping. Although in some platforms, a DMA-Heap responses a IOMMU device as well. For the genernal purpose, we would be better assuming the device mapping should be done for each device itself. The problem here we only know alloc_devs in those DMAbuf methods, which are DMA-heaps in my design, the device from the queue is not enough, a plane may requests another IOMMU device or table for mapping. Signed-off-by: Randy Li --- drivers/media/common/videobuf2/Kconfig| 6 + drivers/media/common/videobuf2/Makefile | 1 + .../common/videobuf2/videobuf2-dma-heap.c | 350 ++ include/media/videobuf2-dma-heap.h| 30 ++ 4 files changed, 387 insertions(+) create mode 100644 drivers/media/common/videobuf2/videobuf2-dma-heap.c create mode 100644 include/media/videobuf2-dma-heap.h First of all, thanks for the series. Possibly a stupid question, but why not just allocate the DMA-bufs directly from the DMA-buf heap device in the userspace and just import the buffers to the V4L2 device using V4L2_MEMORY_DMABUF? Sometimes the allocation policy could be very complex, let's suppose a multiple planes pixel format enabling with frame buffer compression. Its luma, chroma data could be allocated from a pool which is delegated for large buffers while its metadata would come from a pool which many users could take some few slices from it(likes system pool). Then when we have a new users knowing nothing about this platform, if we just configure the alloc_devs in each queues well. The user won't need to know those complex rules. The real situation could be more complex, Samsung MFC's left and right banks could be regarded as two pools, many devices would benefit from this either from the allocation times or the security buffers policy. In our design, when we need to do some security decoding(DRM video), codecs2 would allocate buffers from the pool delegated for that. While the non-DRM video, users could not care about this. I'm a little bit surprised about this, because on Android all the graphics buffers are allocated from the system IAllocator and imported to the specific devices. In the non-tunnel mode, yes it is. While the tunnel mode is completely vendor defined. Neither HWC nor codec2 cares about where the buffers coming from, you could do what ever you want. Besides there are DRM video in GNU Linux platform, I heard the webkit has made huge effort here and Playready is one could work in non-Android Linux. Would it make sense to instead extend the UAPI to expose enough information about the allocation requirements to the userspace, so it can allocate correctly? Yes, it could. But as I said it would need the users to do more works. My reasoning here is that it's not a driver's decision to allocate from a DMA-buf heap (and which one) or not. It's the userspace which knows that, based on the specific use case that it wants to fulfill. Although I would like to let the users decide that, users just can’t do that which would violate the security rules in some platforms. For example, video codec and display device could only access a region of memory, any other device or trusted
Re: [PATCH] [Draft]: media: videobuf2-dma-heap: add a vendor defined memory runtine
On 8/5/22 18:09, Tomasz Figa wrote: CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. On Tue, Aug 2, 2022 at 9:21 PM ayaka wrote: Sorry, the previous one contains html data. On Aug 2, 2022, at 3:33 PM, Tomasz Figa wrote: On Mon, Aug 1, 2022 at 8:43 PM ayaka wrote: Sent from my iPad On Aug 1, 2022, at 5:46 PM, Tomasz Figa wrote: CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. On Mon, Aug 1, 2022 at 3:44 PM Hsia-Jun Li wrote: On 8/1/22 14:19, Tomasz Figa wrote: Hello Tomasz ?Hi Randy, On Mon, Aug 1, 2022 at 5:21 AM wrote: From: Randy Li This module is still at a early stage, I wrote this for showing what APIs we need here. Let me explain why we need such a module here. If you won't allocate buffers from a V4L2 M2M device, this module may not be very useful. I am sure the most of users won't know a device would require them allocate buffers from a DMA-Heap then import those buffers into a V4L2's queue. Then the question goes back to why DMA-Heap. From the Android's description, we know it is about the copyright's DRM. When we allocate a buffer in a DMA-Heap, it may register that buffer in the trusted execution environment so the firmware which is running or could only be acccesed from there could use that buffer later. The answer above leads to another thing which is not done in this version, the DMA mapping. Although in some platforms, a DMA-Heap responses a IOMMU device as well. For the genernal purpose, we would be better assuming the device mapping should be done for each device itself. The problem here we only know alloc_devs in those DMAbuf methods, which are DMA-heaps in my design, the device from the queue is not enough, a plane may requests another IOMMU device or table for mapping. Signed-off-by: Randy Li --- drivers/media/common/videobuf2/Kconfig| 6 + drivers/media/common/videobuf2/Makefile | 1 + .../common/videobuf2/videobuf2-dma-heap.c | 350 ++ include/media/videobuf2-dma-heap.h| 30 ++ 4 files changed, 387 insertions(+) create mode 100644 drivers/media/common/videobuf2/videobuf2-dma-heap.c create mode 100644 include/media/videobuf2-dma-heap.h First of all, thanks for the series. Possibly a stupid question, but why not just allocate the DMA-bufs directly from the DMA-buf heap device in the userspace and just import the buffers to the V4L2 device using V4L2_MEMORY_DMABUF? Sometimes the allocation policy could be very complex, let's suppose a multiple planes pixel format enabling with frame buffer compression. Its luma, chroma data could be allocated from a pool which is delegated for large buffers while its metadata would come from a pool which many users could take some few slices from it(likes system pool). Then when we have a new users knowing nothing about this platform, if we just configure the alloc_devs in each queues well. The user won't need to know those complex rules. The real situation could be more complex, Samsung MFC's left and right banks could be regarded as two pools, many devices would benefit from this either from the allocation times or the security buffers policy. In our design, when we need to do some security decoding(DRM video), codecs2 would allocate buffers from the pool delegated for that. While the non-DRM video, users could not care about this. I'm a little bit surprised about this, because on Android all the graphics buffers are allocated from the system IAllocator and imported to the specific devices. In the non-tunnel mode, yes it is. While the tunnel mode is completely vendor defined. Neither HWC nor codec2 cares about where the buffers coming from, you could do what ever you want. Besides there are DRM video in GNU Linux platform, I heard the webkit has made huge effort here and Playready is one could work in non-Android Linux. Would it make sense to instead extend the UAPI to expose enough information about the allocation requirements to the userspace, so it can allocate correctly? Yes, it could. But as I said it would need the users to do more works. My reasoning here is that it's not a driver's decision to allocate from a DMA-buf heap (and which one) or not. It's the userspace which knows that, based on the specific use case that it wants to fulfill. Although I would like to let the users decide that, users just can’t do that which would violate the security rules in some platforms. For example, video codec and display device could only access a region of memory, any other device or trusted apps can’t access it. Users have to allocate the buffer from the pool the vendor decided. So why not we offer a quick way that users don’t need to try and error. In principle, I'm not against integrating DMA-buf heap with vb2, however I see some problems I mentioned before: 1) How would the driver
Re: [PATCH] [Draft]: media: videobuf2-dma-heap: add a vendor defined memory runtine
On Tue, Aug 2, 2022 at 9:21 PM ayaka wrote: > > Sorry, the previous one contains html data. > > > On Aug 2, 2022, at 3:33 PM, Tomasz Figa wrote: > > > > On Mon, Aug 1, 2022 at 8:43 PM ayaka wrote: > >> Sent from my iPad > On Aug 1, 2022, at 5:46 PM, Tomasz Figa wrote: > >>> CAUTION: Email originated externally, do not click links or open > >>> attachments unless you recognize the sender and know the content is safe. > On Mon, Aug 1, 2022 at 3:44 PM Hsia-Jun Li > wrote: > > On 8/1/22 14:19, Tomasz Figa wrote: > Hello Tomasz > > ?Hi Randy, > > On Mon, Aug 1, 2022 at 5:21 AM wrote: > >> From: Randy Li > >> This module is still at a early stage, I wrote this for showing what > >> APIs we need here. > >> Let me explain why we need such a module here. > >> If you won't allocate buffers from a V4L2 M2M device, this module > >> may not be very useful. I am sure the most of users won't know a > >> device would require them allocate buffers from a DMA-Heap then > >> import those buffers into a V4L2's queue. > >> Then the question goes back to why DMA-Heap. From the Android's > >> description, we know it is about the copyright's DRM. > >> When we allocate a buffer in a DMA-Heap, it may register that buffer > >> in the trusted execution environment so the firmware which is running > >> or could only be acccesed from there could use that buffer later. > >> The answer above leads to another thing which is not done in this > >> version, the DMA mapping. Although in some platforms, a DMA-Heap > >> responses a IOMMU device as well. For the genernal purpose, we would > >> be better assuming the device mapping should be done for each device > >> itself. The problem here we only know alloc_devs in those DMAbuf > >> methods, which are DMA-heaps in my design, the device from the queue > >> is not enough, a plane may requests another IOMMU device or table > >> for mapping. > >> Signed-off-by: Randy Li > >> --- > >> drivers/media/common/videobuf2/Kconfig| 6 + > >> drivers/media/common/videobuf2/Makefile | 1 + > >> .../common/videobuf2/videobuf2-dma-heap.c | 350 ++ > >> include/media/videobuf2-dma-heap.h| 30 ++ > >> 4 files changed, 387 insertions(+) > >> create mode 100644 drivers/media/common/videobuf2/videobuf2-dma-heap.c > >> create mode 100644 include/media/videobuf2-dma-heap.h > > First of all, thanks for the series. > > Possibly a stupid question, but why not just allocate the DMA-bufs > > directly from the DMA-buf heap device in the userspace and just import > > the buffers to the V4L2 device using V4L2_MEMORY_DMABUF? > Sometimes the allocation policy could be very complex, let's suppose a > multiple planes pixel format enabling with frame buffer compression. > Its luma, chroma data could be allocated from a pool which is delegated > for large buffers while its metadata would come from a pool which many > users could take some few slices from it(likes system pool). > Then when we have a new users knowing nothing about this platform, if we > just configure the alloc_devs in each queues well. The user won't need > to know those complex rules. > The real situation could be more complex, Samsung MFC's left and right > banks could be regarded as two pools, many devices would benefit from > this either from the allocation times or the security buffers policy. > In our design, when we need to do some security decoding(DRM video), > codecs2 would allocate buffers from the pool delegated for that. While > the non-DRM video, users could not care about this. > >>> I'm a little bit surprised about this, because on Android all the > >>> graphics buffers are allocated from the system IAllocator and imported > >>> to the specific devices. > >> In the non-tunnel mode, yes it is. While the tunnel mode is completely > >> vendor defined. Neither HWC nor codec2 cares about where the buffers > >> coming from, you could do what ever you want. > >> Besides there are DRM video in GNU Linux platform, I heard the webkit has > >> made huge effort here and Playready is one could work in non-Android Linux. > >>> Would it make sense to instead extend the UAPI to expose enough > >>> information about the allocation requirements to the userspace, so it > >>> can allocate correctly? > >> Yes, it could. But as I said it would need the users to do more works. > >>> My reasoning here is that it's not a driver's decision to allocate > >>> from a DMA-buf heap (and which one) or not. It's the userspace which > >>> knows that, based on the specific use case that it wants to fulfill. > >> Although I would like to let the users decide that, users just can’t do > >> that which would violate the security rules in some platforms. > >> For example, video codec and
Re: [PATCH] [Draft]: media: videobuf2-dma-heap: add a vendor defined memory runtine
Sorry, the previous one contains html data. > On Aug 2, 2022, at 3:33 PM, Tomasz Figa wrote: > > On Mon, Aug 1, 2022 at 8:43 PM ayaka wrote: >> Sent from my iPad On Aug 1, 2022, at 5:46 PM, Tomasz Figa wrote: >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is safe. On Mon, Aug 1, 2022 at 3:44 PM Hsia-Jun Li wrote: > On 8/1/22 14:19, Tomasz Figa wrote: Hello Tomasz > ?Hi Randy, > On Mon, Aug 1, 2022 at 5:21 AM wrote: >> From: Randy Li >> This module is still at a early stage, I wrote this for showing what >> APIs we need here. >> Let me explain why we need such a module here. >> If you won't allocate buffers from a V4L2 M2M device, this module >> may not be very useful. I am sure the most of users won't know a >> device would require them allocate buffers from a DMA-Heap then >> import those buffers into a V4L2's queue. >> Then the question goes back to why DMA-Heap. From the Android's >> description, we know it is about the copyright's DRM. >> When we allocate a buffer in a DMA-Heap, it may register that buffer >> in the trusted execution environment so the firmware which is running >> or could only be acccesed from there could use that buffer later. >> The answer above leads to another thing which is not done in this >> version, the DMA mapping. Although in some platforms, a DMA-Heap >> responses a IOMMU device as well. For the genernal purpose, we would >> be better assuming the device mapping should be done for each device >> itself. The problem here we only know alloc_devs in those DMAbuf >> methods, which are DMA-heaps in my design, the device from the queue >> is not enough, a plane may requests another IOMMU device or table >> for mapping. >> Signed-off-by: Randy Li >> --- >> drivers/media/common/videobuf2/Kconfig| 6 + >> drivers/media/common/videobuf2/Makefile | 1 + >> .../common/videobuf2/videobuf2-dma-heap.c | 350 ++ >> include/media/videobuf2-dma-heap.h| 30 ++ >> 4 files changed, 387 insertions(+) >> create mode 100644 drivers/media/common/videobuf2/videobuf2-dma-heap.c >> create mode 100644 include/media/videobuf2-dma-heap.h > First of all, thanks for the series. > Possibly a stupid question, but why not just allocate the DMA-bufs > directly from the DMA-buf heap device in the userspace and just import > the buffers to the V4L2 device using V4L2_MEMORY_DMABUF? Sometimes the allocation policy could be very complex, let's suppose a multiple planes pixel format enabling with frame buffer compression. Its luma, chroma data could be allocated from a pool which is delegated for large buffers while its metadata would come from a pool which many users could take some few slices from it(likes system pool). Then when we have a new users knowing nothing about this platform, if we just configure the alloc_devs in each queues well. The user won't need to know those complex rules. The real situation could be more complex, Samsung MFC's left and right banks could be regarded as two pools, many devices would benefit from this either from the allocation times or the security buffers policy. In our design, when we need to do some security decoding(DRM video), codecs2 would allocate buffers from the pool delegated for that. While the non-DRM video, users could not care about this. >>> I'm a little bit surprised about this, because on Android all the >>> graphics buffers are allocated from the system IAllocator and imported >>> to the specific devices. >> In the non-tunnel mode, yes it is. While the tunnel mode is completely >> vendor defined. Neither HWC nor codec2 cares about where the buffers coming >> from, you could do what ever you want. >> Besides there are DRM video in GNU Linux platform, I heard the webkit has >> made huge effort here and Playready is one could work in non-Android Linux. >>> Would it make sense to instead extend the UAPI to expose enough >>> information about the allocation requirements to the userspace, so it >>> can allocate correctly? >> Yes, it could. But as I said it would need the users to do more works. >>> My reasoning here is that it's not a driver's decision to allocate >>> from a DMA-buf heap (and which one) or not. It's the userspace which >>> knows that, based on the specific use case that it wants to fulfill. >> Although I would like to let the users decide that, users just can’t do that >> which would violate the security rules in some platforms. >> For example, video codec and display device could only access a region of >> memory, any other device or trusted apps can’t access it. Users have to >> allocate the buffer from the pool the vendor decided. >> So why not we offer a quick way
Re: [PATCH] [Draft]: media: videobuf2-dma-heap: add a vendor defined memory runtine
Sent from my iPad > On Aug 2, 2022, at 3:33 PM, Tomasz Figa wrote: > > On Mon, Aug 1, 2022 at 8:43 PM ayaka wrote: >> >> >> >> Sent from my iPad >> On Aug 1, 2022, at 5:46 PM, Tomasz Figa wrote: >>> >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is safe. >>> >>> On Mon, Aug 1, 2022 at 3:44 PM Hsia-Jun Li wrote: > On 8/1/22 14:19, Tomasz Figa wrote: Hello Tomasz > ?Hi Randy, > On Mon, Aug 1, 2022 at 5:21 AM wrote: >> From: Randy Li >> This module is still at a early stage, I wrote this for showing what >> APIs we need here. >> Let me explain why we need such a module here. >> If you won't allocate buffers from a V4L2 M2M device, this module >> may not be very useful. I am sure the most of users won't know a >> device would require them allocate buffers from a DMA-Heap then >> import those buffers into a V4L2's queue. >> Then the question goes back to why DMA-Heap. From the Android's >> description, we know it is about the copyright's DRM. >> When we allocate a buffer in a DMA-Heap, it may register that buffer >> in the trusted execution environment so the firmware which is running >> or could only be acccesed from there could use that buffer later. >> The answer above leads to another thing which is not done in this >> version, the DMA mapping. Although in some platforms, a DMA-Heap >> responses a IOMMU device as well. For the genernal purpose, we would >> be better assuming the device mapping should be done for each device >> itself. The problem here we only know alloc_devs in those DMAbuf >> methods, which are DMA-heaps in my design, the device from the queue >> is not enough, a plane may requests another IOMMU device or table >> for mapping. >> Signed-off-by: Randy Li >> --- >> drivers/media/common/videobuf2/Kconfig| 6 + >> drivers/media/common/videobuf2/Makefile | 1 + >> .../common/videobuf2/videobuf2-dma-heap.c | 350 ++ >> include/media/videobuf2-dma-heap.h| 30 ++ >> 4 files changed, 387 insertions(+) >> create mode 100644 drivers/media/common/videobuf2/videobuf2-dma-heap.c >> create mode 100644 include/media/videobuf2-dma-heap.h > First of all, thanks for the series. > Possibly a stupid question, but why not just allocate the DMA-bufs > directly from the DMA-buf heap device in the userspace and just import > the buffers to the V4L2 device using V4L2_MEMORY_DMABUF? Sometimes the allocation policy could be very complex, let's suppose a multiple planes pixel format enabling with frame buffer compression. Its luma, chroma data could be allocated from a pool which is delegated for large buffers while its metadata would come from a pool which many users could take some few slices from it(likes system pool). Then when we have a new users knowing nothing about this platform, if we just configure the alloc_devs in each queues well. The user won't need to know those complex rules. The real situation could be more complex, Samsung MFC's left and right banks could be regarded as two pools, many devices would benefit from this either from the allocation times or the security buffers policy. In our design, when we need to do some security decoding(DRM video), codecs2 would allocate buffers from the pool delegated for that. While the non-DRM video, users could not care about this. >>> >>> I'm a little bit surprised about this, because on Android all the >>> graphics buffers are allocated from the system IAllocator and imported >>> to the specific devices. >> In the non-tunnel mode, yes it is. While the tunnel mode is completely >> vendor defined. Neither HWC nor codec2 cares about where the buffers coming >> from, you could do what ever you want. >> >> Besides there are DRM video in GNU Linux platform, I heard the webkit has >> made huge effort here and Playready is one could work in non-Android Linux. >>> Would it make sense to instead extend the UAPI to expose enough >>> information about the allocation requirements to the userspace, so it >>> can allocate correctly? >> Yes, it could. But as I said it would need the users to do more works. >>> My reasoning here is that it's not a driver's decision to allocate >>> from a DMA-buf heap (and which one) or not. It's the userspace which >>> knows that, based on the specific use case that it wants to fulfill. >> Although I would like to let the users decide that, users just can’t do that >> which would violate the security rules in some platforms. >> For example, video codec and display device could only access a region of >> memory, any other device or trusted apps can’t access it. Users have to >> allocate the buffer from the pool the vendor decided. >> >> So why not we
Re: [PATCH] [Draft]: media: videobuf2-dma-heap: add a vendor defined memory runtine
On Mon, Aug 1, 2022 at 8:43 PM ayaka wrote: > > > > Sent from my iPad > > > On Aug 1, 2022, at 5:46 PM, Tomasz Figa wrote: > > > > CAUTION: Email originated externally, do not click links or open > > attachments unless you recognize the sender and know the content is safe. > > > > > >> On Mon, Aug 1, 2022 at 3:44 PM Hsia-Jun Li wrote: > >>> On 8/1/22 14:19, Tomasz Figa wrote: > >> Hello Tomasz > >>> ?Hi Randy, > >>> On Mon, Aug 1, 2022 at 5:21 AM wrote: > From: Randy Li > This module is still at a early stage, I wrote this for showing what > APIs we need here. > Let me explain why we need such a module here. > If you won't allocate buffers from a V4L2 M2M device, this module > may not be very useful. I am sure the most of users won't know a > device would require them allocate buffers from a DMA-Heap then > import those buffers into a V4L2's queue. > Then the question goes back to why DMA-Heap. From the Android's > description, we know it is about the copyright's DRM. > When we allocate a buffer in a DMA-Heap, it may register that buffer > in the trusted execution environment so the firmware which is running > or could only be acccesed from there could use that buffer later. > The answer above leads to another thing which is not done in this > version, the DMA mapping. Although in some platforms, a DMA-Heap > responses a IOMMU device as well. For the genernal purpose, we would > be better assuming the device mapping should be done for each device > itself. The problem here we only know alloc_devs in those DMAbuf > methods, which are DMA-heaps in my design, the device from the queue > is not enough, a plane may requests another IOMMU device or table > for mapping. > Signed-off-by: Randy Li > --- > drivers/media/common/videobuf2/Kconfig| 6 + > drivers/media/common/videobuf2/Makefile | 1 + > .../common/videobuf2/videobuf2-dma-heap.c | 350 ++ > include/media/videobuf2-dma-heap.h| 30 ++ > 4 files changed, 387 insertions(+) > create mode 100644 drivers/media/common/videobuf2/videobuf2-dma-heap.c > create mode 100644 include/media/videobuf2-dma-heap.h > >>> First of all, thanks for the series. > >>> Possibly a stupid question, but why not just allocate the DMA-bufs > >>> directly from the DMA-buf heap device in the userspace and just import > >>> the buffers to the V4L2 device using V4L2_MEMORY_DMABUF? > >> Sometimes the allocation policy could be very complex, let's suppose a > >> multiple planes pixel format enabling with frame buffer compression. > >> Its luma, chroma data could be allocated from a pool which is delegated > >> for large buffers while its metadata would come from a pool which many > >> users could take some few slices from it(likes system pool). > >> Then when we have a new users knowing nothing about this platform, if we > >> just configure the alloc_devs in each queues well. The user won't need > >> to know those complex rules. > >> The real situation could be more complex, Samsung MFC's left and right > >> banks could be regarded as two pools, many devices would benefit from > >> this either from the allocation times or the security buffers policy. > >> In our design, when we need to do some security decoding(DRM video), > >> codecs2 would allocate buffers from the pool delegated for that. While > >> the non-DRM video, users could not care about this. > > > > I'm a little bit surprised about this, because on Android all the > > graphics buffers are allocated from the system IAllocator and imported > > to the specific devices. > In the non-tunnel mode, yes it is. While the tunnel mode is completely vendor > defined. Neither HWC nor codec2 cares about where the buffers coming from, > you could do what ever you want. > > Besides there are DRM video in GNU Linux platform, I heard the webkit has > made huge effort here and Playready is one could work in non-Android Linux. > > Would it make sense to instead extend the UAPI to expose enough > > information about the allocation requirements to the userspace, so it > > can allocate correctly? > Yes, it could. But as I said it would need the users to do more works. > > My reasoning here is that it's not a driver's decision to allocate > > from a DMA-buf heap (and which one) or not. It's the userspace which > > knows that, based on the specific use case that it wants to fulfill. > Although I would like to let the users decide that, users just can’t do that > which would violate the security rules in some platforms. > For example, video codec and display device could only access a region of > memory, any other device or trusted apps can’t access it. Users have to > allocate the buffer from the pool the vendor decided. > > So why not we offer a quick way that users don’t need to try and error. In principle, I'm not against integrating
Re: [PATCH] [Draft]: media: videobuf2-dma-heap: add a vendor defined memory runtine
Sent from my iPad > On Aug 1, 2022, at 5:46 PM, Tomasz Figa wrote: > > CAUTION: Email originated externally, do not click links or open attachments > unless you recognize the sender and know the content is safe. > > >> On Mon, Aug 1, 2022 at 3:44 PM Hsia-Jun Li wrote: >> >> >> >>> On 8/1/22 14:19, Tomasz Figa wrote: >>> >> Hello Tomasz >>> >>> ?Hi Randy, >>> >>> On Mon, Aug 1, 2022 at 5:21 AM wrote: From: Randy Li This module is still at a early stage, I wrote this for showing what APIs we need here. Let me explain why we need such a module here. If you won't allocate buffers from a V4L2 M2M device, this module may not be very useful. I am sure the most of users won't know a device would require them allocate buffers from a DMA-Heap then import those buffers into a V4L2's queue. Then the question goes back to why DMA-Heap. From the Android's description, we know it is about the copyright's DRM. When we allocate a buffer in a DMA-Heap, it may register that buffer in the trusted execution environment so the firmware which is running or could only be acccesed from there could use that buffer later. The answer above leads to another thing which is not done in this version, the DMA mapping. Although in some platforms, a DMA-Heap responses a IOMMU device as well. For the genernal purpose, we would be better assuming the device mapping should be done for each device itself. The problem here we only know alloc_devs in those DMAbuf methods, which are DMA-heaps in my design, the device from the queue is not enough, a plane may requests another IOMMU device or table for mapping. Signed-off-by: Randy Li --- drivers/media/common/videobuf2/Kconfig| 6 + drivers/media/common/videobuf2/Makefile | 1 + .../common/videobuf2/videobuf2-dma-heap.c | 350 ++ include/media/videobuf2-dma-heap.h| 30 ++ 4 files changed, 387 insertions(+) create mode 100644 drivers/media/common/videobuf2/videobuf2-dma-heap.c create mode 100644 include/media/videobuf2-dma-heap.h >>> >>> First of all, thanks for the series. >>> >>> Possibly a stupid question, but why not just allocate the DMA-bufs >>> directly from the DMA-buf heap device in the userspace and just import >>> the buffers to the V4L2 device using V4L2_MEMORY_DMABUF? >> Sometimes the allocation policy could be very complex, let's suppose a >> multiple planes pixel format enabling with frame buffer compression. >> Its luma, chroma data could be allocated from a pool which is delegated >> for large buffers while its metadata would come from a pool which many >> users could take some few slices from it(likes system pool). >> >> Then when we have a new users knowing nothing about this platform, if we >> just configure the alloc_devs in each queues well. The user won't need >> to know those complex rules. >> >> The real situation could be more complex, Samsung MFC's left and right >> banks could be regarded as two pools, many devices would benefit from >> this either from the allocation times or the security buffers policy. >> >> In our design, when we need to do some security decoding(DRM video), >> codecs2 would allocate buffers from the pool delegated for that. While >> the non-DRM video, users could not care about this. > > I'm a little bit surprised about this, because on Android all the > graphics buffers are allocated from the system IAllocator and imported > to the specific devices. > In the non-tunnel mode, yes it is. While the tunnel mode is completely vendor defined. Neither HWC nor codec2 cares about where the buffers coming from, you could do what ever you want. Besides there are DRM video in GNU Linux platform, I heard the webkit has made huge effort here and Playready is one could work in non-Android Linux. > Would it make sense to instead extend the UAPI to expose enough > information about the allocation requirements to the userspace, so it > can allocate correctly? Yes, it could. But as I said it would need the users to do more works. > My reasoning here is that it's not a driver's decision to allocate > from a DMA-buf heap (and which one) or not. It's the userspace which > knows that, based on the specific use case that it wants to fulfill. > Although I would like to let the users decide that, users just can’t do that which would violate the security rules in some platforms. For example, video codec and display device could only access a region of memory, any other device or trusted apps can’t access it. Users have to allocate the buffer from the pool the vendor decided. So why not we offer a quick way that users don’t need to try and error. > Also, FWIW, dma_heap_ioctl_allocate() is a static function not exposed > to other kernel modules: >
Re: [PATCH] [Draft]: media: videobuf2-dma-heap: add a vendor defined memory runtine
Sent from my iPad > On Aug 1, 2022, at 5:46 PM, Tomasz Figa wrote: > > CAUTION: Email originated externally, do not click links or open attachments > unless you recognize the sender and know the content is safe. > > >> On Mon, Aug 1, 2022 at 3:44 PM Hsia-Jun Li wrote: >>> On 8/1/22 14:19, Tomasz Figa wrote: >> Hello Tomasz >>> ?Hi Randy, >>> On Mon, Aug 1, 2022 at 5:21 AM wrote: From: Randy Li This module is still at a early stage, I wrote this for showing what APIs we need here. Let me explain why we need such a module here. If you won't allocate buffers from a V4L2 M2M device, this module may not be very useful. I am sure the most of users won't know a device would require them allocate buffers from a DMA-Heap then import those buffers into a V4L2's queue. Then the question goes back to why DMA-Heap. From the Android's description, we know it is about the copyright's DRM. When we allocate a buffer in a DMA-Heap, it may register that buffer in the trusted execution environment so the firmware which is running or could only be acccesed from there could use that buffer later. The answer above leads to another thing which is not done in this version, the DMA mapping. Although in some platforms, a DMA-Heap responses a IOMMU device as well. For the genernal purpose, we would be better assuming the device mapping should be done for each device itself. The problem here we only know alloc_devs in those DMAbuf methods, which are DMA-heaps in my design, the device from the queue is not enough, a plane may requests another IOMMU device or table for mapping. Signed-off-by: Randy Li --- drivers/media/common/videobuf2/Kconfig| 6 + drivers/media/common/videobuf2/Makefile | 1 + .../common/videobuf2/videobuf2-dma-heap.c | 350 ++ include/media/videobuf2-dma-heap.h| 30 ++ 4 files changed, 387 insertions(+) create mode 100644 drivers/media/common/videobuf2/videobuf2-dma-heap.c create mode 100644 include/media/videobuf2-dma-heap.h >>> First of all, thanks for the series. >>> Possibly a stupid question, but why not just allocate the DMA-bufs >>> directly from the DMA-buf heap device in the userspace and just import >>> the buffers to the V4L2 device using V4L2_MEMORY_DMABUF? >> Sometimes the allocation policy could be very complex, let's suppose a >> multiple planes pixel format enabling with frame buffer compression. >> Its luma, chroma data could be allocated from a pool which is delegated >> for large buffers while its metadata would come from a pool which many >> users could take some few slices from it(likes system pool). >> Then when we have a new users knowing nothing about this platform, if we >> just configure the alloc_devs in each queues well. The user won't need >> to know those complex rules. >> The real situation could be more complex, Samsung MFC's left and right >> banks could be regarded as two pools, many devices would benefit from >> this either from the allocation times or the security buffers policy. >> In our design, when we need to do some security decoding(DRM video), >> codecs2 would allocate buffers from the pool delegated for that. While >> the non-DRM video, users could not care about this. > > I'm a little bit surprised about this, because on Android all the > graphics buffers are allocated from the system IAllocator and imported > to the specific devices. In the non-tunnel mode, yes it is. While the tunnel mode is completely vendor defined. Neither HWC nor codec2 cares about where the buffers coming from, you could do what ever you want. Besides there are DRM video in GNU Linux platform, I heard the webkit has made huge effort here and Playready is one could work in non-Android Linux. > Would it make sense to instead extend the UAPI to expose enough > information about the allocation requirements to the userspace, so it > can allocate correctly? Yes, it could. But as I said it would need the users to do more works. > My reasoning here is that it's not a driver's decision to allocate > from a DMA-buf heap (and which one) or not. It's the userspace which > knows that, based on the specific use case that it wants to fulfill. Although I would like to let the users decide that, users just can’t do that which would violate the security rules in some platforms. For example, video codec and display device could only access a region of memory, any other device or trusted apps can’t access it. Users have to allocate the buffer from the pool the vendor decided. So why not we offer a quick way that users don’t need to try and error. > Also, FWIW, dma_heap_ioctl_allocate() is a static function not exposed > to other kernel modules: >
