Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg
On Wed, 08 Feb 2023 19:38:07 +0200 Rémi Denis-Courmont wrote: > Le keskiviikkona 8. helmikuuta 2023, 16.27.52 EET Devin Heitmueller a > écrit : > > On Wed, Feb 8, 2023 at 8:08 AM Jean-Baptiste Kempf > > wrote: > > > Do we really need those PIX_FMT in FFmpeg public API? > > > > Tiled formats are annoying, but not uncommon in popular embedded > > platforms (e.g. NXP iMX, TI Davinci). > > Uh, not to deny that, but tiled formats are in no way an embedded > thing, as you imply. They are also used by desktop GPUs. And that is > not really relevant, I would argue, to the discussion, either way. > > > Of course ffmpeg could choose > > to ignore them, but it would effectively prevent it from being used > > on those platforms (and it's pretty essential to use the hardware > > blocks to do any real video processing). > > Insofar as the format is *only* used by a single module, I don't see > the need to assign it a pixel format in libavutil. You could just as > well define a generic V4L format that would be used for all weird V4L > formats that nothing other than V4L code understands. > > Ditto tiled DRM pixel formats for that matter. And while V4L is maybe > not there yet, DRM has introduced format modifiers that anyway > preclude any simple enumeration. My point being, there needs to be a > way to further specify framework-specific formats outside of the > avutil pixel format list, in any case. > I never thought about it this way until you mentioned DRM but I think that's the key insight. If we want to take a leaf out of the DRM book (and wouldn't it be wonderful if these m2m decoders used DRM formats?), then this format would NV12 at the pix fmt layer with a modifier indicating the tiling pattern. When we interact with these kinds of formats in VAAPI, for example, VAAPI encapsulates the heavy lifting required to transfer to/from system memory successfully, and when we are doing everything on the GPU side, we pass the surface long with the modifier to whatever eventually consumes it (eg: Display with GL or Vulkan, or encoding). In none of these cases does ffmpeg code really need to care about the modifier. Maybe we've been thinking about these m2m formats the wrong way. --phil ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg
Hi Remi, On Wed, Feb 8, 2023 at 12:38 PM Rémi Denis-Courmont wrote: > Uh, not to deny that, but tiled formats are in no way an embedded thing, as > you imply. They are also used by desktop GPUs. And that is not really > relevant, I would argue, to the discussion, either way. I don't disagree that tiled formats can be found on desktop GPUs. My only point was that the importance of being able to pass through the native format is much greater with embedded SOCs since in many cases on desktops you have enough CPU power to get away with doing a conversion. Also, you're more likely to find that desktop GPUs which support tiled formats also support formats that are likely to be supported by ffmpeg today. This is less likely to be the case with embedded SOCs. > > Of course ffmpeg could choose > > to ignore them, but it would effectively prevent it from being used on > > those platforms (and it's pretty essential to use the hardware blocks > > to do any real video processing). > > Insofar as the format is *only* used by a single module, I don't see the need > to assign it a pixel format in libavutil. You could just as well define a > generic V4L format that would be used for all weird V4L formats that nothing > other than V4L code understands. In many cases the lack of support for a pixel format may be the reason that more platforms aren't supported. Somebody has got to be the first. Suggesting that it's only used by a single module may be the effect of not supporting the pixel format, not the other way around. This is not to suggest that like most developers I don't hate the fact that there are so many different pixel formats out there, and with each new format proposed for addition to ffmpeg we have the same arguments. > Ditto tiled DRM pixel formats for that matter. And while V4L is maybe not > there yet, DRM has introduced format modifiers that anyway preclude any simple > enumeration. My point being, there needs to be a way to further specify > framework-specific formats outside of the avutil pixel format list, in any > case. I can't really offer an opinion on the implementation detail regarding the avutil pixel format list. I suspect the OP is open to alternative approaches as long as the original requirement is met. Devin -- Devin Heitmueller, Senior Software Engineer LTN Global Communications o: +1 (301) 363-1001 w: https://ltnglobal.com e: devin.heitmuel...@ltnglobal.com ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg
Le keskiviikkona 8. helmikuuta 2023, 16.27.52 EET Devin Heitmueller a écrit : > On Wed, Feb 8, 2023 at 8:08 AM Jean-Baptiste Kempf wrote: > > Do we really need those PIX_FMT in FFmpeg public API? > > Tiled formats are annoying, but not uncommon in popular embedded > platforms (e.g. NXP iMX, TI Davinci). Uh, not to deny that, but tiled formats are in no way an embedded thing, as you imply. They are also used by desktop GPUs. And that is not really relevant, I would argue, to the discussion, either way. > Of course ffmpeg could choose > to ignore them, but it would effectively prevent it from being used on > those platforms (and it's pretty essential to use the hardware blocks > to do any real video processing). Insofar as the format is *only* used by a single module, I don't see the need to assign it a pixel format in libavutil. You could just as well define a generic V4L format that would be used for all weird V4L formats that nothing other than V4L code understands. Ditto tiled DRM pixel formats for that matter. And while V4L is maybe not there yet, DRM has introduced format modifiers that anyway preclude any simple enumeration. My point being, there needs to be a way to further specify framework-specific formats outside of the avutil pixel format list, in any case. -- レミ・デニ-クールモン http://www.remlab.net/ ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg
On Wed, Feb 8, 2023 at 8:08 AM Jean-Baptiste Kempf wrote: > Do we really need those PIX_FMT in FFmpeg public API? Tiled formats are annoying, but not uncommon in popular embedded platforms (e.g. NXP iMX, TI Davinci). Of course ffmpeg could choose to ignore them, but it would effectively prevent it from being used on those platforms (and it's pretty essential to use the hardware blocks to do any real video processing). In the past there was an expectation that adding new pixel formats required the contributor to also add corresponding routines to swscale to convert to more common formats that ffmpeg works with. You could alternatively treat it as a packet format and require a decoder to do the conversion to something more common to ffmpeg, but this would both be expensive (in terms of CPU usage and memory copying) and effectively prevent you from being able to capture or decode in the native format and then hand off those buffers to other hardware blocks like encoders, colorspace conversion, etc. Devin -- Devin Heitmueller, Senior Software Engineer LTN Global Communications o: +1 (301) 363-1001 w: https://ltnglobal.com e: devin.heitmuel...@ltnglobal.com ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg
On Wed, Feb 8, 2023 at 2:08 PM Jean-Baptiste Kempf wrote: > > On Wed, 8 Feb 2023, at 10:47, Le Bao Tin Ha wrote: > > Hello, you can found the explanation here: > > > > https://docs.kernel.org/userspace-api/media/v4l/pixfmt-yuv-planar.html > > > > "V4L2_PIX_FMT_NV12M_8L128 is similar to V4L2_PIX_FMT_NV12M but stores > > pixels in 2D 8x128 tiles, and stores tiles linearly in memory. The > > image height must be aligned to a multiple of 128. The layouts of the > > luma and chroma planes are identical. > > > > V4L2_PIX_FMT_NV12_8L128 is similar to V4L2_PIX_FMT_NV12M_8L128 but > > stores two planes in one memory." > > Do we really need those PIX_FMT in FFmpeg public API? > Tiled formats also really don't fit into our format definitions, and would not work with any generic pixel format handling code, as far as I can tell. - Hendrik ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg
On Wed, 8 Feb 2023, at 10:47, Le Bao Tin Ha wrote: > Hello, you can found the explanation here: > > https://docs.kernel.org/userspace-api/media/v4l/pixfmt-yuv-planar.html > > "V4L2_PIX_FMT_NV12M_8L128 is similar to V4L2_PIX_FMT_NV12M but stores > pixels in 2D 8x128 tiles, and stores tiles linearly in memory. The > image height must be aligned to a multiple of 128. The layouts of the > luma and chroma planes are identical. > > V4L2_PIX_FMT_NV12_8L128 is similar to V4L2_PIX_FMT_NV12M_8L128 but > stores two planes in one memory." Do we really need those PIX_FMT in FFmpeg public API? -- Jean-Baptiste Kempf - President +33 672 704 734 ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg
Hello, you can found the explanation here: https://docs.kernel.org/userspace-api/media/v4l/pixfmt-yuv-planar.html "V4L2_PIX_FMT_NV12M_8L128 is similar to V4L2_PIX_FMT_NV12M but stores pixels in 2D 8x128 tiles, and stores tiles linearly in memory. The image height must be aligned to a multiple of 128. The layouts of the luma and chroma planes are identical. V4L2_PIX_FMT_NV12_8L128 is similar to V4L2_PIX_FMT_NV12M_8L128 but stores two planes in one memory." - Original Message - From: "Kieran Kunhya" To: "FFmpeg development discussions and patches" Sent: Monday, February 6, 2023 7:15:22 PM Subject: Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg On Mon, 6 Feb 2023 at 15:43, Le Bao Tin Ha wrote: > I hope this email finds you well. I am currently working on the Toradex > Apalis iMX8QM board and I am trying to perform a hardware decode using the > Amphion driver with ffmpeg h264_v4l2m2m codec. The decoder outputs a pixel > format (NV12_8L128, a linear 8x128 NV12 based tiles) that is not supported > yet by FFmpeg. > Hello, Are you able to provide more information about this nonstandard pixel format? Kieran ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg
On Mon, 6 Feb 2023 at 15:43, Le Bao Tin Ha wrote: > I hope this email finds you well. I am currently working on the Toradex > Apalis iMX8QM board and I am trying to perform a hardware decode using the > Amphion driver with ffmpeg h264_v4l2m2m codec. The decoder outputs a pixel > format (NV12_8L128, a linear 8x128 NV12 based tiles) that is not supported > yet by FFmpeg. > Hello, Are you able to provide more information about this nonstandard pixel format? Kieran ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
