Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On Fri, 26 Jan 2018 15:15:05 + "Li, Zhong" wrote: > > From: libav-devel [mailto:libav-devel-boun...@libav.org] On Behalf Of Luca > > Barbato > > Sent: Tuesday, January 23, 2018 5:42 PM > > To: libav-devel@libav.org > > Subject: Re: [libav-devel] qsv: How about replace software parsers with > > MFXVideoDECODE_DecodeHeader > > > > On 22/01/2018 12:20, Li, Zhong wrote: > > > MSDK provides an API (MFXVideoDECODE_DecodeHeader) to parse video > > > parameters. Currently it hasn't been used. Instead, software parsers > > > are used. It works well for h264 decoder, and basically works well for > > > hevc decoder (some issues found by Mayxm due to width/height are > > > unaligned, I also found some hevc clips without setting profile can be > > > decoded by software but qsv failed) . More issues found on other > > > decoders such as VP8. The decoding conformance pass rate is low and > > > looks like it is due to some missing/incompatible header information > > > is passed to qsv decoder (though Mark provides a patch 182cf17 but > > > still something is missing). Similar issues happens on MJPEG decoding > > > which I am going to add. Maybe we can continue to work on software > > > parsers for qsv, but I believe replace software parser with > > > MFXVideoDECODE_DecodeHeader is a better choice: > > > > > > 1. It can remove the dependence on various software parsers, and > > > just need a unified interface for all codes. > > > > > > 2. It will be very easy to add new decoder such as MJPEG > > > decoding support without any software parser patches. > > > > > > 3. MFXVideoDECODE_DecodeHeader is used by MSDK sample > > decoder > > > (i.e: sample_decode), so it is reliable for MSDK decoder. (As my test, > > > it can effectively improve decoding conformance pass rate, especially > > > for vp8 decoding.) > > > > > > 4. CUVID decoder is using CUVID parser instead of software > > > parser, maybe qsv can align with it. > > > > No it is not, in Libav the CUVID support is implemented as hwaccel1, so it > > leverages the software decoder for the parsing and a little more. > > > > In my opinion it boils down to either have a uniform behavior in parsing and > > formatting the input with its many quirks or using more of the hwaccel/sdk. > > > > I do not have a strong opinion in this regard. > > > > lu > > Thank you all for the comments! > Currently I am inclined to keep current libavcodec parser (i.e; just like > wm4's suggested using as much libavcodec parsing functionality as possible). I didn't really suggest that, though. Sure, using the libavcodec parsers is a step up, but I was arguing that ideal behavior is only possible with a hwaccel approach, where libavcodec's built in decoders handle everything but actual slice decoding. ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
great conclusion and let's have the patches re-viewed. On Fri, Jan 26, 2018 at 6:56 PM, Luca Barbato wrote: > On 26/01/2018 16:15, Li, Zhong wrote: > >> Thank you all for the comments! Currently I am inclined to keep >> current libavcodec parser (i.e; just like wm4's suggested using as >> much libavcodec parsing functionality as possible). And will fix the >> decoder conformance issues (e.g: vp8) based on this. Qsv MJPEG >> decoding should be added later based on mjpeg_parser. >> >> > Thank you a lot :) I'm looking forward to your patches! > > lu > > ___ > libav-devel mailing list > libav-devel@libav.org > https://lists.libav.org/mailman/listinfo/libav-devel ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On 26/01/2018 16:15, Li, Zhong wrote: Thank you all for the comments! Currently I am inclined to keep current libavcodec parser (i.e; just like wm4's suggested using as much libavcodec parsing functionality as possible). And will fix the decoder conformance issues (e.g: vp8) based on this. Qsv MJPEG decoding should be added later based on mjpeg_parser. Thank you a lot :) I'm looking forward to your patches! lu ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
> From: libav-devel [mailto:libav-devel-boun...@libav.org] On Behalf Of Luca > Barbato > Sent: Tuesday, January 23, 2018 5:42 PM > To: libav-devel@libav.org > Subject: Re: [libav-devel] qsv: How about replace software parsers with > MFXVideoDECODE_DecodeHeader > > On 22/01/2018 12:20, Li, Zhong wrote: > > MSDK provides an API (MFXVideoDECODE_DecodeHeader) to parse video > > parameters. Currently it hasn't been used. Instead, software parsers > > are used. It works well for h264 decoder, and basically works well for > > hevc decoder (some issues found by Mayxm due to width/height are > > unaligned, I also found some hevc clips without setting profile can be > > decoded by software but qsv failed) . More issues found on other > > decoders such as VP8. The decoding conformance pass rate is low and > > looks like it is due to some missing/incompatible header information > > is passed to qsv decoder (though Mark provides a patch 182cf17 but > > still something is missing). Similar issues happens on MJPEG decoding > > which I am going to add. Maybe we can continue to work on software > > parsers for qsv, but I believe replace software parser with > > MFXVideoDECODE_DecodeHeader is a better choice: > > > > 1. It can remove the dependence on various software parsers, and > > just need a unified interface for all codes. > > > > 2. It will be very easy to add new decoder such as MJPEG > > decoding support without any software parser patches. > > > > 3. MFXVideoDECODE_DecodeHeader is used by MSDK sample > decoder > > (i.e: sample_decode), so it is reliable for MSDK decoder. (As my test, > > it can effectively improve decoding conformance pass rate, especially > > for vp8 decoding.) > > > > 4. CUVID decoder is using CUVID parser instead of software > > parser, maybe qsv can align with it. > > No it is not, in Libav the CUVID support is implemented as hwaccel1, so it > leverages the software decoder for the parsing and a little more. > > In my opinion it boils down to either have a uniform behavior in parsing and > formatting the input with its many quirks or using more of the hwaccel/sdk. > > I do not have a strong opinion in this regard. > > lu Thank you all for the comments! Currently I am inclined to keep current libavcodec parser (i.e; just like wm4's suggested using as much libavcodec parsing functionality as possible). And will fix the decoder conformance issues (e.g: vp8) based on this. Qsv MJPEG decoding should be added later based on mjpeg_parser. ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On 24/01/2018 04:25, Li, Zhong wrote: From: libav-devel [mailto:libav-devel-boun...@libav.org] On Behalf Of Luca Barbato Sent: Tuesday, January 23, 2018 5:45 PM To: libav-devel@libav.org Subject: Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader On 23/01/2018 06:05, Li, Zhong wrote: avformat_find_stream_info() can provides some information and sw_pix_fmt can be set as NV12. These parameters can be updated after MFXVideoDECODE_DecodeHeader is called. find_stream_info may decode frames using the software decoder to extract the missing information (and it is one of those sore points that sooner or later I'd like to completely remove from the library). lu Exactly, it looks a litter weird but I also see some codec parser provides few information (such as mjpeg parser) then it depends on the decoding frame process. So before removing that, we need to make sure the parsers are powerful? Please correct me if I am wrong. You are perfectly correct, removing and replacing find_stream_info with something less overkill is not exactly easy. lu ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
> From: libav-devel [mailto:libav-devel-boun...@libav.org] On Behalf Of Luca > Barbato > Sent: Tuesday, January 23, 2018 9:53 PM > To: libav-devel@libav.org > Subject: Re: [libav-devel] qsv: How about replace software parsers with > MFXVideoDECODE_DecodeHeader > > On 23/01/2018 12:15, Mark Thompson wrote: > > avconv -hwaccel vaapi -hwaccel_output_format vaapi -i input.mp4 -vf > 'scale_vaapi=1280:720,hwmap=derive_device=qsv,format=qsv' -c:v > h264_qsv -b:v 5M output.mp4 > > > > On Windows, there is similarly: > > > > avconv -hwaccel dxva2 -hwaccel_output_format dxva2_vld -i input.mp4 > -vf 'hwmap=derive_device=qsv,format=qsv' -c:v h264_qsv -b:v 5M > output.mp4 > > I'll nag you soon so we can document the whole thing a bit more :) > > lu Really a good suggestion to document. : ) . ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
> From: libav-devel [mailto:libav-devel-boun...@libav.org] On Behalf Of Luca > Barbato > Sent: Tuesday, January 23, 2018 5:45 PM > To: libav-devel@libav.org > Subject: Re: [libav-devel] qsv: How about replace software parsers with > MFXVideoDECODE_DecodeHeader > > On 23/01/2018 06:05, Li, Zhong wrote: > > avformat_find_stream_info() can provides some information and > > sw_pix_fmt can be set as NV12. These parameters can be updated after > > MFXVideoDECODE_DecodeHeader is called. > > find_stream_info may decode frames using the software decoder to extract > the missing information (and it is one of those sore points that sooner or > later I'd like to completely remove from the library). > > lu Exactly, it looks a litter weird but I also see some codec parser provides few information (such as mjpeg parser) then it depends on the decoding frame process. So before removing that, we need to make sure the parsers are powerful? Please correct me if I am wrong. ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
> Its best if decoders rely as little as possible on information passed to them > externally like this. If you have extradata, you should decode that to > determine the stream properties, and not rely on information set anywhere > else. > You should always keep in mind that a user application could use libavcodec > stand-alone without libavformat, so the amount of stream information they > provide might be different. Thanks for your remind. I believe few information is need to initialize qsv session, however. > > - Hendrik ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On 23/01/2018 12:15, Mark Thompson wrote: avconv -hwaccel vaapi -hwaccel_output_format vaapi -i input.mp4 -vf 'scale_vaapi=1280:720,hwmap=derive_device=qsv,format=qsv' -c:v h264_qsv -b:v 5M output.mp4 On Windows, there is similarly: avconv -hwaccel dxva2 -hwaccel_output_format dxva2_vld -i input.mp4 -vf 'hwmap=derive_device=qsv,format=qsv' -c:v h264_qsv -b:v 5M output.mp4 I'll nag you soon so we can document the whole thing a bit more :) lu ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On 23/01/18 05:05, Li, Zhong wrote: >> From: libav-devel [mailto:libav-devel-boun...@libav.org] On Behalf Of Mark >> Thompson >> Sent: Monday, January 22, 2018 9:19 PM >> To: libav-devel@libav.org >> Subject: Re: [libav-devel] qsv: How about replace software parsers with >> MFXVideoDECODE_DecodeHeader >> >> ... >> >> PS: Feel free to ignore anything I say about qsvdec - I regard qsvdec as >> deprecated, because nowadays I find that using the platform API hwaccels >> (DXVA2/VAAPI) and mapping back if necessary is just more flexible and better >> supported. (Though it would be nice if libmfx encode supported D3D11 >> array textures too...) > > Thanks very much for your detail reply and I'm appreciated for your comments. > IIUC, surface sharing between libmfx and vaapi hasn't be supported, and I'm > not sure it can be done since the release version of MSDK is using a private > libva (instead of upstream libva.) This interop already works and has done so for a while with the proprietary drivers. I'm not next to a suitable machine so this command may be subtly wrong, but on Linux with proprietary Media SDK you can do things like: avconv -hwaccel vaapi -hwaccel_output_format vaapi -i input.mp4 -vf 'scale_vaapi=1280:720,hwmap=derive_device=qsv,format=qsv' -c:v h264_qsv -b:v 5M output.mp4 On Windows, there is similarly: avconv -hwaccel dxva2 -hwaccel_output_format dxva2_vld -i input.mp4 -vf 'hwmap=derive_device=qsv,format=qsv' -c:v h264_qsv -b:v 5M output.mp4 (There aren't any generic filters for D3D9, though.) OpenCL interop also works on both platforms for intermediate processing. > The good news is opensource MSDK on github has been updated with upstream > libva (and iHD driver has removed the dependency for linux kernel patches), > but still needs time to make sure it is stable. Yeah, I've looked at it and done some investigation (and sent some patches), but indeed it does need a bit more time to clean up all of the setup to work as well as the i965 driver. The fact that it doesn't work on any current-generation machines (Kaby Lake / Coffee Lake) is also something of a pain. - Mark ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On Tue, 23 Jan 2018 05:28:35 + "Li, Zhong" wrote: > > wm4 wrote: > > > > > > Appreciated for any comment. If we are agree with that, patches will > > be sent soon (about following next two weeks). > > > > > > I don't know how qsvdec works in particular (doesn't it pretty much > > > parse all the codec stream in MFX?). > > If you check qsvdec.c, av_parser_parse2() is called. That is one thing > different from cuvid. > > But if you look how it went for > > > cuvid vs. nvdec, using as much parsing functionality from libavcodec > > > as possible is superior. > > > > > > Cuvid/nvdec are nvidia's decoder wrappers as they are named in FFmpeg. > > > (They don't necessarily match with the naming nvidia itself uses.) > > > Cuvid is a "full stream" decoder, which lets nvidia's software parse > > > the full stream. nvdec is a hwaccel, using nvidia's low level API on > > > top of libavcodec's own decoders, and which gets only slices > > > > Sorry, accidentally hit sent. Here's the rest: > > > > ... nvdec is a hwaccel, using nvidia's low level API on top of libavcodec's > > own > > decoders, and which gets only slices plus some metadata. nvdec is actually > > pretty much like the dxva2/d3d11va API. > > > > When I tried cuvid vs. nvdec myself, I found nvdec superior, because it > > integrated so much better into libavcodec. It supported parsing all the > > metadata libavcodec did, it avoided some weird nvidia (or cuvid > > wrapper) issues, and it was faster, because it used libavcodec's native data > > flow instead of having to go over weird FIFOs. Also, nvdec is essentially > > feature complete (except missing deinterlacing), while cuvid still misses > > exporting some metadata. With cuvid we'd be busy adding support to export > > new types of metadata every time, while nvdec simply benefits from > > development of the native libavcodec decoders. > > > > I get that Intel devs probably prefer qsvdec over dxva2/vaapi/etc. > > because they develop libmfx, but for libavcodec the hwaccel approach will > > always be superior. > > > > (I.e. basically what Mark said.) > > Yes, basically speaking, cuvid vs nvdec is similar as qsv vs dxva2/vaapi. > For linux, I am open to vaapi and planning do some contribution on this. > But I think it should be another topic. > Using software parser (i.e. av_parser_parse2()) for qsvdec seems can't bring > too much advantage compared with using MSDK API, though it is stable for some > codecs since it is existed a while. Even if you call the parser, you have to duplicate the functionality of the native decoder by adding glue code, and it will always lag behind it. Not to mention things like transparent fallback to software or other hwaccels, and the fact that everything will get parsed at least twice. ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On 23/01/2018 06:05, Li, Zhong wrote: avformat_find_stream_info() can provides some information and sw_pix_fmt can be set as NV12. These parameters can be updated after MFXVideoDECODE_DecodeHeader is called. find_stream_info may decode frames using the software decoder to extract the missing information (and it is one of those sore points that sooner or later I'd like to completely remove from the library). lu ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On 22/01/2018 12:20, Li, Zhong wrote: MSDK provides an API (MFXVideoDECODE_DecodeHeader) to parse video parameters. Currently it hasn't been used. Instead, software parsers are used. It works well for h264 decoder, and basically works well for hevc decoder (some issues found by Mayxm due to width/height are unaligned, I also found some hevc clips without setting profile can be decoded by software but qsv failed) . More issues found on other decoders such as VP8. The decoding conformance pass rate is low and looks like it is due to some missing/incompatible header information is passed to qsv decoder (though Mark provides a patch 182cf17 but still something is missing). Similar issues happens on MJPEG decoding which I am going to add. Maybe we can continue to work on software parsers for qsv, but I believe replace software parser with MFXVideoDECODE_DecodeHeader is a better choice: 1. It can remove the dependence on various software parsers, and just need a unified interface for all codes. 2. It will be very easy to add new decoder such as MJPEG decoding support without any software parser patches. 3. MFXVideoDECODE_DecodeHeader is used by MSDK sample decoder (i.e: sample_decode), so it is reliable for MSDK decoder. (As my test, it can effectively improve decoding conformance pass rate, especially for vp8 decoding.) 4. CUVID decoder is using CUVID parser instead of software parser, maybe qsv can align with it. No it is not, in Libav the CUVID support is implemented as hwaccel1, so it leverages the software decoder for the parsing and a little more. In my opinion it boils down to either have a uniform behavior in parsing and formatting the input with its many quirks or using more of the hwaccel/sdk. I do not have a strong opinion in this regard. lu ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On Tue, Jan 23, 2018 at 6:05 AM, Li, Zhong wrote: >> >> I believe a problem with this method is that you don't have any libmfx >> session at the point where you do the initial parsing (since you only get it >> after the get_format() callback, which needs information from that parsing). >> How would you intend to get the session to use for this purpose? > > avformat_find_stream_info() can provides some information and sw_pix_fmt can > be set as NV12. > These parameters can be updated after MFXVideoDECODE_DecodeHeader is called. > Its best if decoders rely as little as possible on information passed to them externally like this. If you have extradata, you should decode that to determine the stream properties, and not rely on information set anywhere else. You should always keep in mind that a user application could use libavcodec stand-alone without libavformat, so the amount of stream information they provide might be different. Right now the only thing you need to pass to make H264/HEVC decoding work, for example, is the extradata. - Hendrik ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
> wm4 wrote: > > > > Appreciated for any comment. If we are agree with that, patches will > be sent soon (about following next two weeks). > > > > I don't know how qsvdec works in particular (doesn't it pretty much > > parse all the codec stream in MFX?). If you check qsvdec.c, av_parser_parse2() is called. That is one thing different from cuvid. But if you look how it went for > > cuvid vs. nvdec, using as much parsing functionality from libavcodec > > as possible is superior. > > > > Cuvid/nvdec are nvidia's decoder wrappers as they are named in FFmpeg. > > (They don't necessarily match with the naming nvidia itself uses.) > > Cuvid is a "full stream" decoder, which lets nvidia's software parse > > the full stream. nvdec is a hwaccel, using nvidia's low level API on > > top of libavcodec's own decoders, and which gets only slices > > Sorry, accidentally hit sent. Here's the rest: > > ... nvdec is a hwaccel, using nvidia's low level API on top of libavcodec's > own > decoders, and which gets only slices plus some metadata. nvdec is actually > pretty much like the dxva2/d3d11va API. > > When I tried cuvid vs. nvdec myself, I found nvdec superior, because it > integrated so much better into libavcodec. It supported parsing all the > metadata libavcodec did, it avoided some weird nvidia (or cuvid > wrapper) issues, and it was faster, because it used libavcodec's native data > flow instead of having to go over weird FIFOs. Also, nvdec is essentially > feature complete (except missing deinterlacing), while cuvid still misses > exporting some metadata. With cuvid we'd be busy adding support to export > new types of metadata every time, while nvdec simply benefits from > development of the native libavcodec decoders. > > I get that Intel devs probably prefer qsvdec over dxva2/vaapi/etc. > because they develop libmfx, but for libavcodec the hwaccel approach will > always be superior. > > (I.e. basically what Mark said.) Yes, basically speaking, cuvid vs nvdec is similar as qsv vs dxva2/vaapi. For linux, I am open to vaapi and planning do some contribution on this. But I think it should be another topic. Using software parser (i.e. av_parser_parse2()) for qsvdec seems can't bring too much advantage compared with using MSDK API, though it is stable for some codecs since it is existed a while. ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
> From: libav-devel [mailto:libav-devel-boun...@libav.org] On Behalf Of Mark > Thompson > Sent: Monday, January 22, 2018 9:19 PM > To: libav-devel@libav.org > Subject: Re: [libav-devel] qsv: How about replace software parsers with > MFXVideoDECODE_DecodeHeader > > On 22/01/18 11:20, Li, Zhong wrote: > > MSDK provides an API (MFXVideoDECODE_DecodeHeader) to parse > video parameters. Currently it hasn't been used. > > Instead, software parsers are used. It works well for h264 decoder, and > basically works well for hevc decoder (some issues found by Mayxm due to > width/height are unaligned, I also found some hevc clips without setting > profile can be decoded by software but qsv failed) . > > More issues found on other decoders such as VP8. The decoding > conformance pass rate is low and looks like it is due to some > missing/incompatible header information is passed to qsv decoder (though > Mark provides a patch 182cf17 but still something is missing). > > Similar issues happens on MJPEG decoding which I am going to add. > > Have you looked into what is missing for VP8? There shouldn't be much > magic there, so I doubt that particular case is difficult to fix. Just checked some hevc failed cases. Haven't got much detail for vp8 failed cases. I prefer to collect some comment of using MFXVideoDECODE_DecodeHeader. If we decide to stick on software parser, I will look into the detail and try to fix it. And I agree with you it should not be very difficult. > > I imagine MJPEG will be nastier, though, because there is a lot more variation > in layout. > > > Maybe we can continue to work on software parsers for qsv, but I > believe replace software parser with MFXVideoDECODE_DecodeHeader is a > better choice: > > > > 1. It can remove the dependence on various software parsers, and > just need a unified interface for all codes. > > > > 2. It will be very easy to add new decoder such as MJPEG decoding > support without any software parser patches. > > > > 3. MFXVideoDECODE_DecodeHeader is used by MSDK sample > decoder (i.e: sample_decode), so it is reliable for MSDK decoder. (As my test, > it can effectively improve decoding conformance pass rate, especially for > vp8 decoding.) > > > > 4. CUVID decoder is using CUVID parser instead of software parser, > maybe qsv can align with it. > > The CUVID decoder only exists in ffmpeg and is considered deprecated in > favour of the hwaccel (which is the only thing in libav). > > >Negative effect: > > > > 1. May cause some regression since it will take effect to all codecs. > > > > 2. Others? > > I believe a problem with this method is that you don't have any libmfx > session at the point where you do the initial parsing (since you only get it > after the get_format() callback, which needs information from that parsing). > How would you intend to get the session to use for this purpose? avformat_find_stream_info() can provides some information and sw_pix_fmt can be set as NV12. These parameters can be updated after MFXVideoDECODE_DecodeHeader is called. > > - Mark > > > PS: Feel free to ignore anything I say about qsvdec - I regard qsvdec as > deprecated, because nowadays I find that using the platform API hwaccels > (DXVA2/VAAPI) and mapping back if necessary is just more flexible and better > supported. (Though it would be nice if libmfx encode supported D3D11 > array textures too...) Thanks very much for your detail reply and I'm appreciated for your comments. IIUC, surface sharing between libmfx and vaapi hasn't be supported, and I'm not sure it can be done since the release version of MSDK is using a private libva (instead of upstream libva.) The good news is opensource MSDK on github has been updated with upstream libva (and iHD driver has removed the dependency for linux kernel patches), but still needs time to make sure it is stable. ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On Mon, 22 Jan 2018 14:52:00 +0100 wm4 wrote: > > Appreciated for any comment. If we are agree with that, patches will be > > sent soon (about following next two weeks). > > I don't know how qsvdec works in particular (doesn't it pretty much > parse all the codec stream in MFX?). But if you look how it went for > cuvid vs. nvdec, using as much parsing functionality from libavcodec as > possible is superior. > > Cuvid/nvdec are nvidia's decoder wrappers as they are named in FFmpeg. > (They don't necessarily match with the naming nvidia itself uses.) Cuvid > is a "full stream" decoder, which lets nvidia's software parse the full > stream. nvdec is a hwaccel, using nvidia's low level API on top of > libavcodec's own decoders, and which gets only slices Sorry, accidentally hit sent. Here's the rest: ... nvdec is a hwaccel, using nvidia's low level API on top of libavcodec's own decoders, and which gets only slices plus some metadata. nvdec is actually pretty much like the dxva2/d3d11va API. When I tried cuvid vs. nvdec myself, I found nvdec superior, because it integrated so much better into libavcodec. It supported parsing all the metadata libavcodec did, it avoided some weird nvidia (or cuvid wrapper) issues, and it was faster, because it used libavcodec's native data flow instead of having to go over weird FIFOs. Also, nvdec is essentially feature complete (except missing deinterlacing), while cuvid still misses exporting some metadata. With cuvid we'd be busy adding support to export new types of metadata every time, while nvdec simply benefits from development of the native libavcodec decoders. I get that Intel devs probably prefer qsvdec over dxva2/vaapi/etc. because they develop libmfx, but for libavcodec the hwaccel approach will always be superior. (I.e. basically what Mark said.) ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On Mon, 22 Jan 2018 11:20:08 + "Li, Zhong" wrote: > MSDK provides an API (MFXVideoDECODE_DecodeHeader) to parse video > parameters. Currently it hasn't been used. > Instead, software parsers are used. It works well for h264 decoder, and > basically works well for hevc decoder (some issues found by Mayxm due to > width/height are unaligned, I also found some hevc clips without setting > profile can be decoded by software but qsv failed) . > More issues found on other decoders such as VP8. The decoding conformance > pass rate is low and looks like it is due to some missing/incompatible header > information is passed to qsv decoder (though Mark provides a patch 182cf17 > but still something is missing). > Similar issues happens on MJPEG decoding which I am going to add. > Maybe we can continue to work on software parsers for qsv, but I believe > replace software parser with MFXVideoDECODE_DecodeHeader is a better choice: > > 1. It can remove the dependence on various software parsers, and just > need a unified interface for all codes. > > 2. It will be very easy to add new decoder such as MJPEG decoding > support without any software parser patches. > > 3. MFXVideoDECODE_DecodeHeader is used by MSDK sample decoder (i.e: > sample_decode), so it is reliable for MSDK decoder. (As my test, it can > effectively improve decoding conformance pass rate, especially for vp8 > decoding.) > > 4. CUVID decoder is using CUVID parser instead of software parser, maybe > qsv can align with it. > >Negative effect: > > 1. May cause some regression since it will take effect to all codecs. > > 2. Others? > > Appreciated for any comment. If we are agree with that, patches will be > sent soon (about following next two weeks). I don't know how qsvdec works in particular (doesn't it pretty much parse all the codec stream in MFX?). But if you look how it went for cuvid vs. nvdec, using as much parsing functionality from libavcodec as possible is superior. Cuvid/nvdec are nvidia's decoder wrappers as they are named in FFmpeg. (They don't necessarily match with the naming nvidia itself uses.) Cuvid is a "full stream" decoder, which lets nvidia's software parse the full stream. nvdec is a hwaccel, using nvidia's low level API on top of libavcodec's own decoders, and which gets only slices ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
Mark, libmfx supported D3D11 array textures too - approved for implementation and should be there soon On Mon, Jan 22, 2018 at 2:19 PM, Mark Thompson wrote: > On 22/01/18 11:20, Li, Zhong wrote: > > MSDK provides an API (MFXVideoDECODE_DecodeHeader) to parse video > parameters. Currently it hasn't been used. > > Instead, software parsers are used. It works well for h264 decoder, and > basically works well for hevc decoder (some issues found by Mayxm due to > width/height are unaligned, I also found some hevc clips without setting > profile can be decoded by software but qsv failed) . > > More issues found on other decoders such as VP8. The decoding > conformance pass rate is low and looks like it is due to some > missing/incompatible header information is passed to qsv decoder (though > Mark provides a patch 182cf17 but still something is missing). > > Similar issues happens on MJPEG decoding which I am going to add. > > Have you looked into what is missing for VP8? There shouldn't be much > magic there, so I doubt that particular case is difficult to fix. > > I imagine MJPEG will be nastier, though, because there is a lot more > variation in layout. > > > Maybe we can continue to work on software parsers for qsv, but I > believe replace software parser with MFXVideoDECODE_DecodeHeader is a > better choice: > > > > 1. It can remove the dependence on various software parsers, and > just need a unified interface for all codes. > > > > 2. It will be very easy to add new decoder such as MJPEG decoding > support without any software parser patches. > > > > 3. MFXVideoDECODE_DecodeHeader is used by MSDK sample decoder (i.e: > sample_decode), so it is reliable for MSDK decoder. (As my test, it can > effectively improve decoding conformance pass rate, especially for vp8 > decoding.) > > > > 4. CUVID decoder is using CUVID parser instead of software parser, > maybe qsv can align with it. > > The CUVID decoder only exists in ffmpeg and is considered deprecated in > favour of the hwaccel (which is the only thing in libav). > > >Negative effect: > > > > 1. May cause some regression since it will take effect to all > codecs. > > > > 2. Others? > > I believe a problem with this method is that you don't have any libmfx > session at the point where you do the initial parsing (since you only get > it after the get_format() callback, which needs information from that > parsing). How would you intend to get the session to use for this purpose? > > - Mark > > > PS: Feel free to ignore anything I say about qsvdec - I regard qsvdec as > deprecated, because nowadays I find that using the platform API hwaccels > (DXVA2/VAAPI) and mapping back if necessary is just more flexible and > better supported. (Though it would be nice if libmfx encode supported > D3D11 array textures too...) > ___ > libav-devel mailing list > libav-devel@libav.org > https://lists.libav.org/mailman/listinfo/libav-devel ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
On 22/01/18 11:20, Li, Zhong wrote: > MSDK provides an API (MFXVideoDECODE_DecodeHeader) to parse video > parameters. Currently it hasn't been used. > Instead, software parsers are used. It works well for h264 decoder, and > basically works well for hevc decoder (some issues found by Mayxm due to > width/height are unaligned, I also found some hevc clips without setting > profile can be decoded by software but qsv failed) . > More issues found on other decoders such as VP8. The decoding conformance > pass rate is low and looks like it is due to some missing/incompatible header > information is passed to qsv decoder (though Mark provides a patch 182cf17 > but still something is missing). > Similar issues happens on MJPEG decoding which I am going to add. Have you looked into what is missing for VP8? There shouldn't be much magic there, so I doubt that particular case is difficult to fix. I imagine MJPEG will be nastier, though, because there is a lot more variation in layout. > Maybe we can continue to work on software parsers for qsv, but I believe > replace software parser with MFXVideoDECODE_DecodeHeader is a better choice: > > 1. It can remove the dependence on various software parsers, and just > need a unified interface for all codes. > > 2. It will be very easy to add new decoder such as MJPEG decoding > support without any software parser patches. > > 3. MFXVideoDECODE_DecodeHeader is used by MSDK sample decoder (i.e: > sample_decode), so it is reliable for MSDK decoder. (As my test, it can > effectively improve decoding conformance pass rate, especially for vp8 > decoding.) > > 4. CUVID decoder is using CUVID parser instead of software parser, maybe > qsv can align with it. The CUVID decoder only exists in ffmpeg and is considered deprecated in favour of the hwaccel (which is the only thing in libav). >Negative effect: > > 1. May cause some regression since it will take effect to all codecs. > > 2. Others? I believe a problem with this method is that you don't have any libmfx session at the point where you do the initial parsing (since you only get it after the get_format() callback, which needs information from that parsing). How would you intend to get the session to use for this purpose? - Mark PS: Feel free to ignore anything I say about qsvdec - I regard qsvdec as deprecated, because nowadays I find that using the platform API hwaccels (DXVA2/VAAPI) and mapping back if necessary is just more flexible and better supported. (Though it would be nice if libmfx encode supported D3D11 array textures too...) ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
Re: [libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
IMHO - good reason(s) plus : it will be more consistent approach to rely on QSV's API/DecodeHeader functionality. would not expect many regression(s), so patches would be good to have for review. On Mon, Jan 22, 2018 at 12:20 PM, Li, Zhong wrote: > MSDK provides an API (MFXVideoDECODE_DecodeHeader) to parse video > parameters. Currently it hasn't been used. > Instead, software parsers are used. It works well for h264 decoder, and > basically works well for hevc decoder (some issues found by Mayxm due to > width/height are unaligned, I also found some hevc clips without setting > profile can be decoded by software but qsv failed) . > More issues found on other decoders such as VP8. The decoding conformance > pass rate is low and looks like it is due to some missing/incompatible > header information is passed to qsv decoder (though Mark provides a patch > 182cf17 but still something is missing). > Similar issues happens on MJPEG decoding which I am going to add. > Maybe we can continue to work on software parsers for qsv, but I believe > replace software parser with MFXVideoDECODE_DecodeHeader is a better choice: > > 1. It can remove the dependence on various software parsers, and just > need a unified interface for all codes. > > 2. It will be very easy to add new decoder such as MJPEG decoding > support without any software parser patches. > > 3. MFXVideoDECODE_DecodeHeader is used by MSDK sample decoder (i.e: > sample_decode), so it is reliable for MSDK decoder. (As my test, it can > effectively improve decoding conformance pass rate, especially for vp8 > decoding.) > > 4. CUVID decoder is using CUVID parser instead of software parser, > maybe qsv can align with it. > >Negative effect: > > 1. May cause some regression since it will take effect to all codecs. > > 2. Others? > > Appreciated for any comment. If we are agree with that, patches will be > sent soon (about following next two weeks). > Thanks > Zhong > > ___ > libav-devel mailing list > libav-devel@libav.org > https://lists.libav.org/mailman/listinfo/libav-devel ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
[libav-devel] qsv: How about replace software parsers with MFXVideoDECODE_DecodeHeader
MSDK provides an API (MFXVideoDECODE_DecodeHeader) to parse video parameters. Currently it hasn't been used. Instead, software parsers are used. It works well for h264 decoder, and basically works well for hevc decoder (some issues found by Mayxm due to width/height are unaligned, I also found some hevc clips without setting profile can be decoded by software but qsv failed) . More issues found on other decoders such as VP8. The decoding conformance pass rate is low and looks like it is due to some missing/incompatible header information is passed to qsv decoder (though Mark provides a patch 182cf17 but still something is missing). Similar issues happens on MJPEG decoding which I am going to add. Maybe we can continue to work on software parsers for qsv, but I believe replace software parser with MFXVideoDECODE_DecodeHeader is a better choice: 1. It can remove the dependence on various software parsers, and just need a unified interface for all codes. 2. It will be very easy to add new decoder such as MJPEG decoding support without any software parser patches. 3. MFXVideoDECODE_DecodeHeader is used by MSDK sample decoder (i.e: sample_decode), so it is reliable for MSDK decoder. (As my test, it can effectively improve decoding conformance pass rate, especially for vp8 decoding.) 4. CUVID decoder is using CUVID parser instead of software parser, maybe qsv can align with it. Negative effect: 1. May cause some regression since it will take effect to all codecs. 2. Others? Appreciated for any comment. If we are agree with that, patches will be sent soon (about following next two weeks). Thanks Zhong ___ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
