Re: [linux-sunxi] [PATCH v2 1/2] media: v4l: Add definitions for the HEVC slice format and controls
Sent from my iPad > On Jan 30, 2019, at 3:17 PM, Tomasz Figa wrote: > >> On Wed, Jan 30, 2019 at 3:28 PM Ayaka wrote: >> >> >> >> Sent from my iPad >> >>> On Jan 30, 2019, at 11:35 AM, Tomasz Figa wrote: >>> >>> On Wed, Jan 30, 2019 at 11:29 AM Alexandre Courbot >>> wrote: >>>> >>>>>> On Wed, Jan 30, 2019 at 6:41 AM Nicolas Dufresne >>>>>> wrote: >>>>>> >>>>>> Le mardi 29 janvier 2019 à 16:44 +0900, Alexandre Courbot a écrit : >>>>>> On Fri, Jan 25, 2019 at 10:04 PM Paul Kocialkowski >>>>>> wrote: >>>>>>> Hi, >>>>>>> >>>>>>>> On Thu, 2019-01-24 at 20:23 +0800, Ayaka wrote: >>>>>>>> Sent from my iPad >>>>>>>> >>>>>>>>> On Jan 24, 2019, at 6:27 PM, Paul Kocialkowski >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>>> On Thu, 2019-01-10 at 21:32 +0800, ayaka wrote: >>>>>>>>>> I forget a important thing, for the rkvdec and rk hevc decoder, it >>>>>>>>>> would >>>>>>>>>> requests cabac table, scaling list, picture parameter set and >>>>>>>>>> reference >>>>>>>>>> picture storing in one or various of DMA buffers. I am not talking >>>>>>>>>> about >>>>>>>>>> the data been parsed, the decoder would requests a raw data. >>>>>>>>>> >>>>>>>>>> For the pps and rps, it is possible to reuse the slice header, just >>>>>>>>>> let >>>>>>>>>> the decoder know the offset from the bitstream bufer, I would >>>>>>>>>> suggest to >>>>>>>>>> add three properties(with sps) for them. But I think we need a >>>>>>>>>> method to >>>>>>>>>> mark a OUTPUT side buffer for those aux data. >>>>>>>>> >>>>>>>>> I'm quite confused about the hardware implementation then. From what >>>>>>>>> you're saying, it seems that it takes the raw bitstream elements >>>>>>>>> rather >>>>>>>>> than parsed elements. Is it really a stateless implementation? >>>>>>>>> >>>>>>>>> The stateless implementation was designed with the idea that only the >>>>>>>>> raw slice data should be passed in bitstream form to the decoder. For >>>>>>>>> H.264, it seems that some decoders also need the slice header in raw >>>>>>>>> bitstream form (because they take the full slice NAL unit), see the >>>>>>>>> discussions in this thread: >>>>>>>>> media: docs-rst: Document m2m stateless video decoder interface >>>>>>>> >>>>>>>> Stateless just mean it won’t track the previous result, but I don’t >>>>>>>> think you can define what a date the hardware would need. Even you >>>>>>>> just build a dpb for the decoder, it is still stateless, but parsing >>>>>>>> less or more data from the bitstream doesn’t stop a decoder become a >>>>>>>> stateless decoder. >>>>>>> >>>>>>> Yes fair enough, the format in which the hardware decoder takes the >>>>>>> bitstream parameters does not make it stateless or stateful per-se. >>>>>>> It's just that stateless decoders should have no particular reason for >>>>>>> parsing the bitstream on their own since the hardware can be designed >>>>>>> with registers for each relevant bitstream element to configure the >>>>>>> decoding pipeline. That's how GPU-based decoder implementations are >>>>>>> implemented (VAAPI/VDPAU/NVDEC, etc). >>>>>>> >>>>>>> So the format we have agreed on so far for the stateless interface is >>>>>>> to pass parsed elements via v4l2 control structures. >>>>>>> >>>>>>> If the hardware can only work by parsing the bitstream itself, I'm not >>
Re: [linux-sunxi] [PATCH v2 1/2] media: v4l: Add definitions for the HEVC slice format and controls
Sent from my iPad > On Jan 30, 2019, at 5:41 AM, Nicolas Dufresne wrote: > >> Le mardi 29 janvier 2019 à 16:44 +0900, Alexandre Courbot a écrit : >> On Fri, Jan 25, 2019 at 10:04 PM Paul Kocialkowski >> wrote: >>> Hi, >>> >>>> On Thu, 2019-01-24 at 20:23 +0800, Ayaka wrote: >>>> Sent from my iPad >>>> >>>>> On Jan 24, 2019, at 6:27 PM, Paul Kocialkowski >>>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>>> On Thu, 2019-01-10 at 21:32 +0800, ayaka wrote: >>>>>> I forget a important thing, for the rkvdec and rk hevc decoder, it would >>>>>> requests cabac table, scaling list, picture parameter set and reference >>>>>> picture storing in one or various of DMA buffers. I am not talking about >>>>>> the data been parsed, the decoder would requests a raw data. >>>>>> >>>>>> For the pps and rps, it is possible to reuse the slice header, just let >>>>>> the decoder know the offset from the bitstream bufer, I would suggest to >>>>>> add three properties(with sps) for them. But I think we need a method to >>>>>> mark a OUTPUT side buffer for those aux data. >>>>> >>>>> I'm quite confused about the hardware implementation then. From what >>>>> you're saying, it seems that it takes the raw bitstream elements rather >>>>> than parsed elements. Is it really a stateless implementation? >>>>> >>>>> The stateless implementation was designed with the idea that only the >>>>> raw slice data should be passed in bitstream form to the decoder. For >>>>> H.264, it seems that some decoders also need the slice header in raw >>>>> bitstream form (because they take the full slice NAL unit), see the >>>>> discussions in this thread: >>>>> media: docs-rst: Document m2m stateless video decoder interface >>>> >>>> Stateless just mean it won’t track the previous result, but I don’t >>>> think you can define what a date the hardware would need. Even you >>>> just build a dpb for the decoder, it is still stateless, but parsing >>>> less or more data from the bitstream doesn’t stop a decoder become a >>>> stateless decoder. >>> >>> Yes fair enough, the format in which the hardware decoder takes the >>> bitstream parameters does not make it stateless or stateful per-se. >>> It's just that stateless decoders should have no particular reason for >>> parsing the bitstream on their own since the hardware can be designed >>> with registers for each relevant bitstream element to configure the >>> decoding pipeline. That's how GPU-based decoder implementations are >>> implemented (VAAPI/VDPAU/NVDEC, etc). >>> >>> So the format we have agreed on so far for the stateless interface is >>> to pass parsed elements via v4l2 control structures. >>> >>> If the hardware can only work by parsing the bitstream itself, I'm not >>> sure what the best solution would be. Reconstructing the bitstream in >>> the kernel is a pretty bad option, but so is parsing in the kernel or >>> having the data both in parsed and raw forms. Do you see another >>> possibility? >> >> Is reconstructing the bitstream so bad? The v4l2 controls provide a >> generic interface to an encoded format which the driver needs to >> convert into a sequence that the hardware can understand. Typically >> this is done by populating hardware-specific structures. Can't we >> consider that in this specific instance, the hardware-specific >> structure just happens to be identical to the original bitstream >> format? > > At maximum allowed bitrate for let's say HEVC (940MB/s iirc), yes, it Lucky, most of hardware won’t be able to processing such a big buffer. General speaking, the register is 24bits for stream length in bytes. > would be really really bad. In GStreamer project we have discussed for > a while (but have never done anything about) adding the ability through > a bitmask to select which part of the stream need to be parsed, as > parsing itself was causing some overhead. Maybe similar thing applies, > though as per our new design, it's the fourcc that dictate the driver > behaviour, we'd need yet another fourcc for drivers that wants the full > bitstream (which seems odd if you have already parsed everything, I > think this need some clarification). > >> >> I agree that this is not strictly optimal for that particular >> hardware, but such is the cost of abstractions, and in this specific >> case I don't believe the cost would be particularly high? ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
Re: [linux-sunxi] [PATCH v2 1/2] media: v4l: Add definitions for the HEVC slice format and controls
Sent from my iPad > On Jan 30, 2019, at 11:35 AM, Tomasz Figa wrote: > > On Wed, Jan 30, 2019 at 11:29 AM Alexandre Courbot > wrote: >> >>> On Wed, Jan 30, 2019 at 6:41 AM Nicolas Dufresne >>> wrote: >>> >>>> Le mardi 29 janvier 2019 à 16:44 +0900, Alexandre Courbot a écrit : >>>> On Fri, Jan 25, 2019 at 10:04 PM Paul Kocialkowski >>>> wrote: >>>>> Hi, >>>>> >>>>>> On Thu, 2019-01-24 at 20:23 +0800, Ayaka wrote: >>>>>> Sent from my iPad >>>>>> >>>>>>> On Jan 24, 2019, at 6:27 PM, Paul Kocialkowski >>>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>>> On Thu, 2019-01-10 at 21:32 +0800, ayaka wrote: >>>>>>>> I forget a important thing, for the rkvdec and rk hevc decoder, it >>>>>>>> would >>>>>>>> requests cabac table, scaling list, picture parameter set and reference >>>>>>>> picture storing in one or various of DMA buffers. I am not talking >>>>>>>> about >>>>>>>> the data been parsed, the decoder would requests a raw data. >>>>>>>> >>>>>>>> For the pps and rps, it is possible to reuse the slice header, just let >>>>>>>> the decoder know the offset from the bitstream bufer, I would suggest >>>>>>>> to >>>>>>>> add three properties(with sps) for them. But I think we need a method >>>>>>>> to >>>>>>>> mark a OUTPUT side buffer for those aux data. >>>>>>> >>>>>>> I'm quite confused about the hardware implementation then. From what >>>>>>> you're saying, it seems that it takes the raw bitstream elements rather >>>>>>> than parsed elements. Is it really a stateless implementation? >>>>>>> >>>>>>> The stateless implementation was designed with the idea that only the >>>>>>> raw slice data should be passed in bitstream form to the decoder. For >>>>>>> H.264, it seems that some decoders also need the slice header in raw >>>>>>> bitstream form (because they take the full slice NAL unit), see the >>>>>>> discussions in this thread: >>>>>>> media: docs-rst: Document m2m stateless video decoder interface >>>>>> >>>>>> Stateless just mean it won’t track the previous result, but I don’t >>>>>> think you can define what a date the hardware would need. Even you >>>>>> just build a dpb for the decoder, it is still stateless, but parsing >>>>>> less or more data from the bitstream doesn’t stop a decoder become a >>>>>> stateless decoder. >>>>> >>>>> Yes fair enough, the format in which the hardware decoder takes the >>>>> bitstream parameters does not make it stateless or stateful per-se. >>>>> It's just that stateless decoders should have no particular reason for >>>>> parsing the bitstream on their own since the hardware can be designed >>>>> with registers for each relevant bitstream element to configure the >>>>> decoding pipeline. That's how GPU-based decoder implementations are >>>>> implemented (VAAPI/VDPAU/NVDEC, etc). >>>>> >>>>> So the format we have agreed on so far for the stateless interface is >>>>> to pass parsed elements via v4l2 control structures. >>>>> >>>>> If the hardware can only work by parsing the bitstream itself, I'm not >>>>> sure what the best solution would be. Reconstructing the bitstream in >>>>> the kernel is a pretty bad option, but so is parsing in the kernel or >>>>> having the data both in parsed and raw forms. Do you see another >>>>> possibility? >>>> >>>> Is reconstructing the bitstream so bad? The v4l2 controls provide a >>>> generic interface to an encoded format which the driver needs to >>>> convert into a sequence that the hardware can understand. Typically >>>> this is done by populating hardware-specific structures. Can't we >>>> consider that in this specific instance, the hardware-specific >>>> structure just happens to be identical to the original bitstream >>>> format? >
Re: [linux-sunxi] [PATCH v2 1/2] media: v4l: Add definitions for the HEVC slice format and controls
Sent from my iPad
> On Jan 24, 2019, at 6:27 PM, Paul Kocialkowski
> wrote:
>
> Hi,
>
>> On Thu, 2019-01-10 at 21:32 +0800, ayaka wrote:
>> I forget a important thing, for the rkvdec and rk hevc decoder, it would
>> requests cabac table, scaling list, picture parameter set and reference
>> picture storing in one or various of DMA buffers. I am not talking about
>> the data been parsed, the decoder would requests a raw data.
>>
>> For the pps and rps, it is possible to reuse the slice header, just let
>> the decoder know the offset from the bitstream bufer, I would suggest to
>> add three properties(with sps) for them. But I think we need a method to
>> mark a OUTPUT side buffer for those aux data.
>
> I'm quite confused about the hardware implementation then. From what
> you're saying, it seems that it takes the raw bitstream elements rather
> than parsed elements. Is it really a stateless implementation?
>
> The stateless implementation was designed with the idea that only the
> raw slice data should be passed in bitstream form to the decoder. For
> H.264, it seems that some decoders also need the slice header in raw
> bitstream form (because they take the full slice NAL unit), see the
> discussions in this thread:
> media: docs-rst: Document m2m stateless video decoder interface
Stateless just mean it won’t track the previous result, but I don’t think you
can define what a date the hardware would need. Even you just build a dpb for
the decoder, it is still stateless, but parsing less or more data from the
bitstream doesn’t stop a decoder become a stateless decoder.
>
> Can you detail exactly what the rockchip decoder absolutely needs in
> raw bitstream format?
>
> Cheers,
>
> Paul
>
>>> On 1/8/19 6:00 PM, Ayaka wrote:
>>> Sent from my iPad
>>>
>>>> On Jan 8, 2019, at 4:38 PM, Paul Kocialkowski
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>>> On Tue, 2019-01-08 at 09:16 +0800, Ayaka wrote:
>>>>>
>>>>> Sent from my iPad
>>>>>
>>>>>> On Jan 7, 2019, at 5:57 PM, Paul Kocialkowski
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>>> On Mon, 2019-01-07 at 11:49 +0800, Randy Li wrote:
>>>>>>>> On 12/12/18 8:51 PM, Paul Kocialkowski wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On Wed, 2018-12-05 at 21:59 +0100, Jernej Škrabec wrote:
>>>>>>>>
>>>>>>>>>> +
>>>>>>>>>> +#define V4L2_HEVC_DPB_ENTRY_RPS_ST_CURR_BEFORE0x01
>>>>>>>>>> +#define V4L2_HEVC_DPB_ENTRY_RPS_ST_CURR_AFTER0x02
>>>>>>>>>> +#define V4L2_HEVC_DPB_ENTRY_RPS_LT_CURR0x03
>>>>>>>>>> +
>>>>>>>>>> +#define V4L2_HEVC_DPB_ENTRIES_NUM_MAX16
>>>>>>>>>> +
>>>>>>>>>> +struct v4l2_hevc_dpb_entry {
>>>>>>>>>> +__u32buffer_tag;
>>>>>>>>>> +__u8rps;
>>>>>>>>>> +__u8field_pic;
>>>>>>>>>> +__u16pic_order_cnt[2];
>>>>>>>>>> +};
>>>>>>> Please add a property for reference index, if that rps is not used for
>>>>>>> this, some device would request that(not the rockchip one). And
>>>>>>> Rockchip's VDPU1 and VDPU2 for AVC would request a similar property.
>>>>>> What exactly is that reference index? Is it a bitstream element or
>>>>>> something deduced from the bitstream?
>>>>>>
>>>>> picture order count(POC) for HEVC and frame_num in AVC. I think it is
>>>>> the number used in list0(P slice and B slice) and list1(B slice).
>>>> The picture order count is already the last field of the DPB entry
>>>> structure. There is one for each field picture.
>>> As we are not sure whether there is a field coded slice or CTU, I would
>>> hold this part and else about the field.
>>>>>>> Adding another buffer_tag for referring the memory of the motion vectors
>>>>>>> for each frames. Or a better method is add a meta data to echo picture
>>>>>>> buffer, since the picture output is just the same as the original,
>>>>>>> display won't
Re: [linux-sunxi] [PATCH v2 1/2] media: v4l: Add definitions for the HEVC slice format and controls
Sent from my iPad
> On Jan 24, 2019, at 6:36 PM, Paul Kocialkowski
> wrote:
>
> Hi,
>
>> On Tue, 2019-01-08 at 18:00 +0800, Ayaka wrote:
>>
>> Sent from my iPad
>>
>>> On Jan 8, 2019, at 4:38 PM, Paul Kocialkowski
>>> wrote:
>>>
>>> Hi,
>>>
>>>> On Tue, 2019-01-08 at 09:16 +0800, Ayaka wrote:
>>>>
>>>> Sent from my iPad
>>>>
>>>>> On Jan 7, 2019, at 5:57 PM, Paul Kocialkowski
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>>> On Mon, 2019-01-07 at 11:49 +0800, Randy Li wrote:
>>>>>>> On 12/12/18 8:51 PM, Paul Kocialkowski wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Wed, 2018-12-05 at 21:59 +0100, Jernej Škrabec wrote:
>>>>>>>
>>>>>>>>> +
>>>>>>>>> +#define V4L2_HEVC_DPB_ENTRY_RPS_ST_CURR_BEFORE0x01
>>>>>>>>> +#define V4L2_HEVC_DPB_ENTRY_RPS_ST_CURR_AFTER0x02
>>>>>>>>> +#define V4L2_HEVC_DPB_ENTRY_RPS_LT_CURR0x03
>>>>>>>>> +
>>>>>>>>> +#define V4L2_HEVC_DPB_ENTRIES_NUM_MAX16
>>>>>>>>> +
>>>>>>>>> +struct v4l2_hevc_dpb_entry {
>>>>>>>>> +__u32buffer_tag;
>>>>>>>>> +__u8rps;
>>>>>>>>> +__u8field_pic;
>>>>>>>>> +__u16pic_order_cnt[2];
>>>>>>>>> +};
>>>>>>
>>>>>> Please add a property for reference index, if that rps is not used for
>>>>>> this, some device would request that(not the rockchip one). And
>>>>>> Rockchip's VDPU1 and VDPU2 for AVC would request a similar property.
>>>>>
>>>>> What exactly is that reference index? Is it a bitstream element or
>>>>> something deduced from the bitstream?
>>>>>
>>>> picture order count(POC) for HEVC and frame_num in AVC. I think it is
>>>> the number used in list0(P slice and B slice) and list1(B slice).
>>>
>>> The picture order count is already the last field of the DPB entry
>>> structure. There is one for each field picture.
>> As we are not sure whether there is a field coded slice or CTU, I
>> would hold this part and else about the field.
>
> I'm not sure what you meant here, sorry.
As we talked in IRC, I am not sure the field coded picture is supported in HEVC.
And I don’t why there would be two pic order cnt, a picture can only be used a
short term or a long term reference at one picture decoding
>
>>>>>> Adding another buffer_tag for referring the memory of the motion vectors
>>>>>> for each frames. Or a better method is add a meta data to echo picture
>>>>>> buffer, since the picture output is just the same as the original,
>>>>>> display won't care whether the motion vectors are written the button of
>>>>>> picture or somewhere else.
>>>>>
>>>>> The motion vectors are passed as part of the raw bitstream data, in the
>>>>> slices. Is there a case where the motion vectors are coded differently?
>>>> No, it is an additional cache for decoder, even FFmpeg having such
>>>> data, I think allwinner must output it into somewhere.
>>>
>>> Ah yes I see what you mean! This is handled internally by our driver
>>> and not exposed to userspace. I don't think it would be a good idea to
>>> expose this cache or request that userspace allocates it like a video
>>> buffer.
>>>
>> No, usually the driver should allocate, as the user space have no
>> idea on size of each devices.
>> But for advantage user, application can fix a broken picture with a
>> proper data or analysis a object motion from that.
>> So I would suggest attaching this information to a picture buffer as
>> a meta data.
>
> Right, the driver will allocate chunks of memory for the decoding
> metadata used by the hardware decoder.
>
> Well, I don't think V4L2 has any mechanism to expose this data for now
> and since it's very specific to the hardware implementation, I guess
> the interest in having that is generally pretty low.
>
> That's maybe something that could be added later if someone wants to
> work on it, but I think we are better off keeping this metadata hidden
> by
Re: [linux-sunxi] [PATCH v2 1/2] media: v4l: Add definitions for the HEVC slice format and controls
I forget a important thing, for the rkvdec and rk hevc decoder, it would
requests cabac table, scaling list, picture parameter set and reference
picture storing in one or various of DMA buffers. I am not talking about
the data been parsed, the decoder would requests a raw data.
For the pps and rps, it is possible to reuse the slice header, just let
the decoder know the offset from the bitstream bufer, I would suggest to
add three properties(with sps) for them. But I think we need a method to
mark a OUTPUT side buffer for those aux data.
On 1/8/19 6:00 PM, Ayaka wrote:
Sent from my iPad
On Jan 8, 2019, at 4:38 PM, Paul Kocialkowski
wrote:
Hi,
On Tue, 2019-01-08 at 09:16 +0800, Ayaka wrote:
Sent from my iPad
On Jan 7, 2019, at 5:57 PM, Paul Kocialkowski
wrote:
Hi,
On Mon, 2019-01-07 at 11:49 +0800, Randy Li wrote:
On 12/12/18 8:51 PM, Paul Kocialkowski wrote:
Hi,
On Wed, 2018-12-05 at 21:59 +0100, Jernej Škrabec wrote:
+
+#define V4L2_HEVC_DPB_ENTRY_RPS_ST_CURR_BEFORE0x01
+#define V4L2_HEVC_DPB_ENTRY_RPS_ST_CURR_AFTER0x02
+#define V4L2_HEVC_DPB_ENTRY_RPS_LT_CURR0x03
+
+#define V4L2_HEVC_DPB_ENTRIES_NUM_MAX16
+
+struct v4l2_hevc_dpb_entry {
+__u32buffer_tag;
+__u8rps;
+__u8field_pic;
+__u16pic_order_cnt[2];
+};
Please add a property for reference index, if that rps is not used for
this, some device would request that(not the rockchip one). And
Rockchip's VDPU1 and VDPU2 for AVC would request a similar property.
What exactly is that reference index? Is it a bitstream element or
something deduced from the bitstream?
picture order count(POC) for HEVC and frame_num in AVC. I think it is
the number used in list0(P slice and B slice) and list1(B slice).
The picture order count is already the last field of the DPB entry
structure. There is one for each field picture.
As we are not sure whether there is a field coded slice or CTU, I would hold
this part and else about the field.
Adding another buffer_tag for referring the memory of the motion vectors
for each frames. Or a better method is add a meta data to echo picture
buffer, since the picture output is just the same as the original,
display won't care whether the motion vectors are written the button of
picture or somewhere else.
The motion vectors are passed as part of the raw bitstream data, in the
slices. Is there a case where the motion vectors are coded differently?
No, it is an additional cache for decoder, even FFmpeg having such
data, I think allwinner must output it into somewhere.
Ah yes I see what you mean! This is handled internally by our driver
and not exposed to userspace. I don't think it would be a good idea to
expose this cache or request that userspace allocates it like a video
buffer.
No, usually the driver should allocate, as the user space have no idea on size
of each devices.
But for advantage user, application can fix a broken picture with a proper data
or analysis a object motion from that.
So I would suggest attaching this information to a picture buffer as a meta
data.
+
+struct v4l2_hevc_pred_weight_table {
+__u8luma_log2_weight_denom;
+__s8delta_chroma_log2_weight_denom;
+
+__s8delta_luma_weight_l0[V4L2_HEVC_DPB_ENTRIES_NUM_MAX];
+__s8luma_offset_l0[V4L2_HEVC_DPB_ENTRIES_NUM_MAX];
+__s8delta_chroma_weight_l0[V4L2_HEVC_DPB_ENTRIES_NUM_MAX][2];
+__s8chroma_offset_l0[V4L2_HEVC_DPB_ENTRIES_NUM_MAX][2];
+
+__s8delta_luma_weight_l1[V4L2_HEVC_DPB_ENTRIES_NUM_MAX];
+__s8luma_offset_l1[V4L2_HEVC_DPB_ENTRIES_NUM_MAX];
+__s8delta_chroma_weight_l1[V4L2_HEVC_DPB_ENTRIES_NUM_MAX][2];
+__s8chroma_offset_l1[V4L2_HEVC_DPB_ENTRIES_NUM_MAX][2];
+};
+
Those properties I think are not necessary are applying for the
Rockchip's device, may not work for the others.
Yes, it's possible that some of the elements are not necessary for some
decoders. What we want is to cover all the elements that might be
required for a decoder.
I wonder whether allwinner need that, those sao flag usually ignored
by decoder in design. But more is better than less, it is hard to
extend a v4l2 structure in the future, maybe a new HEVC profile
would bring a new property, it is still too early for HEVC.
Yes this is used by our decoder. The idea is to have all the basic
bitstream elements in the structures (even if some decoders don't use
them all) and add others for extension as separate controls later.
+struct v4l2_ctrl_hevc_slice_params {
+__u32bit_size;
+__u32data_bit_offset;
+
+/* ISO/IEC 23008-2, ITU-T Rec. H.265: NAL unit header */
+__u8nal_unit_type;
+__u8nuh_temporal_id_plus1;
+
+/* ISO/IEC 23008-2, ITU-T Rec. H.265: General slice segment header */
+__u8slice_type;
+__u8colour_plane_id;
+__u16slice_pic_order_cnt;
+__u8slice_sao_luma_flag
Re: [linux-sunxi] [PATCH v2 1/2] media: v4l: Add definitions for the HEVC slice format and controls
Sent from my iPad
> On Jan 8, 2019, at 4:38 PM, Paul Kocialkowski
> wrote:
>
> Hi,
>
>> On Tue, 2019-01-08 at 09:16 +0800, Ayaka wrote:
>>
>> Sent from my iPad
>>
>>> On Jan 7, 2019, at 5:57 PM, Paul Kocialkowski
>>> wrote:
>>>
>>> Hi,
>>>
>>>>> On Mon, 2019-01-07 at 11:49 +0800, Randy Li wrote:
>>>>> On 12/12/18 8:51 PM, Paul Kocialkowski wrote:
>>>>> Hi,
>>>>>
>>>>> On Wed, 2018-12-05 at 21:59 +0100, Jernej Škrabec wrote:
>>>>>
>>>>>>> +
>>>>>>> +#define V4L2_HEVC_DPB_ENTRY_RPS_ST_CURR_BEFORE0x01
>>>>>>> +#define V4L2_HEVC_DPB_ENTRY_RPS_ST_CURR_AFTER0x02
>>>>>>> +#define V4L2_HEVC_DPB_ENTRY_RPS_LT_CURR0x03
>>>>>>> +
>>>>>>> +#define V4L2_HEVC_DPB_ENTRIES_NUM_MAX16
>>>>>>> +
>>>>>>> +struct v4l2_hevc_dpb_entry {
>>>>>>> +__u32buffer_tag;
>>>>>>> +__u8rps;
>>>>>>> +__u8field_pic;
>>>>>>> +__u16pic_order_cnt[2];
>>>>>>> +};
>>>>
>>>> Please add a property for reference index, if that rps is not used for
>>>> this, some device would request that(not the rockchip one). And
>>>> Rockchip's VDPU1 and VDPU2 for AVC would request a similar property.
>>>
>>> What exactly is that reference index? Is it a bitstream element or
>>> something deduced from the bitstream?
>>>
>> picture order count(POC) for HEVC and frame_num in AVC. I think it is
>> the number used in list0(P slice and B slice) and list1(B slice).
>
> The picture order count is already the last field of the DPB entry
> structure. There is one for each field picture.
As we are not sure whether there is a field coded slice or CTU, I would hold
this part and else about the field.
>
>>>> Adding another buffer_tag for referring the memory of the motion vectors
>>>> for each frames. Or a better method is add a meta data to echo picture
>>>> buffer, since the picture output is just the same as the original,
>>>> display won't care whether the motion vectors are written the button of
>>>> picture or somewhere else.
>>>
>>> The motion vectors are passed as part of the raw bitstream data, in the
>>> slices. Is there a case where the motion vectors are coded differently?
>> No, it is an additional cache for decoder, even FFmpeg having such
>> data, I think allwinner must output it into somewhere.
>
> Ah yes I see what you mean! This is handled internally by our driver
> and not exposed to userspace. I don't think it would be a good idea to
> expose this cache or request that userspace allocates it like a video
> buffer.
>
No, usually the driver should allocate, as the user space have no idea on size
of each devices.
But for advantage user, application can fix a broken picture with a proper data
or analysis a object motion from that.
So I would suggest attaching this information to a picture buffer as a meta
data.
>>>>>>> +
>>>>>>> +struct v4l2_hevc_pred_weight_table {
>>>>>>> +__u8luma_log2_weight_denom;
>>>>>>> +__s8delta_chroma_log2_weight_denom;
>>>>>>> +
>>>>>>> +__s8delta_luma_weight_l0[V4L2_HEVC_DPB_ENTRIES_NUM_MAX];
>>>>>>> +__s8luma_offset_l0[V4L2_HEVC_DPB_ENTRIES_NUM_MAX];
>>>>>>> +__s8delta_chroma_weight_l0[V4L2_HEVC_DPB_ENTRIES_NUM_MAX][2];
>>>>>>> +__s8chroma_offset_l0[V4L2_HEVC_DPB_ENTRIES_NUM_MAX][2];
>>>>>>> +
>>>>>>> +__s8delta_luma_weight_l1[V4L2_HEVC_DPB_ENTRIES_NUM_MAX];
>>>>>>> +__s8luma_offset_l1[V4L2_HEVC_DPB_ENTRIES_NUM_MAX];
>>>>>>> +__s8delta_chroma_weight_l1[V4L2_HEVC_DPB_ENTRIES_NUM_MAX][2];
>>>>>>> +__s8chroma_offset_l1[V4L2_HEVC_DPB_ENTRIES_NUM_MAX][2];
>>>>>>> +};
>>>>>>> +
>>>> Those properties I think are not necessary are applying for the
>>>> Rockchip's device, may not work for the others.
>>>
>>> Yes, it's possible that some of the elements are not necessary for some
>>> decoders. What we want is to cover all the elements that might be
>>> required for a decoder.
>> I wonder whether allwinner
Re: [linux-sunxi] [PATCH v2 1/2] media: v4l: Add definitions for the HEVC slice format and controls
; >>>>> +__u8five_minus_max_num_merge_cand; >>>>> +__u8use_integer_mv_flag; >>>>> +__s8slice_qp_delta; >>>>> +__s8slice_cb_qp_offset; >>>>> +__s8slice_cr_qp_offset; >>>>> +__s8slice_act_y_qp_offset; >>>>> +__s8slice_act_cb_qp_offset; >>>>> +__s8slice_act_cr_qp_offset; >>>>> +__u8slice_deblocking_filter_disabled_flag; >>>>> +__s8slice_beta_offset_div2; >>>>> +__s8slice_tc_offset_div2; >>>>> +__u8slice_loop_filter_across_slices_enabled_flag; >>>>> + >>>>> +/* ISO/IEC 23008-2, ITU-T Rec. H.265: Picture timing SEI message */ >>>>> +__u8pic_struct; >> I think the decoder doesn't care about this, it is used for display. > > The purpose of this field is to indicate whether the current picture is > a progressive frame or an interlaced field picture, which is useful for > decoding. > > At least our decoder has a register field to indicate frame/top > field/bottom field, so we certainly need to keep the info around. > Looking at the spec and the ffmpeg implementation, it looks like this > flag of the bitstream is the usual way to report field coding. It depends whether the decoder cares about scan type or more, I wonder prefer general_interlaced_source_flag for just scan type, it would be better than reading another SEL. > > Cheers, > > Paul Randy “ayaka” LI > >>>>> + >>>>> +/* ISO/IEC 23008-2, ITU-T Rec. H.265: General slice segment header */ >>>>> +struct v4l2_hevc_dpb_entry dpb[V4L2_HEVC_DPB_ENTRIES_NUM_MAX]; >>>>> +__u8num_active_dpb_entries; >>>>> +__u8ref_idx_l0[V4L2_HEVC_DPB_ENTRIES_NUM_MAX]; >>>>> +__u8ref_idx_l1[V4L2_HEVC_DPB_ENTRIES_NUM_MAX]; >>>>> + >>>>> +__u8num_rps_poc_st_curr_before; >>>>> +__u8num_rps_poc_st_curr_after; >>>>> +__u8num_rps_poc_lt_curr; >>>>> + >>>>> +/* ISO/IEC 23008-2, ITU-T Rec. H.265: Weighted prediction parameter >>>>> */ >>>>> +struct v4l2_hevc_pred_weight_table pred_weight_table; >>>>> +}; >>>>> + >>>>> #endif > -- > Paul Kocialkowski, Bootlin (formerly Free Electrons) > Embedded Linux and kernel engineering > https://bootlin.com > ___ devel mailing list de...@linuxdriverproject.org http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
