Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Michal, On Friday 18 February 2011 15:05:42 Michal Nazarewicz wrote: > > On Friday 18 February 2011 14:19:44 Michal Nazarewicz wrote: > >> Cache operations are always needed, aren't they? Whatever you do, you > >> will always have to handle cache coherency (in one way or another) so > >> there's nothing we can do about it, or is there? > > On Fri, 18 Feb 2011 14:21:53 +0100, Laurent Pinchart wrote: > > To achieve low latency still image capture, you need to minimize the time > > spent reconfiguring the device from viewfinder to still capture. Cache > > cleaning is always needed, but you can prequeue buffers you can clean the > > cache in advance, avoiding an extra delay when the user presses the still > > image capture button. > > If there is enough time to perform those operation while preview is shown > (ie. several frames pare second), why would there not be enough time to > perform those operations for still image capture? For still image capture you want to minimize the shot-to-snapshot delay (delay between the time when the user presses the shot button and the time when the image is captured). Allocating memory for the buffers and prequeuing them should thus be done before the user presses the shot button. > If I understand you correctly, what you are describing is a situation where > one has set of buffers for preview and a buffer for still image laying > around waiting to be used, right? That's correct. That's what we currently do with the OMAP3 ISP. > Such scheme will of course work, but I'm just suggesting to think of > a scheme where the unused buffer for still image is reused for preview > frames when preview is shown. That will work as well, but it will increase the shot-to-snapshot delay as cache management will need to be performed after the user presses the shot button. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
On Friday 18 February 2011 14:19:44 Michal Nazarewicz wrote: Cache operations are always needed, aren't they? Whatever you do, you will always have to handle cache coherency (in one way or another) so there's nothing we can do about it, or is there? On Fri, 18 Feb 2011 14:21:53 +0100, Laurent Pinchart wrote: To achieve low latency still image capture, you need to minimize the time spent reconfiguring the device from viewfinder to still capture. Cache cleaning is always needed, but you can prequeue buffers you can clean the cache in advance, avoiding an extra delay when the user presses the still image capture button. If there is enough time to perform those operation while preview is shown (ie. several frames pare second), why would there not be enough time to perform those operations for still image capture? If I understand you correctly, what you are describing is a situation where one has set of buffers for preview and a buffer for still image laying around waiting to be used, right? Such scheme will of course work, but I'm just suggesting to think of a scheme where the unused buffer for still image is reused for preview frames when preview is shown. -- Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michal "mina86" Nazarewicz(o o) ooo +--ooO--(_)--Ooo-- -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Michal, On Friday 18 February 2011 14:19:44 Michal Nazarewicz wrote: > On Fri, 18 Feb 2011 13:57:25 +0100, Laurent Pinchart wrote: > > On Friday 18 February 2011 12:37:24 Michal Nazarewicz wrote: > > > > [snip] > > > >> What I'm trying to say is that it would be best if one could configure > >> the device in such a way that switching between modes would not require > >> the device to free buffers (even though in user space they could be > >> inaccessible). > >> > >> > >> This is what I have in mind the usage would look like: > >> > >> 1. Open device > >> > >>Kernel creates some control structures, the usual stuff. > >> > >> 2. Initialize multi-format (specifying what formats user space will > >> use). > >> > >>Kernel calculates amount of memory needed for the most > >>demanding format and allocates it. > > > > Don't forget that applications can also use USERPTR. We need a > > low-latency solution for that as well. > > That would probably work best if user provided one big buffer. Again, > I don't know how this maps to V4L API. > > >> 3. Set format (restricted to one of formats specified in step 2) > >> > >>Kernel has memory already allocated, so it only needs to split > >>it to buffers needed in given format. > >> > >> 4. Allocate buffers. > >> > >>Kernel allocates memory needed for most demanding format > >>(calculated in step 2). > >>Once memory is allocated, splits it to buffers needed in > >>given format. > >> > >> 5. Do the loop... queue, dequeue, all the usual stuff. > >> > >>Kernel instructs device to handle buffers, the usual stuff. > > > > When buffers are queued cache needs to be cleaned. This is an expensive > > operation, and we need to be able to pre-queue (or at least pre-clean) > > buffers. > > Cache operations are always needed, aren't they? Whatever you do, you > will always have to handle cache coherency (in one way or another) so > there's nothing we can do about it, or is there? To achieve low latency still image capture, you need to minimize the time spent reconfiguring the device from viewfinder to still capture. Cache cleaning is always needed, but you can prequeue buffers you can clean the cache in advance, avoiding an extra delay when the user presses the still image capture button. > >> 6. Free buffers. > >> > >>Kernel space destroys the buffers needed for given format > >>but DOES NOT free memory. > >> > >> 7. If not done, go to step 3. > >> > >> 8. Finish multi-format mode. > >> > >>Kernel actually frees the memory. > >> > >> 9. Close the device. > >> > >> A V4L device driver could just ignore step 2 and 7 and work in the less > >> optimal mode. > >> > >> If I understand V4L2 correctly, the API does not allow for step 2 and 8. > >> In theory, they could be merged with step 1 and 9 respectively, I don't > >> know id that feasible though. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
On Fri, 18 Feb 2011 13:57:25 +0100, Laurent Pinchart wrote: Hi Michal, On Friday 18 February 2011 12:37:24 Michal Nazarewicz wrote: [snip] What I'm trying to say is that it would be best if one could configure the device in such a way that switching between modes would not require the device to free buffers (even though in user space they could be inaccessible). This is what I have in mind the usage would look like: 1. Open device Kernel creates some control structures, the usual stuff. 2. Initialize multi-format (specifying what formats user space will use). Kernel calculates amount of memory needed for the most demanding format and allocates it. Don't forget that applications can also use USERPTR. We need a low-latency solution for that as well. That would probably work best if user provided one big buffer. Again, I don't know how this maps to V4L API. 3. Set format (restricted to one of formats specified in step 2) Kernel has memory already allocated, so it only needs to split it to buffers needed in given format. 4. Allocate buffers. Kernel allocates memory needed for most demanding format (calculated in step 2). Once memory is allocated, splits it to buffers needed in given format. 5. Do the loop... queue, dequeue, all the usual stuff. Kernel instructs device to handle buffers, the usual stuff. When buffers are queued cache needs to be cleaned. This is an expensive operation, and we need to be able to pre-queue (or at least pre-clean) buffers. Cache operations are always needed, aren't they? Whatever you do, you will always have to handle cache coherency (in one way or another) so there's nothing we can do about it, or is there? 6. Free buffers. Kernel space destroys the buffers needed for given format but DOES NOT free memory. 7. If not done, go to step 3. 8. Finish multi-format mode. Kernel actually frees the memory. 9. Close the device. A V4L device driver could just ignore step 2 and 7 and work in the less optimal mode. If I understand V4L2 correctly, the API does not allow for step 2 and 8. In theory, they could be merged with step 1 and 9 respectively, I don't know id that feasible though. -- Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michal "mina86" Nazarewicz(o o) ooo +--ooO--(_)--Ooo-- -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Michal, On Friday 18 February 2011 12:37:24 Michal Nazarewicz wrote: [snip] > What I'm trying to say is that it would be best if one could configure the > device in such a way that switching between modes would not require the > device to free buffers (even though in user space they could be > inaccessible). > > > This is what I have in mind the usage would look like: > > 1. Open device > Kernel creates some control structures, the usual stuff. > 2. Initialize multi-format (specifying what formats user space will use). > Kernel calculates amount of memory needed for the most > demanding format and allocates it. Don't forget that applications can also use USERPTR. We need a low-latency solution for that as well. > 3. Set format (restricted to one of formats specified in step 2) > Kernel has memory already allocated, so it only needs to split > it to buffers needed in given format. > 4. Allocate buffers. > Kernel allocates memory needed for most demanding format > (calculated in step 2). > Once memory is allocated, splits it to buffers needed in > given format. > 5. Do the loop... queue, dequeue, all the usual stuff. > Kernel instructs device to handle buffers, the usual stuff. When buffers are queued cache needs to be cleaned. This is an expensive operation, and we need to be able to pre-queue (or at least pre-clean) buffers. > 6. Free buffers. > Kernel space destroys the buffers needed for given format > but DOES NOT free memory. > > 7. If not done, go to step 3. > > 8. Finish multi-format mode. > Kernel actually frees the memory. > > 9. Close the device. > > A V4L device driver could just ignore step 2 and 7 and work in the less > optimal mode. > > If I understand V4L2 correctly, the API does not allow for step 2 and 8. > In theory, they could be merged with step 1 and 9 respectively, I don't > know id that feasible though. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Em 18-02-2011 08:31, Laurent Pinchart escreveu: It's a trade-off between memory and speed. Preallocating still image capture buffers will give you better snapshot performances, at the expense of memory. The basic problems we have here is that taking snapshots is slow with the current API if we need to stop capture, free buffers, change the format, allocate new buffers (and perform cache management operations) and restart the stream. To fix this we're considering a way to preallocate still image capture buffers, but I'm open to proposals for other ways to solve the issue :-) On Fri, 18 Feb 2011 13:33:12 +0100, Mauro Carvalho Chehab wrote: From the above operations, considering that CMA is used to reserve a non-shared memory with enough space for the new buffer size/qtd, I don't think that the most expensive operation would be to realloc the memory. The logic to stop/start streaming seems to be the most consuming one, as driver will need to wait for the current I/O operation to complete, and this can take hundreds of milisseconds (the duration of one frame). How much time would CMA need to free and re-allocate the buffers for, let's say, something in the range of 1-10 MB, on a pre-allocated, non shared memory space? Like I said, if memory is shared with page allocator this can potentially take a lot of time. If device driver could preallocate some space and then spit it into buffers when needed, it could be a better solution then to free chunk and allocate a new one. CMA was not designed with very low latency in mind. -- Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michal "mina86" Nazarewicz(o o) ooo +--ooO--(_)--Ooo-- -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Mauro, On Friday 18 February 2011 13:33:12 Mauro Carvalho Chehab wrote: > Em 18-02-2011 08:31, Laurent Pinchart escreveu: > > It's a trade-off between memory and speed. Preallocating still image > > capture buffers will give you better snapshot performances, at the > > expense of memory. > > > > The basic problems we have here is that taking snapshots is slow with the > > current API if we need to stop capture, free buffers, change the format, > > allocate new buffers (and perform cache management operations) and > > restart the stream. To fix this we're considering a way to preallocate > > still image capture buffers, but I'm open to proposals for other ways to > > solve the issue :-) > > From the above operations, considering that CMA is used to reserve a > non-shared memory with enough space for the new buffer size/qtd, I don't > think that the most expensive operation would be to realloc the memory. > > The logic to stop/start streaming seems to be the most consuming one, as > driver will need to wait for the current I/O operation to complete, and > this can take hundreds of milisseconds (the duration of one frame). > > How much time would CMA need to free and re-allocate the buffers for, let's > say, something in the range of 1-10 MB, on a pre-allocated, non shared > memory space? CMA won't solve the problem, as not all drivers require continuous memory. The OMAP3 ISP has an IOMMU, and two very time-consuming tasks are mapping the memory to the IOMMU and cleaning the cache. This is why preallocation *and* prequeuing is needed. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Em 18-02-2011 08:31, Laurent Pinchart escreveu: > It's a trade-off between memory and speed. Preallocating still image capture > buffers will give you better snapshot performances, at the expense of memory. > > The basic problems we have here is that taking snapshots is slow with the > current API if we need to stop capture, free buffers, change the format, > allocate new buffers (and perform cache management operations) and restart > the > stream. To fix this we're considering a way to preallocate still image > capture > buffers, but I'm open to proposals for other ways to solve the issue :-) >From the above operations, considering that CMA is used to reserve a non-shared memory with enough space for the new buffer size/qtd, I don't think that the most expensive operation would be to realloc the memory. The logic to stop/start streaming seems to be the most consuming one, as driver will need to wait for the current I/O operation to complete, and this can take hundreds of milisseconds (the duration of one frame). How much time would CMA need to free and re-allocate the buffers for, let's say, something in the range of 1-10 MB, on a pre-allocated, non shared memory space? Cheers Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
On Fri, 18 Feb 2011 11:31:30 +0100, Laurent Pinchart wrote: Hi Michal, On Friday 18 February 2011 00:09:51 Michal Nazarewicz wrote: >>> On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote: There's an additional problem with that: assume that streaming is happening, and a S_FMT changing the resolution was sent. There's no way to warrant that the very next frame will have the new resolution. So, a meta-data with the frame resolution (and format) would be needed. >> >> Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu: >>> Sorry, we are not going to allow format changes during a running >>> capture. You have to stop streaming, set new formats (possibly >>> switch to another queue) and restart streaming. >>> >>> What am I missing? > > On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote: >> If you're stopping the stream, the current API will work as-is. >> >> If all of your concerns is about reserving a bigger buffer queue, I >> think that one of the reasons for the CMA allocator it for such usage. From: Guennadi Liakhovetski > Not just bigger, say, with our preview / still-shot example, we would > have one queue with a larger number of small buffers for drop-free > preview, and a small number of larger buffers for still images. Ie. waste memory? As in you have both those queues allocated but only one is used at given time? It's a trade-off between memory and speed. Preallocating still image capture buffers will give you better snapshot performances, at the expense of memory. The basic problems we have here is that taking snapshots is slow with the current API if we need to stop capture, free buffers, change the format, allocate new buffers (and perform cache management operations) and restart the stream. To fix this we're considering a way to preallocate still image capture buffers, but I'm open to proposals for other ways to solve the issue :-) What I'm trying to say is that it would be best if one could configure the device in such a way that switching between modes would not require the device to free buffers (even though in user space they could be inaccessible). This is what I have in mind the usage would look like: 1. Open device Kernel creates some control structures, the usual stuff. 2. Initialize multi-format (specifying what formats user space will use). Kernel calculates amount of memory needed for the most demanding format and allocates it. 3. Set format (restricted to one of formats specified in step 2) Kernel has memory already allocated, so it only needs to split it to buffers needed in given format. 4. Allocate buffers. Kernel allocates memory needed for most demanding format (calculated in step 2). Once memory is allocated, splits it to buffers needed in given format. 5. Do the loop... queue, dequeue, all the usual stuff. Kernel instructs device to handle buffers, the usual stuff. 6. Free buffers. Kernel space destroys the buffers needed for given format but DOES NOT free memory. 7. If not done, go to step 3. 8. Finish multi-format mode. Kernel actually frees the memory. 9. Close the device. A V4L device driver could just ignore step 2 and 7 and work in the less optimal mode. If I understand V4L2 correctly, the API does not allow for step 2 and 8. In theory, they could be merged with step 1 and 9 respectively, I don't know id that feasible though. -- Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michal "mina86" Nazarewicz(o o) ooo +--ooO--(_)--Ooo-- -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Michal, On Friday 18 February 2011 00:09:51 Michal Nazarewicz wrote: > >>> On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote: > There's an additional problem with that: assume that streaming is > happening, and a S_FMT changing the resolution was sent. There's > no way to warrant that the very next frame will have the new > resolution. So, a meta-data with the frame resolution (and format) > would be needed. > >> > >> Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu: > >>> Sorry, we are not going to allow format changes during a running > >>> capture. You have to stop streaming, set new formats (possibly > >>> switch to another queue) and restart streaming. > >>> > >>> What am I missing? > > > > On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote: > >> If you're stopping the stream, the current API will work as-is. > >> > >> If all of your concerns is about reserving a bigger buffer queue, I > >> think that one of the reasons for the CMA allocator it for such usage. > > From: Guennadi Liakhovetski > > > Not just bigger, say, with our preview / still-shot example, we would > > have one queue with a larger number of small buffers for drop-free > > preview, and a small number of larger buffers for still images. > > Ie. waste memory? As in you have both those queues allocated but only > one is used at given time? It's a trade-off between memory and speed. Preallocating still image capture buffers will give you better snapshot performances, at the expense of memory. The basic problems we have here is that taking snapshots is slow with the current API if we need to stop capture, free buffers, change the format, allocate new buffers (and perform cache management operations) and restart the stream. To fix this we're considering a way to preallocate still image capture buffers, but I'm open to proposals for other ways to solve the issue :-) > > Currently you would have to allocate a large number of large buffers, > > which would waste memory. Or you would have to reallocate the queue, > > losing time. > > > > AFAICS, CMA doesn't manage our memory for us. It only provides an API > > to reserve memory for various uses with various restrictions (alignment, > > etc.) and use different allocators to obtain that memory. > > I'm not sure if I understand you here. CMA has some API for reserving > memory at boot time but it sure does manage this reserved memory, ie. > when system is running you can allocate chunks of memory from this > reserved block. > > Also note, that latest CMA uses only one allocator. > > > So, are you suggesting, that with that in place, we would > > first allocate the preview queue from this memory, then free it, when > > switching to snapshooting, allocate our large-buffer queue from the > > _same_ memory, capture images, free and allocate preview queue again? > > Would that be fast enough? > > If CMA is considered, the most important thing to note is that CMA may > share memory with page allocator (so that other parts of the system can > use it if CMA-compatible devices are not using it). When CMA allocates > memory chunk it may potentially need to migrate memory pages which may > take so time (there is room for improvement, but still). > > Sharing can be disabled in which case allocation should be quite fast > (the last CMA patchset uses a first-fit bitmap-based gen_allocator API > but O(log n) best-fit algorithm can easily used instead). > > To sum things up, if sharing is disabled, CMA should be able to fulfil > your requirements, however it may be undesirable as it wastes space. > If sharing is enabled, on the other hand, the delay may potentially be > noticeable. > > > In fact, it would be kind of smart to reuse the same memory for both > > queues, but if we could do it without re-allocations?... > > What I would do is allocate a few big buffers and when needed divide them > into smaller chunks (or even allocate one big block and later divide it in > whatever way needed). I'm not sure if such usage would map well to V4L2 > API. > > This usage is, as a matter of fact, supported by CMA. You can allocate > a big block and then run cma_create() on it to create a new CMA context. > Using this context you can allocate a lot of small blocks, then free them > all, to finally allocate few big blocks. > > Again, I'm not sure how it maps to V4L2 API. If you can change formats > while retaining V4L device instance's state, this should be doable. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Mauro,
On Thursday 17 February 2011 20:04:33 Mauro Carvalho Chehab wrote:
> Em 16-02-2011 08:35, Guennadi Liakhovetski escreveu:
> But from the point of view of the application it makes more sense to
> actually have two video nodes. The only difference is that when one
> starts streaming it pre-empts the other.
> >>>
> >>> Well, I don't think I like it all that much... One reason - yes, two
> >>> "independent" video device-nodes, which actually only can stream in
> >>> turn seems somewhat counterintuitive to me, specifically, I don't
> >>> think there is a way to tell the application about this. What if we
> >>> have more than two video devices on the system? Or what if you need
> >>> more than two video queues / formats? Or actually only one? The kernel
> >>> doesn't know initially how many, this is a dynamic resource, not a
> >>> fixed static interface, IMHO.
> >>
> >> I agree with this, which is why I don't think two (or more) video nodes
> >> would be a good solution.
>
> I agree. Video nodes shouldn't be bind to an specific format. A device with
> 2 video nodes should be able to retrieve images from two independent video
> sources.
>
> Unfortunately, ivtv driver was merged with the bad concept of one video
> node per different type of formats (partially my fault: I remember I
> commented about it at the time it was submitted, but, as its merge took a
> long time, and there were several other issues that were needed to be
> solved there, I ended by giving up and letting it to come with this API
> non-compliance, hoping that a fix would happen at the next kernel release.
> Unfortunately, it was never fixed).
>
> >> We've hit the exact same issue with the OMAP3 ISP driver. Our current
> >> solution is to allocate video buffer queues at the file handle level
> >> instead of the video node level. Applications can open the same video
> >> device twice and allocate buffers for the viewfinder on one of the
> >> instances and for still image capture on the other. When switching from
> >> viewfinder to still image capture, all it needs to do (beside obviously
> >> reconfiguring the media controller pipeline if required) is to issue
> >> VIDIOC_STREAMOFF on the viewfinder file handle and VIDIOC_STREAMON on
> >> the still capture file handle.
>
> This seems to be the proper way. solution (2) of using read()/mmap() is
> just an special case of per-file handle stream control, as applications
> that used this approach in the past were, in fact, using two opens, one
> for read, and another for mmap (to be clear, I'm not in favor of 2, I'm
> just saying that a per-file handle solution will allow (2) also).
>
> >> One issue with this approach is that allocating buffers requires
> >> knowledge of the format and resolution. The driver traditionally
> >> computes the buffer size from the parameters set by VIDIOC_S_FMT. This
> >> would prevent an application opening the video node a second time and
> >> setting up buffers for still image capture a second time while the
> >> viewfinder is running, as the VIDIOC_S_FMT on the second file handle
> >> won't be allowed then.
> >>
> >> Changes to the V4L2 spec would be needed to allow this to work properly.
> >
> > The spec is actually saying about the S_FMT ioctl():
> >
> > "On success the driver may program the hardware, allocate resources and
> > generally prepare for data exchange."
> >
> > - _may_ program the hardware. So, if we don't do that and instead only
> > verify the format and store it for future activation upon a call to
> > STREAMON we are not violating the spec, thus, no change is required.
> > OTOH, another spec sections "V4L2 close()" says:
> >
> > "data format parameters, current input or output, control values or other
> > properties remain unchanged."
> >
> > which is usually interpreted as "a sequence open(); ioctl(S_FMT);
> > close(); open(); ioctl(STREAMON);" _must_ use the format, set by the
> > call to S_FMT, which is not necessarily logical, if we adopt the
> > per-file-descriptor format / stream approach.
>
> Every time a "may" appears on a spec, we'll have troubles, as some drivers
> will follow the "may" and others won't follow. Changing the behaviour will
> likely cause regressions, whatever direction is taken.
>
> One alternative would be to have a better way to negotiate features than
> what's provided by QUERYCAP. If we look for some protocols with a long
> life, like telnet, they don't have a one-way to check/set capabilities.
> Instead, both parties should present their capabilities and the client
> need to negotiate what he wants.
>
> We could do something like:
>
> ret = ioctl(fd, VIDIOC_QUERYCAP, &cap);
> if (cap.capabilities & CAN_PER_FD_FMT) {
> setcap.capabilities |= SHOULD_PER_FD_FMT;
> ret = ioctl(fd, VIDIOC_SETCAP, &setcap);
> }
>
> To be sure that the kernel driver will behave fine. Yet, in this particular
> case, this would mean that drivers or core w
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Guennadi,
On Thursday 17 February 2011 17:46:15 Guennadi Liakhovetski wrote:
> Hi all
>
> Let me try to further elaborate on this topic. So far, I think, the
> following provides the cleanest solution to our quick format-switching /
> multiple video-queue problem:
>
> On Wed, 16 Feb 2011, Guennadi Liakhovetski wrote:
> > (2) cleanly separate setting video data format (S_FMT) from specifying
> > the allocated buffer size.
>
> The control flow then would look like:
>
> struct v4l2_requestbuffers req;
>
> fd = open();
>
> /* initialise and configure the first "still-shot" buffer-queue */
> bufq.id = -EINVAL;
> ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);
If we're going to introduce new ioctls, I think we should do it right from the
start. Your proposal looks like an interim solution that would be deprecated
when buffer pools will be implemented. I think we should work directly on
buffer pools instead.
> /*
>* bufq.id now contains a handle to a newly allocated videobuf
>* queue, it also becomes the current queue.
>*/
>
> still_qid = buf.id;
>
> /*
>* specify explicitly required buffer size - use one of reserved
>* fields
>*/
> req.size = still_x * still_y * bpp;
> ret = ioctl(fd, VIDIOC_REQBUFS, &req);
>
> for (i = 0; i < req.count; i++) {
> buf.index = i;
> ret = ioctl(fd, VIDIOC_QUERYBUF, &buf);
> start = mmap();
> ret = ioctl(fd, VIDIOC_QBUF, &buf);
> }
>
> /* Allocate a second "preview" queue */
>
> bufq.id = -EINVAL;
> ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);
>
> preview_qid = bufq.id;
>
> req.size = preview_x * preview_y * bpp;
> ret = ioctl(fd, VIDIOC_REQBUFS, &req);
>
> for (i = 0; i < req.count; i++) {
> buf.index = i;
> ret = ioctl(fd, VIDIOC_QUERYBUF, &buf);
> start = mmap();
> ret = ioctl(fd, VIDIOC_QBUF, &buf);
> }
>
> /* Stay on the preview queue */
> ret = ioctl(fd, VIDIOC_STREAMON, &type);
>
> for (i = 0; i < n; i++) {
> ret = ioctl(fd, VIDIOC_DQBUF, &buf);
> ret = ioctl(fd, VIDIOC_QBUF, &buf);
> }
>
> ret = ioctl(fd, VIDIOC_STREAMOFF, &type);
>
> /* Switch to the still-shot queue */
> bufq.id = still_qid;
> ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);
>
> ret = ioctl(fd, VIDIOC_STREAMON, &type);
>
> for (i = 0; i < n; i++) {
> ret = ioctl(fd, VIDIOC_DQBUF, &buf);
> ret = ioctl(fd, VIDIOC_QBUF, &buf);
> }
>
> ret = ioctl(fd, VIDIOC_STREAMOFF, &type);
>
> And so on. If the above is _conceptually_ acceptable, we need 1 to 2 new
> ioctl()s and an extension to the VIDIOC_REQBUFS ioctl() to specify the
> buffer size. I wrote "1 or 2 new ioctl()s," because above the new
> VIDIOC_BUFQ_SELECT ioctl() performs two functions: if the handle is
> negative, it allocates a new queue, if it containes a valid (per-device or
> per file-handle?) queue ID, it switches to it. We might also decide to use
> two ioctl()s for these two functions.
--
Regards,
Laurent Pinchart
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote: There's an additional problem with that: assume that streaming is happening, and a S_FMT changing the resolution was sent. There's no way to warrant that the very next frame will have the new resolution. So, a meta-data with the frame resolution (and format) would be needed. Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu: Sorry, we are not going to allow format changes during a running capture. You have to stop streaming, set new formats (possibly switch to another queue) and restart streaming. What am I missing? On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote: If you're stopping the stream, the current API will work as-is. If all of your concerns is about reserving a bigger buffer queue, I think that one of the reasons for the CMA allocator it for such usage. From: Guennadi Liakhovetski Not just bigger, say, with our preview / still-shot example, we would have one queue with a larger number of small buffers for drop-free preview, and a small number of larger buffers for still images. Ie. waste memory? As in you have both those queues allocated but only one is used at given time? Currently you would have to allocate a large number of large buffers, which would waste memory. Or you would have to reallocate the queue, losing time. AFAICS, CMA doesn't manage our memory for us. It only provides an API to reserve memory for various uses with various restrictions (alignment, etc.) and use different allocators to obtain that memory. I'm not sure if I understand you here. CMA has some API for reserving memory at boot time but it sure does manage this reserved memory, ie. when system is running you can allocate chunks of memory from this reserved block. Also note, that latest CMA uses only one allocator. So, are you suggesting, that with that in place, we would first allocate the preview queue from this memory, then free it, when switching to snapshooting, allocate our large-buffer queue from the _same_ memory, capture images, free and allocate preview queue again? Would that be fast enough? If CMA is considered, the most important thing to note is that CMA may share memory with page allocator (so that other parts of the system can use it if CMA-compatible devices are not using it). When CMA allocates memory chunk it may potentially need to migrate memory pages which may take so time (there is room for improvement, but still). Sharing can be disabled in which case allocation should be quite fast (the last CMA patchset uses a first-fit bitmap-based gen_allocator API but O(log n) best-fit algorithm can easily used instead). To sum things up, if sharing is disabled, CMA should be able to fulfil your requirements, however it may be undesirable as it wastes space. If sharing is enabled, on the other hand, the delay may potentially be noticeable. In fact, it would be kind of smart to reuse the same memory for both queues, but if we could do it without re-allocations?... What I would do is allocate a few big buffers and when needed divide them into smaller chunks (or even allocate one big block and later divide it in whatever way needed). I'm not sure if such usage would map well to V4L2 API. This usage is, as a matter of fact, supported by CMA. You can allocate a big block and then run cma_create() on it to create a new CMA context. Using this context you can allocate a lot of small blocks, then free them all, to finally allocate few big blocks. Again, I'm not sure how it maps to V4L2 API. If you can change formats while retaining V4L device instance's state, this should be doable. -- Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michal "mina86" Nazarewicz(o o) ooo +--ooO--(_)--Ooo-- -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote: > Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu: > > Hi Mauro > > > > Thanks for your comments. > > > > On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote: > > > >> Em 16-02-2011 08:35, Guennadi Liakhovetski escreveu: > > > > [snip] > > > >>> (2) cleanly separate setting video data format (S_FMT) from specifying > >>> the > >>> allocated buffer size. > >> > >> This would break existing applications. Too late for that, except if > >> negotiated > >> with a "SETCAP" like approach. > > > > Sorry, don't see how my proposal from my last mail would change existing > > applications. As long as no explicit buffer-queue management is performed, > > no new queues are allocated, the driver will just implicitly allocate one > > queue and use it. I.e., no change in behaviour. > > Using the same ioctl to explicitly or to implicitly allocating memory > depending > on the context would make the API more complicated than it should be. Sorry again, of course, we're adding new functionality, so, it may well happen, that the API will become more complicated too. But that was not the question. The question was - would it break any existing users? And it looks like it wouldn't, at the same time giving new users the required additional flexibility and functionality. > >> There's an additional problem with that: assume that streaming is > >> happening, > >> and a S_FMT changing the resolution was sent. There's no way to warrant > >> that > >> the very next frame will have the new resolution. So, a meta-data with the > >> frame resolution (and format) would be needed. > > > > Sorry, we are not going to allow format changes during a running capture. > > You have to stop streaming, set new formats (possibly switch to another > > queue) and restart streaming. > > > What am I missing? > > If you're stopping the stream, the current API will work as-is. > > If all of your concerns is about reserving a bigger buffer queue, I think > that one > of the reasons for the CMA allocator it for such usage. Not just bigger, say, with our preview / still-shot example, we would have one queue with a larger number of small buffers for drop-free preview, and a small number of larger buffers for still images. Currently you would have to allocate a large number of large buffers, which would waste memory. Or you would have to reallocate the queue, losing time. AFAICS, CMA doesn't manage our memory for us. It only provides an API to reserve memory for various uses with various restrictions (alignment, etc.) and use different allocators to obtain that memory. So, are you suggesting, that with that in place, we would first allocate the preview queue from this memory, then free it, when switching to snapshooting, allocate our large-buffer queue from the _same_ memory, capture images, free and allocate preview queue again? Would that be fast enough? In fact, it would be kind of smart to reuse the same memory for both queues, but if we could do it without re-allocations?... Thanks Guennadi --- Guennadi Liakhovetski, Ph.D. Freelance Open-Source Software Developer http://www.open-technology.de/ -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu: > Hi Mauro > > Thanks for your comments. > > On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote: > >> Em 16-02-2011 08:35, Guennadi Liakhovetski escreveu: > > [snip] > >>> (2) cleanly separate setting video data format (S_FMT) from specifying the >>> allocated buffer size. >> >> This would break existing applications. Too late for that, except if >> negotiated >> with a "SETCAP" like approach. > > Sorry, don't see how my proposal from my last mail would change existing > applications. As long as no explicit buffer-queue management is performed, > no new queues are allocated, the driver will just implicitly allocate one > queue and use it. I.e., no change in behaviour. Using the same ioctl to explicitly or to implicitly allocating memory depending on the context would make the API more complicated than it should be. >> There's an additional problem with that: assume that streaming is happening, >> and a S_FMT changing the resolution was sent. There's no way to warrant that >> the very next frame will have the new resolution. So, a meta-data with the >> frame resolution (and format) would be needed. > > Sorry, we are not going to allow format changes during a running capture. > You have to stop streaming, set new formats (possibly switch to another > queue) and restart streaming. > What am I missing? If you're stopping the stream, the current API will work as-is. If all of your concerns is about reserving a bigger buffer queue, I think that one of the reasons for the CMA allocator it for such usage. Am I missing something? Cheers, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Mauro Thanks for your comments. On Thu, 17 Feb 2011, Mauro Carvalho Chehab wrote: > Em 16-02-2011 08:35, Guennadi Liakhovetski escreveu: [snip] > > (2) cleanly separate setting video data format (S_FMT) from specifying the > > allocated buffer size. > > This would break existing applications. Too late for that, except if > negotiated > with a "SETCAP" like approach. Sorry, don't see how my proposal from my last mail would change existing applications. As long as no explicit buffer-queue management is performed, no new queues are allocated, the driver will just implicitly allocate one queue and use it. I.e., no change in behaviour. > There's an additional problem with that: assume that streaming is happening, > and a S_FMT changing the resolution was sent. There's no way to warrant that > the very next frame will have the new resolution. So, a meta-data with the > frame resolution (and format) would be needed. Sorry, we are not going to allow format changes during a running capture. You have to stop streaming, set new formats (possibly switch to another queue) and restart streaming. What am I missing? Thanks Guennadi --- Guennadi Liakhovetski, Ph.D. Freelance Open-Source Software Developer http://www.open-technology.de/ -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Em 16-02-2011 08:35, Guennadi Liakhovetski escreveu:
But from the point of view of the application it makes more sense to
actually have two video nodes. The only difference is that when one
starts streaming it pre-empts the other.
>>>
>>> Well, I don't think I like it all that much... One reason - yes, two
>>> "independent" video device-nodes, which actually only can stream in turn
>>> seems somewhat counterintuitive to me, specifically, I don't think there
>>> is a way to tell the application about this. What if we have more than two
>>> video devices on the system? Or what if you need more than two video
>>> queues / formats? Or actually only one? The kernel doesn't know initially
>>> how many, this is a dynamic resource, not a fixed static interface, IMHO.
>>
>> I agree with this, which is why I don't think two (or more) video nodes
>> would
>> be a good solution.
I agree. Video nodes shouldn't be bind to an specific format. A device with 2
video nodes should be able to retrieve images from two independent video
sources.
Unfortunately, ivtv driver was merged with the bad concept of one video node per
different type of formats (partially my fault: I remember I commented about it
at
the time it was submitted, but, as its merge took a long time, and there were
several other issues that were needed to be solved there, I ended by giving up
and letting it to come with this API non-compliance, hoping that a fix would
happen at the next kernel release. Unfortunately, it was never fixed).
>> We've hit the exact same issue with the OMAP3 ISP driver. Our current
>> solution
>> is to allocate video buffer queues at the file handle level instead of the
>> video node level. Applications can open the same video device twice and
>> allocate buffers for the viewfinder on one of the instances and for still
>> image capture on the other. When switching from viewfinder to still image
>> capture, all it needs to do (beside obviously reconfiguring the media
>> controller pipeline if required) is to issue VIDIOC_STREAMOFF on the
>> viewfinder file handle and VIDIOC_STREAMON on the still capture file handle.
This seems to be the proper way. solution (2) of using read()/mmap() is just
an special case of per-file handle stream control, as applications that used
this approach in the past were, in fact, using two opens, one for read, and
another for mmap (to be clear, I'm not in favor of 2, I'm just saying that
a per-file handle solution will allow (2) also).
>> One issue with this approach is that allocating buffers requires knowledge
>> of
>> the format and resolution. The driver traditionally computes the buffer size
>> from the parameters set by VIDIOC_S_FMT. This would prevent an application
>> opening the video node a second time and setting up buffers for still image
>> capture a second time while the viewfinder is running, as the VIDIOC_S_FMT
>> on
>> the second file handle won't be allowed then.
>>
>> Changes to the V4L2 spec would be needed to allow this to work properly.
>
> The spec is actually saying about the S_FMT ioctl():
>
> "On success the driver may program the hardware, allocate resources and
> generally prepare for data exchange."
>
> - _may_ program the hardware. So, if we don't do that and instead only
> verify the format and store it for future activation upon a call to
> STREAMON we are not violating the spec, thus, no change is required. OTOH,
> another spec sections "V4L2 close()" says:
>
> "data format parameters, current input or output, control values or other
> properties remain unchanged."
>
> which is usually interpreted as "a sequence open(); ioctl(S_FMT); close();
> open(); ioctl(STREAMON);" _must_ use the format, set by the call to S_FMT,
> which is not necessarily logical, if we adopt the per-file-descriptor
> format / stream approach.
Every time a "may" appears on a spec, we'll have troubles, as some drivers
will follow the "may" and others won't follow. Changing the behaviour will
likely cause regressions, whatever direction is taken.
One alternative would be to have a better way to negotiate features than what's
provided by QUERYCAP. If we look for some protocols with a long life, like
telnet,
they don't have a one-way to check/set capabilities. Instead, both parties
should
present their capabilities and the client need to negotiate what he wants.
We could do something like:
ret = ioctl(fd, VIDIOC_QUERYCAP, &cap);
if (cap.capabilities & CAN_PER_FD_FMT) {
setcap.capabilities |= SHOULD_PER_FD_FMT;
ret = ioctl(fd, VIDIOC_SETCAP, &setcap);
}
To be sure that the kernel driver will behave fine. Yet, in this particular
case,
this would mean that drivers or core will need to handle both per-fd and
per-node
S_FMT & friends.
> I think, we have two options:
>
> (1) adopt Laurent's proposal of per-fd contexts, but that would require a
> pretty heave modification of the spec -
Re: [RFD] frame-size switching: preview / single-shot use-case
Em 15-02-2011 15:33, Guennadi Liakhovetski escreveu: > Hi > > This topic has been slightly discussed several times [1] before, but there > has been no conclusion, nor I'm aware of any implementation, suitably > resolving this problem. I've added to CC all involved in earlier > discussions, that I managed to find. > > What seems a typical use-case to me is a system with a vewfinder or a > display, providing a live preview of the video data from a source, like a > camera, with a relatively low resolution, and a possibility to take > high-resolution still photos with a very short delay. > > Currently this is pretty difficult to realise, e.g., with soc-camera > drivers you have to free the videobuf(2) queue, by either closing and > re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS, > count=0) if your driver is already using videobuf2 and if this is really > working;), configure with a different resolution and re-allocate > videobuffers (or use different buffers, allocated per USERPTR). Another > possibility would be to allocate and use buffers large enough for still > photos, also for the preview, which would be wasteful, because one can > well need many more preview than still-shot buffers. > > So, it seems to me, we could live with a better solution. > > 1. We could use separate inputs for different capture modes and support > per-input videobuf queues. Advantage: no API changes required. > Disadvantages: confusing, especially, if a driver already exports multiple > inputs. The driver does not know, whether this mode is required or not, > always exporting 2 inputs for this purpose doesn't seem like a good idea. > Eventually, the user might want not 2, but 3 or more of such videobuf > queues. Very bad. The high res is not a new input. > > 2. Use different IO methods, e.g., mmap() for preview and read() for still > shots. I'm just mentioning this possibility here, because it occurred in > one of previous threads, but I don't really like it either. What if you > want to use the same IO method for all? Etc. Can be done. > > 3. Not liking either of the above, it seems we need yet a new API for > this... How about extending VIDIOC_REQBUFS with a videobuf queue index, > thus using up one of the remaining two 32-bit reserved fields? Then we > need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to > another, after which setting frame format and queuing and dequeuing > buffers will affect this currently selected queue. We could also keep > REQBUFS as is and require BUFQ_SELECT to be called before it for any queue > except the default 0. > > Yes, I know, that some video sensors have a double register set for this > dual-format operation, so, for them it is natural to support two queues, > and drivers are certainly most welcome to use this feature for, say, the > first two queues. On other sensors and for any further queues switching > will have to be done in software. Seems too hacky to me. There's a 4th alternative: open the device twice, and use different settings on each open. Userspace can stop streaming on one file descriptor and start the other one when he wants to take a high-res picture. Cheers, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi all
Let me try to further elaborate on this topic. So far, I think, the
following provides the cleanest solution to our quick format-switching /
multiple video-queue problem:
On Wed, 16 Feb 2011, Guennadi Liakhovetski wrote:
> (2) cleanly separate setting video data format (S_FMT) from specifying the
> allocated buffer size.
The control flow then would look like:
struct v4l2_requestbuffers req;
fd = open();
/* initialise and configure the first "still-shot" buffer-queue */
bufq.id = -EINVAL;
ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);
/*
* bufq.id now contains a handle to a newly allocated videobuf
* queue, it also becomes the current queue.
*/
still_qid = buf.id;
/*
* specify explicitly required buffer size - use one of reserved
* fields
*/
req.size = still_x * still_y * bpp;
ret = ioctl(fd, VIDIOC_REQBUFS, &req);
for (i = 0; i < req.count; i++) {
buf.index = i;
ret = ioctl(fd, VIDIOC_QUERYBUF, &buf);
start = mmap();
ret = ioctl(fd, VIDIOC_QBUF, &buf);
}
/* Allocate a second "preview" queue */
bufq.id = -EINVAL;
ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);
preview_qid = bufq.id;
req.size = preview_x * preview_y * bpp;
ret = ioctl(fd, VIDIOC_REQBUFS, &req);
for (i = 0; i < req.count; i++) {
buf.index = i;
ret = ioctl(fd, VIDIOC_QUERYBUF, &buf);
start = mmap();
ret = ioctl(fd, VIDIOC_QBUF, &buf);
}
/* Stay on the preview queue */
ret = ioctl(fd, VIDIOC_STREAMON, &type);
for (i = 0; i < n; i++) {
ret = ioctl(fd, VIDIOC_DQBUF, &buf);
ret = ioctl(fd, VIDIOC_QBUF, &buf);
}
ret = ioctl(fd, VIDIOC_STREAMOFF, &type);
/* Switch to the still-shot queue */
bufq.id = still_qid;
ret = ioctl(fd, VIDIOC_BUFQ_SELECT, &bufq);
ret = ioctl(fd, VIDIOC_STREAMON, &type);
for (i = 0; i < n; i++) {
ret = ioctl(fd, VIDIOC_DQBUF, &buf);
ret = ioctl(fd, VIDIOC_QBUF, &buf);
}
ret = ioctl(fd, VIDIOC_STREAMOFF, &type);
And so on. If the above is _conceptually_ acceptable, we need 1 to 2 new
ioctl()s and an extension to the VIDIOC_REQBUFS ioctl() to specify the
buffer size. I wrote "1 or 2 new ioctl()s," because above the new
VIDIOC_BUFQ_SELECT ioctl() performs two functions: if the handle is
negative, it allocates a new queue, if it containes a valid (per-device or
per file-handle?) queue ID, it switches to it. We might also decide to use
two ioctl()s for these two functions.
Comments?
Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Laurent Thanks for commenting. On Wed, 16 Feb 2011, Laurent Pinchart wrote: > Hi Guennadi, > > On Wednesday 16 February 2011 10:02:18 Guennadi Liakhovetski wrote: > > On Wed, 16 Feb 2011, Hans Verkuil wrote: > > > On Wednesday, February 16, 2011 08:42:51 Guennadi Liakhovetski wrote: > > [snip] > > I definitely don't like solutions 1 and 2 either, so I'll only comment on 3. > > > > > > 3. Not liking either of the above, it seems we need yet a new API for > > > > > this... How about extending VIDIOC_REQBUFS with a videobuf queue > > > > > index, thus using up one of the remaining two 32-bit reserved > > > > > fields? Then we need one more ioctl() like VIDIOC_BUFQ_SELECT to > > > > > switch from one queue to another, after which setting frame format > > > > > and queuing and dequeuing buffers will affect this currently > > > > > selected queue. We could also keep REQBUFS as is and require > > > > > BUFQ_SELECT to be called before it for any queue except the default > > > > > 0. > > > > > > What you really are doing here is creating the functionality of two video > > > nodes without actually making two nodes. > > > > > > But from the point of view of the application it makes more sense to > > > actually have two video nodes. The only difference is that when one > > > starts streaming it pre-empts the other. > > > > Well, I don't think I like it all that much... One reason - yes, two > > "independent" video device-nodes, which actually only can stream in turn > > seems somewhat counterintuitive to me, specifically, I don't think there > > is a way to tell the application about this. What if we have more than two > > video devices on the system? Or what if you need more than two video > > queues / formats? Or actually only one? The kernel doesn't know initially > > how many, this is a dynamic resource, not a fixed static interface, IMHO. > > I agree with this, which is why I don't think two (or more) video nodes would > be a good solution. > > We've hit the exact same issue with the OMAP3 ISP driver. Our current > solution > is to allocate video buffer queues at the file handle level instead of the > video node level. Applications can open the same video device twice and > allocate buffers for the viewfinder on one of the instances and for still > image capture on the other. When switching from viewfinder to still image > capture, all it needs to do (beside obviously reconfiguring the media > controller pipeline if required) is to issue VIDIOC_STREAMOFF on the > viewfinder file handle and VIDIOC_STREAMON on the still capture file handle. > > One issue with this approach is that allocating buffers requires knowledge of > the format and resolution. The driver traditionally computes the buffer size > from the parameters set by VIDIOC_S_FMT. This would prevent an application > opening the video node a second time and setting up buffers for still image > capture a second time while the viewfinder is running, as the VIDIOC_S_FMT on > the second file handle won't be allowed then. > > Changes to the V4L2 spec would be needed to allow this to work properly. The spec is actually saying about the S_FMT ioctl(): "On success the driver may program the hardware, allocate resources and generally prepare for data exchange." - _may_ program the hardware. So, if we don't do that and instead only verify the format and store it for future activation upon a call to STREAMON we are not violating the spec, thus, no change is required. OTOH, another spec sections "V4L2 close()" says: "data format parameters, current input or output, control values or other properties remain unchanged." which is usually interpreted as "a sequence open(); ioctl(S_FMT); close(); open(); ioctl(STREAMON);" _must_ use the format, set by the call to S_FMT, which is not necessarily logical, if we adopt the per-file-descriptor format / stream approach. I think, we have two options: (1) adopt Laurent's proposal of per-fd contexts, but that would require a pretty heave modification of the spec - S_FMT is not kept across close() / open() pair. (2) cleanly separate setting video data format (S_FMT) from specifying the allocated buffer size. Of course, there are further possibilities, like my switching ioctl() above, or we could extend the enum v4l2_priority with an extra application priority, saying, that this file-descriptor can maintain a separate S_FMT / STREAMON thread, without immediately affecting other descriptors, but this seems too obscure to me. My vote goes for (2) above, which is also what Laurent has mentioned here: > One > possible direction would be to enhance the buffers allocation API to allow > applications to set the buffer size. I've been thinking about this, and I > believe this is where the "global buffers pool" API could come into play. I just wouldn't necessarily bind it to "global buffer pools." Thanks Guennadi --- Guennadi Liakhovetski, Ph.D. Freelance Open-Source Software Develo
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Guennadi, On Wednesday 16 February 2011 10:02:18 Guennadi Liakhovetski wrote: > On Wed, 16 Feb 2011, Hans Verkuil wrote: > > On Wednesday, February 16, 2011 08:42:51 Guennadi Liakhovetski wrote: > [snip] I definitely don't like solutions 1 and 2 either, so I'll only comment on 3. > > > > 3. Not liking either of the above, it seems we need yet a new API for > > > > this... How about extending VIDIOC_REQBUFS with a videobuf queue > > > > index, thus using up one of the remaining two 32-bit reserved > > > > fields? Then we need one more ioctl() like VIDIOC_BUFQ_SELECT to > > > > switch from one queue to another, after which setting frame format > > > > and queuing and dequeuing buffers will affect this currently > > > > selected queue. We could also keep REQBUFS as is and require > > > > BUFQ_SELECT to be called before it for any queue except the default > > > > 0. > > > > What you really are doing here is creating the functionality of two video > > nodes without actually making two nodes. > > > > But from the point of view of the application it makes more sense to > > actually have two video nodes. The only difference is that when one > > starts streaming it pre-empts the other. > > Well, I don't think I like it all that much... One reason - yes, two > "independent" video device-nodes, which actually only can stream in turn > seems somewhat counterintuitive to me, specifically, I don't think there > is a way to tell the application about this. What if we have more than two > video devices on the system? Or what if you need more than two video > queues / formats? Or actually only one? The kernel doesn't know initially > how many, this is a dynamic resource, not a fixed static interface, IMHO. I agree with this, which is why I don't think two (or more) video nodes would be a good solution. We've hit the exact same issue with the OMAP3 ISP driver. Our current solution is to allocate video buffer queues at the file handle level instead of the video node level. Applications can open the same video device twice and allocate buffers for the viewfinder on one of the instances and for still image capture on the other. When switching from viewfinder to still image capture, all it needs to do (beside obviously reconfiguring the media controller pipeline if required) is to issue VIDIOC_STREAMOFF on the viewfinder file handle and VIDIOC_STREAMON on the still capture file handle. One issue with this approach is that allocating buffers requires knowledge of the format and resolution. The driver traditionally computes the buffer size from the parameters set by VIDIOC_S_FMT. This would prevent an application opening the video node a second time and setting up buffers for still image capture a second time while the viewfinder is running, as the VIDIOC_S_FMT on the second file handle won't be allowed then. Changes to the V4L2 spec would be needed to allow this to work properly. One possible direction would be to enhance the buffers allocation API to allow applications to set the buffer size. I've been thinking about this, and I believe this is where the "global buffers pool" API could come into play. > > Of course, implementing this correctly will probably require some changes > > in videobuf2 to make it easy. > > > > When it comes to the userspace API I wonder if we shouldn't add a > > VIDIOC_STREAM_FRAMES(u32 framecnt), which streams just 'framecnt' frames. > > I can imagine that it can be hard to stream just a single frame > > otherwise. > > Don't think you actually need to take just 1 frame for still photography. > As far as I understand, in many cases they anyway want to take several of > them to then be able to choose the best one or do some processing. > Regardless, maybe that ioctl() would make sense, but that's something > slightly different from what we're trying to figure out here. Yes, it is > also related to still photography, but it's addressing a different aspect > of it. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Hans Thanks for looking at it! On Wed, 16 Feb 2011, Hans Verkuil wrote: > Hi Guennadi, > > Here is my take on this: > > On Wednesday, February 16, 2011 08:42:51 Guennadi Liakhovetski wrote: [snip] > > > 3. Not liking either of the above, it seems we need yet a new API for > > > this... How about extending VIDIOC_REQBUFS with a videobuf queue index, > > > thus using up one of the remaining two 32-bit reserved fields? Then we > > > need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to > > > another, after which setting frame format and queuing and dequeuing > > > buffers will affect this currently selected queue. We could also keep > > > REQBUFS as is and require BUFQ_SELECT to be called before it for any queue > > > except the default 0. > > What you really are doing here is creating the functionality of two video > nodes without actually making two nodes. > > But from the point of view of the application it makes more sense to actually > have two video nodes. The only difference is that when one starts streaming it > pre-empts the other. Well, I don't think I like it all that much... One reason - yes, two "independent" video device-nodes, which actually only can stream in turn seems somewhat counterintuitive to me, specifically, I don't think there is a way to tell the application about this. What if we have more than two video devices on the system? Or what if you need more than two video queues / formats? Or actually only one? The kernel doesn't know initially how many, this is a dynamic resource, not a fixed static interface, IMHO. > Of course, implementing this correctly will probably require some changes in > videobuf2 to make it easy. > > When it comes to the userspace API I wonder if we shouldn't add a > VIDIOC_STREAM_FRAMES(u32 framecnt), which streams just 'framecnt' frames. I > can imagine that it can be hard to stream just a single frame otherwise. Don't think you actually need to take just 1 frame for still photography. As far as I understand, in many cases they anyway want to take several of them to then be able to choose the best one or do some processing. Regardless, maybe that ioctl() would make sense, but that's something slightly different from what we're trying to figure out here. Yes, it is also related to still photography, but it's addressing a different aspect of it. Thanks Guennadi --- Guennadi Liakhovetski, Ph.D. Freelance Open-Source Software Developer http://www.open-technology.de/ -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFD] frame-size switching: preview / single-shot use-case
Hi Guennadi, Here is my take on this: On Wednesday, February 16, 2011 08:42:51 Guennadi Liakhovetski wrote: > Hi, > > On Tue, 15 Feb 2011, Qing Xu wrote: > > Please, don't top-post and use a proper quoting. > > > Hi, > > > > I have a question that why we must check "icf->vb_vidq.bufs[0]" in > > s_fmt_vid_cap()? The application mainly calling sequence at switching > > fmt could be like this: > > streamoff > > s_fmt_vid_cap > > request_buf > > qbuf...qbuf > > streamon > > qbuf/dqbuf > > ... > > The application should also aware that they are switching the fmt, > > so they should release their buffer(per in usrptr method), and re-call > > request-buf/qbuf, so based on this assumption, how about we check "bufs[0]" > > in request_buf or qbuf according to the new fmt, if we find the buffer > > size is not correct, then indicate error in request_buf or qbuf. > > However,in s_fmt_vid_cap, > > we only need to check whether streaming is on/off, if it is still on, > > that means HW resource is not available or IO/buffer is in progress, then > > we reject s_fmt. What do you think? > > Yes, that restriction is in a way more conservative, than it needs to be. > I did that to (a) simplify the design and implementation: that way you > don't have to worry about S_FMT calls in your drivers during a running > capture, and to (b) verify whether that can be a problem for anyone and to > hear from them;) > > > For the idea 3, (what is the difference between idea 1 and 3?) > > (1) proposes to add multiple inputs to the video device, selectable per > VIDIOC_S_INPUT, (3) doesn't touch inputs and only proposes to add multiple > videobuf queues to devices, adding a special ioctl(). > > > in our real usage cases of camera(our product is a phone), > > we will set many formats, such as: > > preview@VGA, photos@5M/3M/QVGA/QCIF..., video@1080p/720p/VGA/QVGA, if we > > maintain each queue for each fmt, it seems that there are too many queues, > > and, in this way, the application need to be changed, it should quite > > aware of VIDIOC_BUFQ_SELECT queue-id for each fmt, then it could > > allocate/release > > the required buffer queue by new ioctl. Is my understanding correct? > > Of course, with case (3) from my original post, if you want to use those > multiple queues, you _have_ to modify your application. Same holds for (1) > and (2) too. As for too many queues - well, you can decide then. Maybe you > don't need a separate queue for each format, maybe you could use one queue > for several still-shot queues with similar buffer sizes and buffer > numbers. Or you don't need them all simultaneously, depending on your > use-case. If you have an interface, where the user only switches between > preview and still images, the still image format is usually fixed. To > select a different format the user has to go to some kind of a > configuration menu and select a different image format, right? Which would > be a good time to release the previous still image queue and allocate a > new one. > > Thanks > Guennadi > > > > > Thanks! > > -Qing > > > > -Original Message- > > From: Guennadi Liakhovetski [mailto:g.liakhovet...@gmx.de] > > Sent: 2011Äê2ÔÂ16ÈÕ 1:34 > > To: Linux Media Mailing List > > Cc: Qing Xu; Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; Karicheri, Muralidharan; Eino-Ville Talvala > > Subject: [RFD] frame-size switching: preview / single-shot use-case > > > > Hi > > > > This topic has been slightly discussed several times [1] before, but there > > has been no conclusion, nor I'm aware of any implementation, suitably > > resolving this problem. I've added to CC all involved in earlier > > discussions, that I managed to find. > > > > What seems a typical use-case to me is a system with a vewfinder or a > > display, providing a live preview of the video data from a source, like a > > camera, with a relatively low resolution, and a possibility to take > > high-resolution still photos with a very short delay. > > > > Currently this is pretty difficult to realise, e.g., with soc-camera > > drivers you have to free the videobuf(2) queue, by either closing and > > re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS, > > count=0) if your driver is already using videobuf2 and if this is really > > working;), configure with a different resolution and re-allocate > > videobuffers (or use different buffers, allocated per USERPTR). Another > > possibility would be to allocate and use buffers large enough for still > > photos, also for the preview, which would be wasteful, because one can > > well need many more preview than still-shot buffers. > > > > So, it seems to me, we could live with a better solution. > > > > 1. We could use separate inputs for different capture modes and support > > per-input videobuf queues. Advantage: no API changes required. > > Disadvantages: confusing, especially, if a driver already exports multiple > > inputs. The driver does not know, whether
RE: [RFD] frame-size switching: preview / single-shot use-case
Hi, On Tue, 15 Feb 2011, Qing Xu wrote: Please, don't top-post and use a proper quoting. > Hi, > > I have a question that why we must check "icf->vb_vidq.bufs[0]" in > s_fmt_vid_cap()? The application mainly calling sequence at switching > fmt could be like this: > streamoff > s_fmt_vid_cap > request_buf > qbuf...qbuf > streamon > qbuf/dqbuf > ... > The application should also aware that they are switching the fmt, > so they should release their buffer(per in usrptr method), and re-call > request-buf/qbuf, so based on this assumption, how about we check "bufs[0]" > in request_buf or qbuf according to the new fmt, if we find the buffer > size is not correct, then indicate error in request_buf or qbuf. > However,in s_fmt_vid_cap, > we only need to check whether streaming is on/off, if it is still on, > that means HW resource is not available or IO/buffer is in progress, then > we reject s_fmt. What do you think? Yes, that restriction is in a way more conservative, than it needs to be. I did that to (a) simplify the design and implementation: that way you don't have to worry about S_FMT calls in your drivers during a running capture, and to (b) verify whether that can be a problem for anyone and to hear from them;) > For the idea 3, (what is the difference between idea 1 and 3?) (1) proposes to add multiple inputs to the video device, selectable per VIDIOC_S_INPUT, (3) doesn't touch inputs and only proposes to add multiple videobuf queues to devices, adding a special ioctl(). > in our real usage cases of camera(our product is a phone), > we will set many formats, such as: > preview@VGA, photos@5M/3M/QVGA/QCIF..., video@1080p/720p/VGA/QVGA, if we > maintain each queue for each fmt, it seems that there are too many queues, > and, in this way, the application need to be changed, it should quite > aware of VIDIOC_BUFQ_SELECT queue-id for each fmt, then it could > allocate/release > the required buffer queue by new ioctl. Is my understanding correct? Of course, with case (3) from my original post, if you want to use those multiple queues, you _have_ to modify your application. Same holds for (1) and (2) too. As for too many queues - well, you can decide then. Maybe you don't need a separate queue for each format, maybe you could use one queue for several still-shot queues with similar buffer sizes and buffer numbers. Or you don't need them all simultaneously, depending on your use-case. If you have an interface, where the user only switches between preview and still images, the still image format is usually fixed. To select a different format the user has to go to some kind of a configuration menu and select a different image format, right? Which would be a good time to release the previous still image queue and allocate a new one. Thanks Guennadi > > Thanks! > -Qing > > -Original Message- > From: Guennadi Liakhovetski [mailto:g.liakhovet...@gmx.de] > Sent: 2011Äê2ÔÂ16ÈÕ 1:34 > To: Linux Media Mailing List > Cc: Qing Xu; Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; > Karicheri, Muralidharan; Eino-Ville Talvala > Subject: [RFD] frame-size switching: preview / single-shot use-case > > Hi > > This topic has been slightly discussed several times [1] before, but there > has been no conclusion, nor I'm aware of any implementation, suitably > resolving this problem. I've added to CC all involved in earlier > discussions, that I managed to find. > > What seems a typical use-case to me is a system with a vewfinder or a > display, providing a live preview of the video data from a source, like a > camera, with a relatively low resolution, and a possibility to take > high-resolution still photos with a very short delay. > > Currently this is pretty difficult to realise, e.g., with soc-camera > drivers you have to free the videobuf(2) queue, by either closing and > re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS, > count=0) if your driver is already using videobuf2 and if this is really > working;), configure with a different resolution and re-allocate > videobuffers (or use different buffers, allocated per USERPTR). Another > possibility would be to allocate and use buffers large enough for still > photos, also for the preview, which would be wasteful, because one can > well need many more preview than still-shot buffers. > > So, it seems to me, we could live with a better solution. > > 1. We could use separate inputs for different capture modes and support > per-input videobuf queues. Advantage: no API changes required. > Disadvantages: confusing, especially, if a driver already exports multiple > inputs. The driver does not know, whether this mode is required or not, > always exporting 2 inputs for this purpose doesn't seem like a good idea. > Eventually, the user might want not 2, but 3 or more of such videobuf > queues. > > 2. Use different IO methods, e.g., mmap() for preview and read() for still > shots. I'm just mentioning this possibility her
RE: [RFD] frame-size switching: preview / single-shot use-case
Add Kassey and Angela in the loop. Thanks!
-Qing
-Original Message-
From: Qing Xu
Sent: 2011年2月16日 10:51
To: 'Guennadi Liakhovetski'; Linux Media Mailing List
Cc: Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; Karicheri,
Muralidharan; Eino-Ville Talvala
Subject: RE: [RFD] frame-size switching: preview / single-shot use-case
Hi,
I have a question that why we must check "icf->vb_vidq.bufs[0]" in
s_fmt_vid_cap()? The application mainly calling sequence at switching
fmt could be like this:
streamoff
s_fmt_vid_cap
request_buf
qbuf...qbuf
streamon
qbuf/dqbuf
...
The application should also aware that they are switching the fmt,
so they should release their buffer(per in usrptr method), and re-call
request-buf/qbuf, so based on this assumption, how about we check "bufs[0]"
in request_buf or qbuf according to the new fmt, if we find the buffer
size is not correct, then indicate error in request_buf or qbuf.
However,in s_fmt_vid_cap,
we only need to check whether streaming is on/off, if it is still on,
that means HW resource is not available or IO/buffer is in progress, then
we reject s_fmt. What do you think?
For the idea 3, (what is the difference between idea 1 and 3?)
in our real usage cases of camera(our product is a phone),
we will set many formats, such as:
preview@VGA, photos@5M/3M/QVGA/QCIF..., video@1080p/720p/VGA/QVGA, if we
maintain each queue for each fmt, it seems that there are too many queues,
and, in this way, the application need to be changed, it should quite aware of
VIDIOC_BUFQ_SELECT queue-id for each fmt, then it could allocate/release
the required buffer queue by new ioctl. Is my understanding correct?
Thanks!
-Qing
-Original Message-
From: Guennadi Liakhovetski [mailto:g.liakhovet...@gmx.de]
Sent: 2011年2月16日 1:34
To: Linux Media Mailing List
Cc: Qing Xu; Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert;
Karicheri, Muralidharan; Eino-Ville Talvala
Subject: [RFD] frame-size switching: preview / single-shot use-case
Hi
This topic has been slightly discussed several times [1] before, but there
has been no conclusion, nor I'm aware of any implementation, suitably
resolving this problem. I've added to CC all involved in earlier
discussions, that I managed to find.
What seems a typical use-case to me is a system with a vewfinder or a
display, providing a live preview of the video data from a source, like a
camera, with a relatively low resolution, and a possibility to take
high-resolution still photos with a very short delay.
Currently this is pretty difficult to realise, e.g., with soc-camera
drivers you have to free the videobuf(2) queue, by either closing and
re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS,
count=0) if your driver is already using videobuf2 and if this is really
working;), configure with a different resolution and re-allocate
videobuffers (or use different buffers, allocated per USERPTR). Another
possibility would be to allocate and use buffers large enough for still
photos, also for the preview, which would be wasteful, because one can
well need many more preview than still-shot buffers.
So, it seems to me, we could live with a better solution.
1. We could use separate inputs for different capture modes and support
per-input videobuf queues. Advantage: no API changes required.
Disadvantages: confusing, especially, if a driver already exports multiple
inputs. The driver does not know, whether this mode is required or not,
always exporting 2 inputs for this purpose doesn't seem like a good idea.
Eventually, the user might want not 2, but 3 or more of such videobuf
queues.
2. Use different IO methods, e.g., mmap() for preview and read() for still
shots. I'm just mentioning this possibility here, because it occurred in
one of previous threads, but I don't really like it either. What if you
want to use the same IO method for all? Etc.
3. Not liking either of the above, it seems we need yet a new API for
this... How about extending VIDIOC_REQBUFS with a videobuf queue index,
thus using up one of the remaining two 32-bit reserved fields? Then we
need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to
another, after which setting frame format and queuing and dequeuing
buffers will affect this currently selected queue. We could also keep
REQBUFS as is and require BUFQ_SELECT to be called before it for any queue
except the default 0.
Yes, I know, that some video sensors have a double register set for this
dual-format operation, so, for them it is natural to support two queues,
and drivers are certainly most welcome to use this feature for, say, the
first two queues. On other sensors and for any further queues switching
will have to be done in software.
Ideas? Comments?
Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/
N�Р骒r��yb�X�肚�v�^�)藓{.n�+�伐�{���bj)��骅w* jg�报 �茛j/�赇z罐���2���ㄨ��&�)摺�a囤�� �G���h��j:+v���w��佶
RE: [RFD] frame-size switching: preview / single-shot use-case
Hi, I have a question that why we must check "icf->vb_vidq.bufs[0]" in s_fmt_vid_cap()? The application mainly calling sequence at switching fmt could be like this: streamoff s_fmt_vid_cap request_buf qbuf...qbuf streamon qbuf/dqbuf ... The application should also aware that they are switching the fmt, so they should release their buffer(per in usrptr method), and re-call request-buf/qbuf, so based on this assumption, how about we check "bufs[0]" in request_buf or qbuf according to the new fmt, if we find the buffer size is not correct, then indicate error in request_buf or qbuf. However,in s_fmt_vid_cap, we only need to check whether streaming is on/off, if it is still on, that means HW resource is not available or IO/buffer is in progress, then we reject s_fmt. What do you think? For the idea 3, (what is the difference between idea 1 and 3?) in our real usage cases of camera(our product is a phone), we will set many formats, such as: preview@VGA, photos@5M/3M/QVGA/QCIF..., video@1080p/720p/VGA/QVGA, if we maintain each queue for each fmt, it seems that there are too many queues, and, in this way, the application need to be changed, it should quite aware of VIDIOC_BUFQ_SELECT queue-id for each fmt, then it could allocate/release the required buffer queue by new ioctl. Is my understanding correct? Thanks! -Qing -Original Message- From: Guennadi Liakhovetski [mailto:g.liakhovet...@gmx.de] Sent: 2011年2月16日 1:34 To: Linux Media Mailing List Cc: Qing Xu; Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; Karicheri, Muralidharan; Eino-Ville Talvala Subject: [RFD] frame-size switching: preview / single-shot use-case Hi This topic has been slightly discussed several times [1] before, but there has been no conclusion, nor I'm aware of any implementation, suitably resolving this problem. I've added to CC all involved in earlier discussions, that I managed to find. What seems a typical use-case to me is a system with a vewfinder or a display, providing a live preview of the video data from a source, like a camera, with a relatively low resolution, and a possibility to take high-resolution still photos with a very short delay. Currently this is pretty difficult to realise, e.g., with soc-camera drivers you have to free the videobuf(2) queue, by either closing and re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS, count=0) if your driver is already using videobuf2 and if this is really working;), configure with a different resolution and re-allocate videobuffers (or use different buffers, allocated per USERPTR). Another possibility would be to allocate and use buffers large enough for still photos, also for the preview, which would be wasteful, because one can well need many more preview than still-shot buffers. So, it seems to me, we could live with a better solution. 1. We could use separate inputs for different capture modes and support per-input videobuf queues. Advantage: no API changes required. Disadvantages: confusing, especially, if a driver already exports multiple inputs. The driver does not know, whether this mode is required or not, always exporting 2 inputs for this purpose doesn't seem like a good idea. Eventually, the user might want not 2, but 3 or more of such videobuf queues. 2. Use different IO methods, e.g., mmap() for preview and read() for still shots. I'm just mentioning this possibility here, because it occurred in one of previous threads, but I don't really like it either. What if you want to use the same IO method for all? Etc. 3. Not liking either of the above, it seems we need yet a new API for this... How about extending VIDIOC_REQBUFS with a videobuf queue index, thus using up one of the remaining two 32-bit reserved fields? Then we need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to another, after which setting frame format and queuing and dequeuing buffers will affect this currently selected queue. We could also keep REQBUFS as is and require BUFQ_SELECT to be called before it for any queue except the default 0. Yes, I know, that some video sensors have a double register set for this dual-format operation, so, for them it is natural to support two queues, and drivers are certainly most welcome to use this feature for, say, the first two queues. On other sensors and for any further queues switching will have to be done in software. Ideas? Comments? Thanks Guennadi --- Guennadi Liakhovetski, Ph.D. Freelance Open-Source Software Developer http://www.open-technology.de/
RE: [RFD] frame-size switching: preview / single-shot use-case
Adding Manjunath Hadli to the list... Murail -Original Message- From: Guennadi Liakhovetski [mailto:g.liakhovet...@gmx.de] Sent: Tuesday, February 15, 2011 12:34 PM To: Linux Media Mailing List Cc: Qing Xu; Hans Verkuil; Neil Johnson; Robert Jarzmik; Uwe Taeubert; Karicheri, Muralidharan; Eino-Ville Talvala Subject: [RFD] frame-size switching: preview / single-shot use-case Hi This topic has been slightly discussed several times [1] before, but there has been no conclusion, nor I'm aware of any implementation, suitably resolving this problem. I've added to CC all involved in earlier discussions, that I managed to find. What seems a typical use-case to me is a system with a vewfinder or a display, providing a live preview of the video data from a source, like a camera, with a relatively low resolution, and a possibility to take high-resolution still photos with a very short delay. Currently this is pretty difficult to realise, e.g., with soc-camera drivers you have to free the videobuf(2) queue, by either closing and re-opening the interface, or by issuing an ioctl(VIDIOC_REQBUFS, count=0) if your driver is already using videobuf2 and if this is really working;), configure with a different resolution and re-allocate videobuffers (or use different buffers, allocated per USERPTR). Another possibility would be to allocate and use buffers large enough for still photos, also for the preview, which would be wasteful, because one can well need many more preview than still-shot buffers. So, it seems to me, we could live with a better solution. 1. We could use separate inputs for different capture modes and support per-input videobuf queues. Advantage: no API changes required. Disadvantages: confusing, especially, if a driver already exports multiple inputs. The driver does not know, whether this mode is required or not, always exporting 2 inputs for this purpose doesn't seem like a good idea. Eventually, the user might want not 2, but 3 or more of such videobuf queues. 2. Use different IO methods, e.g., mmap() for preview and read() for still shots. I'm just mentioning this possibility here, because it occurred in one of previous threads, but I don't really like it either. What if you want to use the same IO method for all? Etc. 3. Not liking either of the above, it seems we need yet a new API for this... How about extending VIDIOC_REQBUFS with a videobuf queue index, thus using up one of the remaining two 32-bit reserved fields? Then we need one more ioctl() like VIDIOC_BUFQ_SELECT to switch from one queue to another, after which setting frame format and queuing and dequeuing buffers will affect this currently selected queue. We could also keep REQBUFS as is and require BUFQ_SELECT to be called before it for any queue except the default 0. Yes, I know, that some video sensors have a double register set for this dual-format operation, so, for them it is natural to support two queues, and drivers are certainly most welcome to use this feature for, say, the first two queues. On other sensors and for any further queues switching will have to be done in software. Ideas? Comments? Thanks Guennadi --- Guennadi Liakhovetski, Ph.D. Freelance Open-Source Software Developer http://www.open-technology.de/ -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
