I have collected presentations from this media summit here:
http://hverkuil.home.xs4all.nl/presentations/summit-2013/
Please contact me if a presentation you gave is missing so I can add it.
Regards,
Hans
On 10/31/2013 12:27 PM, Mauro Carvalho Chehab wrote:
> Notes on the Media summit 2013-10-23
>
> List of Participants:
>
> Ricardo Ribalda - Qtechnology
> Hans Verkuil - Cisco
> Sakari Ailus - Intel
> Mauro Carvalho Chehab - Samsung
> Kieran Kunhya - Open Broadcast Systems
> Sylwester Nawrocki - Samsung
> Guennadi Liakhovetski
> Benjamin Gaignard - ST Microelectronics
> Hugues Fruchet - ST Microelectronics
> Andrzej Hajda - Samsung
> Peter Senna Tschudin - Coccinelle?
> Hans de Goede - Red Hat
> Michael Krufky - Samsung
> Paul Walmsley - Nvidia
> Laurent Pinchart - Ideas on board
>
> 1. Mauro Chehab: is the submaintainer arrangement working?
>
> General consensus is that it is working.
>
> Hans pointed that the commits ML is not working. Mauro will check what's
> happening at LinuxTV website after returning back.
>
> Hans also pointed that patches for the rcX kernel aren't always picked
> up quickly enough. Mauro says this is due to his move to Samsung which
> took a lot of his time. This should improve.
>
> 2. Ricardo Ribalda Delgado: Multiple selections
>
> For a more flexible cropping selection a new extended rectangle struct has
> to be added with possibly negative top and left offsets. This would allow to
> explore capabilities of industry grade sensors, that can be configured to crop
> multiple rectangles from the input area.
>
> Sylwester: an alternative approach would be to use indexed windows. The issue
> of atomicity should also be addressed.
>
> Ricardo is proposing to add the capability of adding more than one selection
> areas
>
> - No objections so far.
> - Helper functions to detect if rectangles overlap or are contained on other
> rectangles
> are needed.
> - Documentation changes for V4L2 subdev selection API. Special care needs to
> be taken
> for too few or too many rectangles. In both cases the rectangles field
> should be
> set to the actual number of allowed rectangles. In the too few case an
> error should
> be returned, in the too many case the extra rects should just be ignored.
>
> Mauro thinks that it will be useful to see if the width/height fields in
> v4l2_rect
> can be changes to u32 instead of s32: will require code review of drivers.
>
> 3. Hans Verkuil: colorspaces
>
> - Colorspace: limited/full range
>
> Draft presentation for all these topics:
> http://hverkuil.home.xs4all.nl/presentations/summit2013.odp
>
> Proposal accepted, although the "_SRGB_LIM" name might need to be improved,
> although
> no alternatives were given.
>
> ALso mention in the docs that S_FMT can overwrite the colorspace for output
> devices
> if the selected one isn't supported.
>
> 4. Kieran Kunhya: SDI
>
> The biggest problem is separate audio and video file descriptors. Audio data
> is
> transfered in the video blanking areas, where should they be separated - in
> the
> kernel, using multiple planes, or in the user-space? One possibility is to
> consider
> this data as similar to an "uncompressed MPEG" datastream. It might be
> possible to
> use hardware-specific time-stamps to synchronise the data.
>
> Separate ALSA and V4L2 nodes are inherently problematic since they do not
> keep the
> audio and video together. Another possibility is to offer two APIs: for
> professional
> applications, where data is perfectly synchronised, and separate audio and
> video
> for "human consumption."
>
> A format for professional applications were VBI/HBI/Video is all captured in
> one
> big frame is OK (provided an open source lib is created to parse it).
>
> The proposal for using a multiplanar API where the audio is in a separate
> plane
> ran into opposition: the alsa devs should be asked whether it is possible to
> return the audio data in such a way that it can be exactly associated with the
> corresponding video buffer. This also requires that the audio can be variable
> length since for NTSC the size of the audio data will alternate between
> frames.
> If this is possible, then the alsa API can be used and things can still remain
> in sync.
>
> UPDATE: Mauro and Hans had some discussions with Takashi (ALSA Maintainer),
> in order to have some shed about the possibilities. We're planning to discuss
> more on this Friday's lunch.
>
> 5. Paul Walmsley on behalf of Bryan Wu: LED / flash API integration
>
> Currently functionality is accessible from two APIs: LED and flash. A proposal
> is presented to put V4L2 flash subdev on top of LED core. It would be better
> not to go via the LED trigger layer.
>
> - One to one mapping between LED chips and V4L2 sub-devices. This probably
> means
> the LED flash driver needs to register the sub-device.
> - Two interfaces: should the sysfs interface be disabled if an application
> chose
> the flash mode on V4L2 API? The flash driver has no knowledge of streaming
> state
> or capturing frames.
> - Same requirements as for the LED API as the V4L2 flash API needs (LED
> flashes only).
> - Currently there's no V4L2 API to put a sensor into a flash mode to
> automatically
> trigger a flash pin. This would have to be handled. It is also possible,
> that
> a V4L2 application uses a sensor, but only puts it into a "flash mode"
> (single-shot
> mode) for several frames. The LED might therefore have to be made busy
> always when
> the sensor is used. Situations, where an LED is busy should already happen,
> e.g.
> if one application wants to use it for a torch and another one for
> notifications.
>
> Results:
>
> a. There seemed to be broad consensus that there's no need to use the
> ledtrig-camera.c layer. Instead it's probably better for the v4l2
> flash subdev code to call directly into the LED framework core.
>
> b. How to register LED devices initially? How to associate LED chips to
> v4l2? In particular, how should this be modeled in DT data? There needs
> to be some link to connect sensors to flash chips in the DT data. The
> media controller API has the notion of groups; perhaps that can be
> expressed in DT with phandles.
>
> c. How should hardware triggers be implemented? Some sensor hardware has
> the ability to automatically fire the flash when the sensor activates,
> without the involvement of any Linux-side software (beyond the initial
> configuration of the feature). There needs to be some knowledge in the
> LED core that can enable this.
>
> d. Several concerns over races and multiple users: should the sysfs LED
> interface always be exposed, or not? For example, if a sysfs request
> comes in to the LED core while a hardware capture is occurring, should it
> return -EBUSY? Sakari felt that it was better to arbitrate this in
> userspace, since it was a matter of policy, while some others wanted it to
> be handled in the kernel. There was a debate and divergence of opinion on
> this. If the kernel will be responsible for this mutal exclusion, then it
> should also be possible to disable the use of other LED triggers once v4l2
> is using it.
>
> 6. Mauro Chehab: DVB / V4L2 / MC integration
>
> Mauro wants to extend Media Controller/Subdevice functionality to DVB, in
> order
> to allow setting a complete pipeline that would cover V4L2/ALSA/DVB/DRM as a
> hole. This is needed by Consumers Electronics producs like digital TV and STB.
>
> Problem: multiple subsystems might be using the MC API with or without
> connections
> between them. How to set such a topology up?
>
> Possibilities:
>
> a. To have just one global media controller device, not all graphs inside it
> would need to be connectable.
>
> b. The first user creates an MC device. Any further users detect that their
> MC
> device is already present and will reuse it
>
> c. a separate driver creates an MC device instance, e.g. a driver for links
> between subsystems
>
> One problem to be discussed with DT people is how to express the hardware
> connections between the different subsystems. This information should be
> provided
> by the open firmware, in order for the media controller to create the proper
> bus
> links internally on their graphs.
>
> 7. Hans Verkuil: try_fmt compliance
>
> Problem: VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported
> pixelformat is provided,
> but, in practice, video capture board tend to do that, while webcam drivers
> tend to map
> it silently to a valid pixelformat.
>
> Also, some applications rely on the -EINVAL error code, with is wrong, as not
> all drivers
> currently returns it: some just return 0, and fills the pixelformat data with
> a supported
> format.
>
> Resolution: fix applications, schedule driver change to comply with the spec
> and not
> return -EINVAL for an unsupported pixel format. New drivers should never
> return -EINVAL
> and should comply to the spec instead.
>
> 8. Hans Verkuil: V4L2 support libraries
>
> - libv4l2util library does not offer a public API and has a funny name. To be
> renamed as libv4l2misc.
> - create a new libv4l2misc (a libv4l2util already exists) where all the
> various useful
> bits of functionality can be placed. Posted publicly to properly discuss
> the API.
> - Doxygen to be used to document the library API.
>
> 9. Hans Verkuil: Interaction between selection API, ENUM_FRAMESIZES and S_FMT
>
> - Hans proposed to add G/S_FRAMESIZE ioctls in order to make clearer about
> what size is being changed, on scenarios that have both scaling and
> cropping.
> - E.g. UVC webcams have an interface to specify the image size. The camera
> achieves that using cropping and scaling, but the driver does not have
> information on which one.
> - sn9c102 does some weird non-standard things. This driver were obsoleted by
> gspca (except for a few really old devices that gspca developers were unable
> to use). This driver were removed on Fedora several versions ago, and
> nobody ever
> complained about it. So, it should be moved to staging and removed in a
> few kernel cycles' time.
> - Most of the developers didn't buy the idea of creating a new pair of ioctls.
> - Hans will write a new proposal on the topic that, instead of adding new
> ioctls
> (G/S_FRAMESIZE),it will add a new selection target and try to avoid the use
> of
> flags (in opposite to the current proposal).
>
> 10. Laurent Pinchart: ALSA and Media controller
>
> - ALSA folks thought of using Media controller for pipeline control but miss
> features:
> - dynamic creation/removal of MC entities;
> - lack of a way to store value pairs.
>
> - ALSA entities would need to be added and removed at runtime: DSP firmware
> decides what kind of entities exist
>
> - E.g. equalisers could be created as needed
>
> - ALSA already has APIs for that
>
> - Detailed sub-system specific data is needed:
>
> - ALSA: what kind of features the entity has
>
> - Proposal: binary blob specific to entity that contains key: value pairs
> --- this
> could be used in V4L2 as well
>
> 11. Laurent Pinchart: sharing I2C sub-device drivers between V4L2 and KMS
>
> - Dynamic assignment of devices between display and capture pipelines
> - The DT won't contain information on which driver should be used
> - Two separate drivers are currently facing such issue, but it seems that lot
> more will be added;
> - Something similar to the V4L2 sub-device is required on KMS. Without this
> feature, sharing sub-devices will be hard if not impossible.
> - We need something similar for DVB/V4L tuners, but it's much easier to push
> that upstream, as the media subsystem already uses MC subdevs.
>
> 12. Sakari Ailus: Multi format streams / metadata
>
> - metadata has to be separated from the image, but kept in sync with frames
> - some hardware is able to separate metadata automatically
> - Frame format descriptors to be extended
> - Set operation should be removed and replaced with a different solution
> on existing drivers
>
> - Format index to be added to struct v4l2_mbus_framefmt
> - No conclusion reached on how other streams are captured in the user space,
> requires further discussion and time
> - Multiple video nodes or multiplexing by the type field?
> - The upper 16 bits in the type field could be used to index the buffer
> queue
> - No need to split the field; macros can be used to access different
> parts of it
> - Stealing bytes from struct v4l2_format may have issues: the struct is
> not
> packed, and at least the alignment would need to be verified on
> supported archs
> - Multiple buffer types are not supported by videobuf2 currently with the
> exception of memory-to-memory devices and capture/overlay
> - Changes could be required in videobuf2
> - Events could be used to pass the metadata to the user space
> - Extended events required
> - Does not help with sensors that produce multiple images of the same
> frame
> - Causes two copies of the data: one to the event buffer, the other to
> the user
> space when the event is dequeued
> - Unwanted especially on embedded systems where copying costs: there's a
> measurable effect from e.g. copying 4 kiB 60 times per second from two
> sensors
> - DQBUF takes the buffer type as an argument, and the user would need to
> try different queues once select(2) returns
>
> - Multi-format buffers are not an answer for metadata since metadata must be
> in
> the user space as fast as possible
> - Multi-format buffers could be implemented by extending v4l2_format
> - The name of the IOCTLs taking that as an argument will stay the same,
> as well
> as the IOCTL number
> - The old definition is still kept around for binary compatibility
>
> - The UVC 1.5 might be a use case for multi format streams, too
> - The SDI has multiple streams, too, but could work on multi-format buffers
> - The user shouldn't have to choose whether to use multi-format buffers or
> multiple
> video nodes
> - Painful for drivers to handle, duplicate code in drivers
> - Drivers must only implement one
>
> - No consensus was reached. Sakari has to update the RFC to include all three
> topics
>
> 13: Hugues Fruchet: Video codecs
>
> - There's a need to parse bitstream fields for those codecs, but that requires
> complex code (10K lines). Moving it to kernel could make it unstable, as
> it is harsh to write those parsers without any risk of causing crashes.
> - It seems to be better to put those parsers inside libv4l, using an open
> source
> license.
>
> Results:
> - Drivers that require proprietary user space components should stay out of
> mainline
> - Multi-format buffers could be useful here
> - The hardware/firmware needs a lot of data extracted from the bitstream next
> to the bitstream itself. This is a custom format, so it is OK to add a new
> pixelformat for each of those formats. Such complex parsing should be done
> in
> userspace in libv4l2.
> - If very little parsing is required (MPEG), then that can be done in the
> kernel
> instead.
> - Recommendation is to start simple with e.g. just an MPEG implementation.
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
