This series is a RFC for how I think one could add CSI-2 virtual channel 
support to the V4L2 framework. The problem is that there is no way to in 
the media framework describe and control links between subdevices which 
carry more then one video stream, for example a CSI-2 bus which can have 
4 virtual channels carrying different video streams.

This series adds a new pad flag which would indicate that a pad carries 
multiplexed streams, adds a new s_stream() operation to the pad 
operations structure which takes a new argument 'stream'. This new 
s_stream() operation then is both pad and stream aware. It also extends 
struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how 
a CSI-2 link multiplexes virtual channels. I also include one 
implementation based on Renesas R-Car which makes use of these patches 
as I think they help with understanding but they have no impact on the 
RFC feature itself.

The idea is that on both sides of the multiplexed media link there are 
one multiplexer subdevice and one demultiplexer subdevice. These two 
subdevices can't do any format conversions, there sole purpose is to 
(de)multiplex the CSI-2 link. If there is hardware which can do both 
CSI-2 multiplexing and format conversions they can be modeled as two 
subdevices from the same device driver and using the still pending 
incremental async mechanism to connect the external pads. The reason 
there is no format conversion is important as the multiplexed pads can't 
have a format in the current V4L2 model, get/set_fmt are not aware of 

        +------------------+              +------------------+
     +-------+  subdev 1   |              |  subdev 2   +-------+
  +--+ Pad 1 |             |              |             | Pad 3 +---+
     +--+----+   +---------+---+      +---+---------+   +----+--+
        |        | Muxed pad A +------+ Muxed pad B |        |
     +--+----+   +---------+---+      +---+---------+   +----+--+
  +--+ Pad 2 |             |              |             | Pad 4 +---+
     +-------+             |              |             +-------+
        +------------------+              +------------------+

In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, 
and the video data for both of them travels the link between pad A and 
B. One shortcoming of this RFC is that there currently are no way to 
express to user-space which pad is routed to which stream of the 
multiplexed link. But inside the kernel this is known and format 
validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 
to Pad 4 by the V4L2 framework. But it would be nice for the user to 
also be able to get this information while setting up the MC graph in 

Obviously there are things that are not perfect in this RFC, one is the 
above mentioned lack of user-space visibility of that Pad 1 is in fact 
routed to Pad 3. Others are lack of Documentation/ work and I'm sure 
there are error path shortcuts which are not fully thought out. One big 
question is also if the s_stream() operation added to ops structure 
should be a compliment to the existing ones in video and audio ops or 
aim to replace the one in video ops. I'm also unsure of the CSI2 flag of 
struct v4l2_mbus_frame_desc_entry don't really belong in struct 
v4l2_mbus_frame_desc. And I'm sure there are lots of other stuff that's 
why this is a RFC...

A big thanks to Laurent and Sakari for being really nice and taking time 
helping me grasp all the possibilities and issues with this problem, all 
cred to them and all blame to me for misunderstanding there guidance :-)

This series based on the latest R-Car CSI-2 and VIN patches which can be 
found at [1], but that is a dependency only for the driver specific
implementation which acts as an example of implementation. For the V4L2 
framework patches the media-tree is the base.

1. https://git.ragnatech.se/linux#rcar-vin-elinux-v12

Niklas Söderlund (20):
  media.h: add MEDIA_PAD_FL_MUXED flag
  v4l2-subdev.h: add pad and stream aware s_stream
  v4l2-subdev.h: add CSI-2 bus description to struct
  v4l2-core: check that both pads in a link are muxed if one are
  v4l2-core: verify all streams formats on multiplexed links
  rcar-vin: use the pad and stream aware s_stream
  rcar-csi2: declare sink pad as multiplexed
  rcar-csi2: switch to pad and stream aware s_stream
  rcar-csi2: figure out remote pad and stream which are starting
  rcar-csi2: count usage for each source pad
  rcar-csi2: when starting CSI-2 receiver use frame descriptor
  rcar-csi2: only allow formats on source pads
  rcar-csi2: implement get_frame_desc
  adv748x: add module param for virtual channel
  adv748x: declare source pad as multiplexed
  adv748x: add translation from pixelcode to CSI-2 datatype
  adv748x: implement get_frame_desc
  adv748x: switch to pad and stream aware s_stream
  adv748x: only allow formats on sink pads
  arm64: dts: renesas: salvator: use VC1 for CVBS

 arch/arm64/boot/dts/renesas/salvator-common.dtsi |   2 +-
 drivers/media/i2c/adv748x/adv748x-core.c         |  10 +
 drivers/media/i2c/adv748x/adv748x-csi2.c         |  78 +++++++-
 drivers/media/i2c/adv748x/adv748x.h              |   1 +
 drivers/media/platform/rcar-vin/rcar-csi2.c      | 239 ++++++++++++++++-------
 drivers/media/platform/rcar-vin/rcar-dma.c       |   6 +-
 drivers/media/v4l2-core/v4l2-subdev.c            |  65 ++++++
 include/media/v4l2-subdev.h                      |  16 ++
 include/uapi/linux/media.h                       |   1 +
 9 files changed, 341 insertions(+), 77 deletions(-)


Reply via email to