Hi,
This series is a RFC for how I think one could add CSI-2 virtual channel
support to the V4L2 framework. The problem is that there is no way to in
the media framework describe and control links between subdevices which
carry more then one video stream, for example a CSI-2 bus which can have
4 virtual channels carrying different video streams.
This series adds a new pad flag which would indicate that a pad carries
multiplexed streams, adds a new s_stream() operation to the pad
operations structure which takes a new argument 'stream'. This new
s_stream() operation then is both pad and stream aware. It also extends
struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how
a CSI-2 link multiplexes virtual channels. I also include one
implementation based on Renesas R-Car which makes use of these patches
as I think they help with understanding but they have no impact on the
RFC feature itself.
The idea is that on both sides of the multiplexed media link there are
one multiplexer subdevice and one demultiplexer subdevice. These two
subdevices can't do any format conversions, there sole purpose is to
(de)multiplex the CSI-2 link. If there is hardware which can do both
CSI-2 multiplexing and format conversions they can be modeled as two
subdevices from the same device driver and using the still pending
incremental async mechanism to connect the external pads. The reason
there is no format conversion is important as the multiplexed pads can't
have a format in the current V4L2 model, get/set_fmt are not aware of
streams.
+------------------+ +------------------+
+-------+ subdev 1 | | subdev 2 +-------+
+--+ Pad 1 | | | | Pad 3 +---+
+--+----+ +---------+---+ +---+---------+ +----+--+
| | Muxed pad A +------+ Muxed pad B | |
+--+----+ +---------+---+ +---+---------+ +----+--+
+--+ Pad 2 | | | | Pad 4 +---+
+-------+ | | +-------+
+------------------+ +------------------+
In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4,
and the video data for both of them travels the link between pad A and
B. One shortcoming of this RFC is that there currently are no way to
express to user-space which pad is routed to which stream of the
multiplexed link. But inside the kernel this is known and format
validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2
to Pad 4 by the V4L2 framework. But it would be nice for the user to
also be able to get this information while setting up the MC graph in
user-space.
Obviously there are things that are not perfect in this RFC, one is the
above mentioned lack of user-space visibility of that Pad 1 is in fact
routed to Pad 3. Others are lack of Documentation/ work and I'm sure
there are error path shortcuts which are not fully thought out. One big
question is also if the s_stream() operation added to ops structure
should be a compliment to the existing ones in video and audio ops or
aim to replace the one in video ops. I'm also unsure of the CSI2 flag of
struct v4l2_mbus_frame_desc_entry don't really belong in struct
v4l2_mbus_frame_desc. And I'm sure there are lots of other stuff that's
why this is a RFC...
A big thanks to Laurent and Sakari for being really nice and taking time
helping me grasp all the possibilities and issues with this problem, all
cred to them and all blame to me for misunderstanding there guidance :-)
This series based on the latest R-Car CSI-2 and VIN patches which can be
found at [1], but that is a dependency only for the driver specific
implementation which acts as an example of implementation. For the V4L2
framework patches the media-tree is the base.
1. https://git.ragnatech.se/linux#rcar-vin-elinux-v12
Niklas Söderlund (20):
media.h: add MEDIA_PAD_FL_MUXED flag
v4l2-subdev.h: add pad and stream aware s_stream
v4l2-subdev.h: add CSI-2 bus description to struct
v4l2_mbus_frame_desc_entry
v4l2-core: check that both pads in a link are muxed if one are
v4l2-core: verify all streams formats on multiplexed links
rcar-vin: use the pad and stream aware s_stream
rcar-csi2: declare sink pad as multiplexed
rcar-csi2: switch to pad and stream aware s_stream
rcar-csi2: figure out remote pad and stream which are starting
rcar-csi2: count usage for each source pad
rcar-csi2: when starting CSI-2 receiver use frame descriptor
information
rcar-csi2: only allow formats on source pads
rcar-csi2: implement get_frame_desc
adv748x: add module param for virtual channel
adv748x: declare source pad as multiplexed
adv748x: add translation from pixelcode to CSI-2 datatype
adv748x: implement get_frame_desc
adv748x: switch to pad and stream aware s_stream
adv748x: only allow formats on sink pads
arm64: dts: renesas: salvator: use VC1 for CVBS
arch/arm64/boot/dts/renesas/salvator-common.dtsi | 2 +-
drivers/media/i2c/adv748x/adv748x-core.c | 10 +
drivers/media/i2c/adv748x/adv748x-csi2.c | 78 +++++++-
drivers/media/i2c/adv748x/adv748x.h | 1 +
drivers/media/platform/rcar-vin/rcar-csi2.c | 239 ++++++++++++++++-------
drivers/media/platform/rcar-vin/rcar-dma.c | 6 +-
drivers/media/v4l2-core/v4l2-subdev.c | 65 ++++++
include/media/v4l2-subdev.h | 16 ++
include/uapi/linux/media.h | 1 +
9 files changed, 341 insertions(+), 77 deletions(-)
--
2.13.3
