@Andrew
> @Andrea, for some reason I can't download attaches from forum (because I
have no login there?), can you add CMS.txt as attachment to email (again)?
I'm opening a new email because color has nothing to do with the
Vdpau/Vaapi issues.
It was just an attempt to put together all the information on how
CinGG treats color. Lately I found emails from Hermann Vosseler that
are very indicative. However, the fact remains that each plugins of
CinGG treats the color in its own way, so we should find a way to
understand how all the native plugins act, one by one (if I'm not
mistaken, the plugins of ffmpeg clip everything to 8-bit and sRGB).
Only with this information can we reconstruct a chain of steps that
accounts for color transformations and, perhaps, keep them under
control.
Sorry, I've been pushing too much color lately, it's best to let it go.
Hermann already knew in 2012 that it was not possible to treat it in
CinCV and that's also why he went for Lumiera.
CinGG does not have support for ICC color profiles or a global color management
to standardize and facilitate the management of the various files with which it
works. But it has its own way of managing color spaces and conversions; let's
see how.
CMS
A color management describes how translates the colors of images/videos from
their current color space to the color space of the output devices like
monitors. The basic problem is to be able to display the same colors in every
device we use for editing and every device on which our work will be watched.
Calibrating and keeping our hardware under control is feasible, when instead we
come out on the internet or DVD, etc. will be impossible to maintain the same
colors on which we have worked us and so we just have to hope that there are
not too many and bad alterations. But if the basis that we have set us is
consistent, the alterations could be acceptable because they do not result from
the sum of more alterations at each step.
There are two types of color management: Display referred (DRC) and Scene
referred (SRC).
- DRC is based on having a calibrated monitor. What it displays is considered
correct and becomes the basis of our color grading. The hope is that the colors
of the final render won't change too much when displayed in other
hardware/contexts. Be careful though, there must be a color profile for each
type of color space you choose for your monitor. If we work for the web we have
to set the monitor in sRGB with its color profile, if we go out for HDTV we
have to set the monitor in rec.709 with its color profile, for 4k in Rec 2020;
for Cinema in DCP-P3; etc.
- SRC instead uses three steps: 1- The input color space: whatever it is, it
can be converted manually or automatically to a color space of your choice. 2-
The color space of the timeline: we can choose and set the color space on which
to work. 3- The colour space of the output: we can choose the colour space of
the output (on other monitors or of the final rendering). ACES and OpenColorIO
have an SRC workflow. NB: the monitor must still be calibrated to avoid
unwanted color shifts.
- There would also be a third type of CMS: the one through the LUTs. In
practice, the SRC workflow is followed through the appropriate 3D LUTs, instead
of relying on the internal (automatic) management of the program. The LUT
combined with the camera, used to display it correctly in the timeline and the
LUT for the final output. Using LUTs, however, always involves preparation,
selection of the appropriate LUT and post-correction. Also, as they are fixed
conversion tables, they can always result in clipping and banding.
Color Space
A color space consists of primaries (gamut), transfer function (gamma) and
matrix coefficients (scaler). The gamma adjustment is encoded in the transfer
function (and not in the matrix, as many people mistakenly think).
- Color primaries: the gamut of the color space associated with the media,
sensor or device (monitor, for example).
- Transfer characteristic function: converts linear values to non-linear values
(e.g. logarithmic). It is also called Gamma correction.
- Color matrix function: converts from one color model to another. RGB <-->
YUV; RGB <--> YCbCr; etc.
[Question: are these concepts correct? We needed more dettails?]
The camera sensors are always RGB and linear. Generally, those values get
converted to YUV, in the files that produce, because it is a more efficient
format thanks to chroma subsampling, and produces smaller files (even if of
lower quality, i.e. you lose part of the color data). The conversion is
nonlinear and so it concerns the "transfer characteristic" or gamma. The
encoder gets input YUV and compresses that. It stores the transfer function as
metadata, if provided. This is the type of file we will use in CinGG. Once
imported into the program, the decoder generates YUV, ideally close to the
input values, and shows it on the timeline. In CinGG, it is converted to RGB
and displayed in sRGB [???]. As you can see, in this step the gamma doesn't
play a role (but the metadata is attached, available when needed).
Monitors
Not having CinGG a CMS, it becomes essential to have a monitor calibrated and
set in sRGB that is just the output displayed on the timeline of the program.
You have these cases:
timeline: sRGB --> monitor: sRGB (we get a correct color reproduction)
timeline: sRGB --> monitor: rec709 (we get slightly dark colors)
timeline: sRGB:--> monitor: DCI-P3 (we get over-saturated colors)
timeline: rec709 --> monitor: rec709 (we get a correct color reproduction)
timeline: rec709 --> monitor: sRGB (we get slightly faded colors)
timeline: rec709 --> monitor: DCI-P3 (we get over-saturated colors)
Pipeline CMS
INPUT --> PROCESSING --> OUTPUT --> DISPLAY
Input: color space of the source file; better if combined with an ICC profile.
Processing: How CinGG transforms and uses the input file (it is a temporary
transformation, for use and consumption of the internal engine and plugins)
Output: our setting of the project for the output. This signal is taken in
charge by the operating system and the graphics card (and then transformed, if
any) to be sent to the monitor)
Display: As the monitor equipped with its color space (and profiled with ICC or
LUT) displays the signal that reaches him and that we see.
How CinGG works
CinGG, like many other similar programs, works internally at 32 bits in RGB
(and therefore 4:4:4, but the output in timeline is sRGB, as mentioned above)
and performs a first conversion from YUV of the original file to RGB with the
scaler, not the decoder, using its internal settings.
We can do without converting any media into the preferred color space, because
the quality of the internal engine is already maximum.
Specifically, using X11: "playback single step” and “plugins” cause the render
to be in the session color model, while continuous playback with no plugins
tries to use the file’s best color model for the display (for speed). See
manual 17.4.
This can create a visible effect of a switch in color in the Compositor,
usually shown as grayish versus over-bright. The cause of the issue is that X11
is RGB only and it is used to draw the "refresh" frame. So single step is
always drawn in RGB. To make a YUV frame into RGB, a color model transfer
function is used. The math equations are based on color_space and color_range.
In this case, color_range is the cause of the "gray" offset. The YUV mpeg color
range is 16..235 for Y, 16..240 for UV, and the color range used by YUV jpeg is
0..255 for YUV". (GG/Phyllis) [Question: If we set OpenGL do we have the same
behavior?]
In addition to forcing the internal motor to use sRGB, Each plugin we add
converts and uses the color information in its own way. Some limit the gamut
and depth of color by clipping (i.e. Histogram); others convert and reconvert
color spaces for their convenience; others introduce artifacts and
posterization; etc. For example, the Chroma Key (HSV) plugin converts any
signal to HSV for its operation.
If we want to better control and target this color management in CinGG, we can
take advantage of its internal ffmpeg engine:
"There is a optional feature that can be used via .opts lines from the ffmpeg
decoded files. This is via the video_filter=colormatrix=...ffmpeg plugin. There
may be other good plugins (lut3d...) that can also accomplish a desired color
transform. This .opts feature affects the file colorspace on a file by file
basis, although in principle it should be possible to setup a histogram plugin
or any of the F_lut* plugins to remap the colortable, either by table or interp.
For output, the yuv<-->rgb transformations are via the YUV class, and its
tables are initialized using YUV::yuv_set_colors. This sets up the transfer
tables for just one version (one of bt601, bt709, bt2020) [Question: transfer
function or scaler? I think scaler.] and the color range for mpeg or jpeg
[question: primaries?]. This is limited, but since the product is usually a
render, and this transform is designed to match display parameters, it is often
correct to select the output colorspace once. If the render needs a colorspace
mapping, then the session can be nested and the session output remapped by a
plugin.
This is all not very glitzy or highly automated, but it does provide a wide
color mapping capability." [GG]
In practice, in CinGG, you can have an SRC, but you have to do it manually step
by step with the filters of ffmpeg because we do not have available automatisms
and metadata management. However, with a good monitor well calibrated, we can
always use the DRC (which, we remember, until a few years ago was the std).
From Ichthyostega mail (2012)
when discussing colour models, we should consider that any conversion is
(slightly) lossy. Because you always have to interpolate an exact value
when mapping it into the other colour space. Obviously, when we use
floating point numbers to represent values, these losses become small
and close to negligible.
So, theoretically speaking, it would always be the best options to perform
all effects in the same colour space, and that colour space should be the
one of the original footage. BUT -- since the two relevant colour models
(RGB and YUV) work somewhat different, basically this requires coding
each effect twice. And on top of that, it requires a thorough understanding
of the colour models. Because this way, basically the programmer does the
conversion "in pure math", i.e. in his brain, when he adapts the algorithm
to a different colour space. Sometimes this isn't even feasible, or it is
very demanding (requiring a solid knowledge of math). Thus, in practice,
many plug-ins just have their "natural" home colour model, and either
the other operation modes are sort-of broken, or implemented in a
sub optimal way.
So, bottom line: its not desirable to have just one colour model.
- YUV is the natural model of most camcorder produced video footage
- RGB is the most easy-to-understand model for the programmers
- additionally, float is the most precise and desirable model,
but lacks support in (consumer) hardware
On colour spce selection, Cinelerra will configure the frame buffer for your
resulting video to be able to hold data in that colour model. Then, for each
plug-in, it will pick the variant of the algorithm coded for that model. If
that variant of the algorithm happens to be buggy, you've lost. We know since
several years, that some models are coded erroneously for some plug-ins,
especially when combined with an alpha channel. Unfortunately fixing that
requires really intense and systematic work; often it's not even outright
clear, how the "correct" implementation should work; thus, additionally
it would require some research and learning of theory.
We, as a community, simply didn't manage to get that done.
[...] according to Monty, Cinelerra already performs more colour
conversions than it ought to, under the hood.
The OpenGL plugin engine attempts to aggregate effects; treating a
chain of effects as one combined effect. I think the same could be
done with colour formats, avoiding a few noisy conversions.
My tests yesterday showed that even _one_ conversion is visibly lossy:
The gradient effect becomes "striped" in YUV8 mode. The gradient is
generated, so it does not process input pixels. Hence, there is only
However, conversion in FLOAT looks pretty smooth. It can't be
technically lossless, but it seems perceptually lossless, which is
our main concern here. Doesn't blur work with floats in the
inner loops anyway? And it works on a temporary buffer, since it
needs two passes, to you don't have to stick to the same component
size as the input and output.
What I'm suggesting is: Don't do the YUV<->RGB conversions in 8bit.
Always do them in FLOAT. Cinelerra has functions for that; they are
called in the YUV effect plugin. You probably have to move the
conversions closer to the inner loop to do this.
From mailing-list CV (2008)
Cinelerra loads MPEGs and still frames as-is, without thinking about it's color
range. This has a multitude of consequences. For example, when I load an MPEG4
in Cinelerra and also render it to MPEG4, the output file has the correct color
scale, because RGB value 16-16-16 was black in the original, and will be black
when playing the rendered file. However, when you render that same project to
PNGs and then use an external encoder (like mencoder) to encode to MPEG4 (which
I prefer, because mencoder is a very good encoder), the resulting video is too
bright, because the external encoder will think that the color range of PNG is
0-255, and it would be right about that. Therefore, 16-16-16 will turn out as
grey.
Right now, one has to use the RGB-601 filter to do it manually, but this is
very cumbersome. For example, if I render to any mpeg format with Cinelerra, I
have to compress the range of any format which uses 0-255, like still PNGs.
But, should I choose to render to a PNG sequence, I have to do the exact
reverse: expand the color range of all the mpeg source material and leave the
still frame as-is.
From renomath.org:
Cinelerra uses a 4:4:4 colorspace for editing; however, current versions of
Cinelerra do not properly convert interlaced 4:2:0 source to 4:4:4. The
upsampling of interlaced chroma fields is done using a progressive algorithm,
even when the interlaced option is set in the video format section. Similarly,
subsampling from 4:4:4 to 4:2:0 is always done using the progessive algorithm.
It should be pointed out that the conversion Cinelerra uses to get from 4:2:0
to 4:4:4 followed by the conversion from 4:4:4 to 4:2:0 results in the original
frame. Therefore the interlaced chroma error only effects projects that use
video effects or start with a DV source and render it to mpeg2. In particular,
the video quality of a project consisting of HDV source spliced together with
transitions and rendered through a yuv4mpeg stream to x264 will not be affected.
Suggestion
- Ultimately it would be better to set the project as RGB(A)-FLOAT, allowing
system performance, because it collects all available data and does not make
rounding errors. If we can't afford it, starting from YUV type media it is
better to set the project as YUV(A)8, so as not to have a darker rendering in
the timeline. On the contrary, if we start from RGB signals, it is better to
use RGB(A)8. (Mag) [Question: Why this behavior?]. If we don't display
correctly on the timeline, we'll make adjustments from the wrong base
(metamerism) and get false results.
- If we import multiple media types with different color spaces into the
project, an automatic CMS is counterproductive and it is worthwhile to manually
convert each file to the color space we have decided for the project.
- If we go out for internet/TV it is better to set the colour space of the
monitor to sRGB, which is the most used one. Some high quality TVs support
bt.709, so it would be good to ask about the features of your TV.
- Another tip is to specify, when rendering, the color parameters in the
relevant wrench options window. These parameters are:
colorspace= ... (color matrix)
color_trc= … (transfer characteristic or gamma)
color_primaries= … (gamut)
They can take the values, for example, bt601, bt709 and bt2020. However, you
have to understand that these parameters are only added metadata and do not
affect the render. Their importance comes out when we open the file in a video
player. “The color space information must be used explicitly so that it can be
included in the video. CinGG or FFmpeg does not write it by itself. Without
this information the players (e.g. MPV) stick to the dimensions of the video
and take the assumed color model from a table. With videos in the dimensions
from 720 to 1080 this is like written bt709. For smaller dimensions, e.g. DVD,
bt601 is assumed and for 4k and above it is bt2020. Normally this is not a
problem, but if you want to export a FullHD without color loss to a smaller
size like 576 for example, you have to inform the encoder as well as the
decoder of the player. This also applies if the videos are to be loaded on
video platforms, where they are then converted into videos of different sizes.
It is a security measure to prevent false colors, such as the color profiles in
digital photos and the copies made from them. Some players always use one or
the other matrix, even if the matrix is specified. And your graphics card
settings can effect this too.” [Olaf]
- If we work with good quality files, it is always good to do the final
rendering by setting the "pixel format" to at least 4:2:2, so as not to
compromise too much the color rendering in the video produced. We have to
evaluate the larger file size and rendering time, but if there are no problems
in this sense, the resulting increase in quality is significant.
- Try setting your video card (if you allow!) to the highest quality or color
setting. This is because it often produces a signal [16 - 235] instead of the
full signal [0 - 255].
- Set CinGG with the "YUV color space" to bt709 (HD) or bt2020 (UHD) and the
"YUV color range" to JPEG; this tip, like the previous one, does not limit the
color range to [16 - 235].
- To avoid interference from the video card and operating system, which could
impose their color space, you can use an IO device, also called mini monitor,
which collects the signal of the software used and brings it unchanged to the
monitor. They are expensive though.
-
[More suggestion?]
--
Cin mailing list
[email protected]
https://lists.cinelerra-gg.org/mailman/listinfo/cin
