On Sat, Sep 3, 2022 at 11:57 AM Brian Matherly via Mlt-devel <mlt-devel@lists.sourceforge.net> wrote: > > Thanks for investigating this topic. Here are some comments from me: > > 1) I support an attempt to reduce unnecessary image format conversions. Your > focus is on track compositing. But it would also be good if we could find a > general pattern that would work for filters as well. Some filters can operate > on multiple image formats. But there is not a good way for the filter to know > what image format to request from get_image to result in the fewest > conversions. > > 2) I am not excited about adding more parameters to the get_image function. > But I understand this might be a necessary concession to improve efficiency. > > 3) I have a long-term vision to replace the current get_image function which > takes multiple parameters with a new version that takes a single mlt_image > structure. In general, I would like to increase the use of mlt_image objects > in the framework. In my current vision, we could add a new get_image function > that passes an mlt_image and then create a wrapper that allows legacy > services to continue to work until they are converted to use mlt_image. Maybe > your suggestion could help to move us in that direction. >
Just to be clear, his proposal is not to add a function parameter to either the get_frame or get_image functions. Rather to proactively set the existing (image) "format" property within each producer's get_frame function. This can inform downstream services about which image format they request in a call to mlt_frame_get_image(). The idea makes some sense, is easy to add, and can even be optional. However, it seems it can be easily defeated as JB pointed out with "effects can change the format of a frame." Still, it is a low impact, low effort change worthy of the experiment. IOW, I am OK with it. Let's see the PR. > 4) I had previously thought of an idea to add a new function called > get_image_dry_run() (or something like that) which would allow a service to > query the next service/producer to find out what format would be returned. I > think the function could return the image type (or maybe a list of possible > types) that would be returned when the service calls get_image(). The > downside to this idea is that someone would need to implement this new > function for all services. But maybe there could be a default implementation > that would just be a pass-through and we would only need to implement the new > function for some key services. > > 5) As you mentioned, our current method to find out if an image has any > non-zejust ro alpha pixels is to request the alpha mask and then seek for > values in the returned mask.. I wonder if an image should have a variable > called "alpha_status" that can have one of three values: "alpha", "no_alpha" > and "unknown". If the framework is certain that there is no alpha (for > example, because it just converted RGB to RGBA), then it would set the status > to "no alpha". In the case of unknown, a service could request a full scan of > the alpha channel to determine the status. Some accounting would be required > to make sure that services are setting the variable appropriately when > modifying an image. > > 6) It would be good if we had a better way to measure the number of > conversions that are occurring on an image/frame for a given configuration. > My current thought is that if we can make the use of mlt_image more prolific, > then we could add a "conversion_count" variable that gets incremented every > time a conversion occurs on the image. Maybe there are other ideas that could > help us better understand how many conversions are occuring. > > Those are some comments from me. I would be interested in continued > development of this optimization since it could have a huge impact on the > framework performance. I do not think most of the above have a good return on investment. The best way to reduce image conversions is not to require them. I have been slowly working on that over time in little bites including on the audio side. For example, one could choose RGB throughout as Movit and Olive are doing. If there will ever be another high bit depth, high performance image processing framework plugged in it should be operating in linear RGB just like these. Of course that necessitates an input and output conversion even when there is no processing required, but these can be done quickly now. That could be further optimized by carrying the original image data and pixel format on a frame, clearing that when dirtied, and only doing a single conversion on output if needed. Another approach that can be done within the existing framework and services is to: 1. identify all YUV only services 2. convert some of these to also handle RGB(A) 3. find suitable replacements for some from libavfilter 4. deprecate the remaining and hide them in the UI to prevent users from adding them 5. optimize the opaque alpha test functions This approach would also improve full color range end-to-end. That works now today but only properly when avoiding services that force a conversion to limited range mlt_image_yuv*. > Regards, > > ~Brian > > > > On Friday, September 2, 2022 at 12:38:40 PM CDT, jb <j...@kdenlive.org> wrote: > > > Hi all, > > In Kdenlive, we often have projects with multiple tracks, and track > compositing is one of the bottleneck we have with MLT. > > Shotcut uses the frei0r.cairoblend transition for compositing. cairoblend > first > requests an rgba image for the top (b_frame), and checks if it really has > transparency with an "is_opaque" function that check the alpha value of every > pixel in the image. The downside is that if we have an opaque video on the top > track, this causes a noticeable slowdown when playing. > > In the qtblend transition, I tried to work around it be first requesting the > top (b frame) image in the consumer requested format (usually yuv). Then we > check if the frame has an alpha channel using the "mlt_frame_get_alpha" > function. If there is no alpha, we directly return the frame, otherwise we > do another request for an rgba image to process. The advantage over cairoblend > is that there is almost no overhead when the top track contains an opaque > video. The downside is that there is a noticeable overhead when there is an > alpha channel as we first request an yuv frame and then an rgba. > > So to improve this an try to reduce yuv <> rgb conversions, I was thinking to > add a property to all producers "producer_get_frame" method (that is defined > before we attempt to fetch an image), to inform the framework about the > default mlt_image_format produced by the producer. For example, the image > producer would set the "format" property to mlt_image_rgba if it is an image > with alpha channel, and mlt_image_rgb if there is no alpha channel. This > information could then be retrieved by the framework to optimize the frame > conversion processes. > > In the case of my qtblend transition, if we know that the producer creates > rgba images by default, we can directly retrieve the rgba frame, leading to > much better performance. > > Of course, effects can change the format of a frame, adding or removing an > alpha channel, but in many cases this would allow us to optimize the > performance. > > What do you think of this ? If the idea seems ok, I volunteer to produce a > patch implementing the feature (I already tested it for qimage and > kdenlivetitle producers). > > If you don't like the idea, do you have any better idea to implement a good > performance transition that would work efficiently for both cases (top frames > with and without alpha channel) ? > > Thanks in advance and best regards, > > Jean-Baptiste > > > > > > > _______________________________________________ > Mlt-devel mailing list > Mlt-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/mlt-devel > _______________________________________________ > Mlt-devel mailing list > Mlt-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/mlt-devel _______________________________________________ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
