On samedi, 3 septembre 2022 22.15:13 h CEST Dan Dennedy wrote: > On Sat, Sep 3, 2022 at 11:57 AM Brian Matherly via Mlt-devel > > <mlt-devel@lists.sourceforge.net> wrote: > > Thanks for investigating this topic. Here are some comments from me: > > > > 1) I support an attempt to reduce unnecessary image format conversions. > > Your focus is on track compositing. But it would also be good if we could > > find a general pattern that would work for filters as well. Some filters > > can operate on multiple image formats. But there is not a good way for > > the filter to know what image format to request from get_image to result > > in the fewest conversions. > > > > 2) I am not excited about adding more parameters to the get_image > > function. But I understand this might be a necessary concession to > > improve efficiency. > > > > 3) I have a long-term vision to replace the current get_image function > > which takes multiple parameters with a new version that takes a single > > mlt_image structure. In general, I would like to increase the use of > > mlt_image objects in the framework. In my current vision, we could add a > > new get_image function that passes an mlt_image and then create a wrapper > > that allows legacy services to continue to work until they are converted > > to use mlt_image. Maybe your suggestion could help to move us in that > > direction. > Just to be clear, his proposal is not to add a function parameter to > either the get_frame or get_image functions. Rather to proactively set > the existing (image) "format" property within each producer's > get_frame function. This can inform downstream services about which > image format they request in a call to mlt_frame_get_image(). The idea > makes some sense, is easy to add, and can even be optional. However, > it seems it can be easily defeated as JB pointed out with "effects can > change the format of a frame." Still, it is a low impact, low effort > change worthy of the experiment. IOW, I am OK with it. Let's see the > PR.
Thanks to both of you for sharing your thoughts on this matter. I will work further on this and can hopefully propose a PR in 10 days. As Dan pointed out, my proposal is a very modest change that could spare a few conversions between YUV and RGB. In my tests using qtblend, rendering an HD video with a transparent image overlay, I get a 20% speed bump by directly requesting the correct rgba image format instead of doing an rgba > yuv > rgba conversion. > > > 4) I had previously thought of an idea to add a new function called > > get_image_dry_run() (or something like that) which would allow a service > > to query the next service/producer to find out what format would be > > returned. I think the function could return the image type (or maybe a > > list of possible types) that would be returned when the service calls > > get_image(). The downside to this idea is that someone would need to > > implement this new function for all services. But maybe there could be a > > default implementation that would just be a pass-through and we would > > only need to implement the new function for some key services. > > > > 5) As you mentioned, our current method to find out if an image has any > > non-zejust ro alpha pixels is to request the alpha mask and then seek for > > values in the returned mask.. I wonder if an image should have a variable > > called "alpha_status" that can have one of three values: "alpha", > > "no_alpha" and "unknown". If the framework is certain that there is no > > alpha (for example, because it just converted RGB to RGBA), then it would > > set the status to "no alpha". In the case of unknown, a service could > > request a full scan of the alpha channel to determine the status. Some > > accounting would be required to make sure that services are setting the > > variable appropriately when modifying an image. > > > > 6) It would be good if we had a better way to measure the number of > > conversions that are occurring on an image/frame for a given > > configuration. My current thought is that if we can make the use of > > mlt_image more prolific, then we could add a "conversion_count" variable > > that gets incremented every time a conversion occurs on the image. Maybe > > there are other ideas that could help us better understand how many > > conversions are occuring. > > > > Those are some comments from me. I would be interested in continued > > development of this optimization since it could have a huge impact on the > > framework performance. > I do not think most of the above have a good return on investment. The > best way to reduce image conversions is not to require them. I have > been slowly working on that over time in little bites including on the > audio side. For example, one could choose RGB throughout as Movit and > Olive are doing. If there will ever be another high bit depth, high > performance image processing framework plugged in it should be > operating in linear RGB just like these. Of course that necessitates > an input and output conversion even when there is no processing > required, but these can be done quickly now. That could be further > optimized by carrying the original image data and pixel format on a > frame, clearing that when dirtied, and only doing a single conversion > on output if needed. Another approach that can be done within the > existing framework and services is to: > > 1. identify all YUV only services > 2. convert some of these to also handle RGB(A) > 3. find suitable replacements for some from libavfilter > 4. deprecate the remaining and hide them in the UI to prevent users > from adding them > 5. optimize the opaque alpha test functions > > This approach would also improve full color range end-to-end. That > works now today but only properly when avoiding services that force a > conversion to limited range mlt_image_yuv*. Yes, I like this idea. At least in my experience, the current format conversions have a noticeable impact on performance and having all services processing the same format would be cleaner and faster. Thanks and will come back soon with some patch regarding my format trick. Jean-Baptiste > > > Regards, > > > > ~Brian > > > > > > > > On Friday, September 2, 2022 at 12:38:40 PM CDT, jb <j...@kdenlive.org> > > wrote: > > > > > > Hi all, > > > > In Kdenlive, we often have projects with multiple tracks, and track > > compositing is one of the bottleneck we have with MLT. > > > > Shotcut uses the frei0r.cairoblend transition for compositing. cairoblend > > first requests an rgba image for the top (b_frame), and checks if it > > really has transparency with an "is_opaque" function that check the alpha > > value of every pixel in the image. The downside is that if we have an > > opaque video on the top track, this causes a noticeable slowdown when > > playing. > > > > In the qtblend transition, I tried to work around it be first requesting > > the top (b frame) image in the consumer requested format (usually yuv). > > Then we check if the frame has an alpha channel using the > > "mlt_frame_get_alpha" function. If there is no alpha, we directly return > > the frame, otherwise we do another request for an rgba image to process. > > The advantage over cairoblend is that there is almost no overhead when > > the top track contains an opaque video. The downside is that there is a > > noticeable overhead when there is an alpha channel as we first request an > > yuv frame and then an rgba. > > > > So to improve this an try to reduce yuv <> rgb conversions, I was thinking > > to add a property to all producers "producer_get_frame" method (that is > > defined before we attempt to fetch an image), to inform the framework > > about the default mlt_image_format produced by the producer. For example, > > the image producer would set the "format" property to mlt_image_rgba if > > it is an image with alpha channel, and mlt_image_rgb if there is no alpha > > channel. This information could then be retrieved by the framework to > > optimize the frame conversion processes. > > > > In the case of my qtblend transition, if we know that the producer creates > > rgba images by default, we can directly retrieve the rgba frame, leading > > to > > much better performance. > > > > Of course, effects can change the format of a frame, adding or removing an > > alpha channel, but in many cases this would allow us to optimize the > > performance. > > > > What do you think of this ? If the idea seems ok, I volunteer to produce a > > patch implementing the feature (I already tested it for qimage and > > kdenlivetitle producers). > > > > If you don't like the idea, do you have any better idea to implement a > > good > > performance transition that would work efficiently for both cases (top > > frames with and without alpha channel) ? > > > > Thanks in advance and best regards, > > > > Jean-Baptiste > > > > > > > > > > > > > > _______________________________________________ > > Mlt-devel mailing list > > Mlt-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/mlt-devel > > _______________________________________________ > > Mlt-devel mailing list > > Mlt-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/mlt-devel _______________________________________________ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
