In a pull request, Vincent raised an interesting point regarding
frame-threading in MLT:
https://github.com/mltframework/mlt/pull/621#issuecomment-741087704
As a reminder, frame threading is when abs(consumer.get_int("real_time")) >
1. There are some problems with it.
1. sometimes there is a crash
2. image artifacts due to race conditions
3. slowness due to accessing slow-to-seek sources in non-sequential order
4. inefficient usage of CPU cache memory
5. increased memory usage
#1 and 2 have improved significantly over the years through diligence and
hard work, but it still occurs sometimes because the framework permits so
many combinations of things. I think the most recent move to use
thread-local frei0r instances (context) helped, but that also adds some
inefficiency.
For #1, in Shotcut, if an export job with frame-threading fails, Shotcut
automatically restarts the job without frame-threading. And for all of the
above reasons, Shotcut no longer defaults to enable frame-threading. I
should also point out that due to the inefficiencies, Shotcut also limits
the amount of threads to 4 (and automatically computes a lower amount on
systems with <= 4 CPU threads).
There is no solution today for #3. In the avformat producer, we could cache
decoded frames such that an out-of-order frame request will get a decoded
frame. Today, the producer caches converted video frames. Converted frames
are frames that have been converted their pixel format and colorspace to
something available in MLT based upon the get_image request such that
requesting the same frame requires no re-conversion. I do not want to cache
both because that is memory hungry. I do not want to convert every decoded
frame to reach the requested frame because that will make it slower. I
might convert this to cache unconverted decoded frames.
#4 and #5 rather come with the territory especially #5 when frames hold a
whole uncompressed image.
I want to expand the usage of slice threading. I have a branch ready to
merge that improves mlt_slices in frei0r, but you still need to opt into it
on a per-plugin basis since some of them are incompatible. Basically, this
simple change automatically adjusts the number of slices until each slice
has the same height.
I think it will be easy to add slice processing to all cases of pixel
format and colorspace conversion using swscale as proven by the existing
code for mlt_image_yuv422 in the avformat producer. Slice-based scaling is
still not reliable in swscale, and I might add a z.img scaler, which
supports tiles.
Of course, there are still more places in MLT where one can apply
mlt_slices and convert floating point code to integer. And both frei0r and
MLT can use OpenMP and more SIMD. But all that for an 8-bit pipeline? I am
not so sure.
_______________________________________________
Mlt-devel mailing list
Mlt-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mlt-devel
