I did a lot of video processing in the past.
What you want is a frameserver like
[Avisynth](http://avisynth.nl/index.php/First_script) (Windows-only,
multithreading an afterthought, custom scripting language) or
[Vapoursynth](http://www.vapoursynth.com/doc/gettingstarted.html#example-script)
(Windows, Mac, Linux, scripting through Python)
Example Vapoursynth to flip a video:
from vapoursynth import core
video = core.ffms2.Source(source='Rule6.mkv')
video = core.std.Transpose(video)
video.set_output()
This can be forwarded to mencoder, ffmpeg, x264 or x265 encoder. There are
facilities to address particular frame, use GPU for lots of thing ([example
page of all the denoising
filters](http://avisynth.nl/index.php/External_filters#Spatio-Temporal_Denoisers)
including GPU accelerated)
This is a huge undertaking between SIMD, platform specific video API,
multiprocessing, the avisynth/vapoursynth API, creating fast filters (which
usually needs assembler ...), mathematical/signal processing knowledge.
