I'm currently working on building a low-latency screen capture application using FFmpeg, and VP8. Unfortunately, I've run into a latency issue that I'm trying to resolve, and hoping someone can help provide some insight into x11grab. Ideally, my target is <1 sec latency, though decreasing that to 500ms or less would be ideal so my application will be more tolerant of network delays and less-than-ideal situations. In an attempt to understand latency delays, I've boiled down my testcases to the bare minimium amount of processing required.
I've used netcat and ffplay to remove as many factors as possible in isolating sources of latency. Latency was measured by the viewer using a stopwatch. Baseline video test: Streamer side: $ ./ffmpeg -f x11grab -s 1280x720 -i :0 -r 30 -deadline realtime -b:v 1M -threads 7 -qmin 0 -qmax 50 -crf 5 -f ivf pipe:1 | nc *target machine* Viewer side: $ nc -l *port* | ffplay -i ivf After consider research, I found I could use the ivf container to get the bare minimium required to record/stream VP8 data. What I've found is the following 1. Under best conditions, latency for recording from x11grab is about 500-600ms. Encoding time to VP8 is roughly 100ms. 2. When recording from V4L2, or transcoding from media that is YUV based, the latency drops to <100ms. 3. x11grab uses the MIT-SHM extension to grab the raw RGB data from X11. This has to be transcoded to YUV for almost any encoder to work (x264rgb appears to be an exception). With a variety of encoder settings, the "base latency" is about 700ms, most of which I believe is caused by x11grab. Use of other codecs as a test have shown similar results. Going further, I've identified other points that have caused latency, including the use of webm and such. Container latency test: $ ./ffmpeg -f x11grab -s 1280x720 -i :0 -r 30 -deadline realtime -b:v 1M -threads 7 -qmin 0 -qmax 50 -crf 5 -f webm pipe:1 | nc *target machine* Viewer side: $ nc -l *port* | ffplay -i webm When switching to using webm with only a video stream, we found that another 400ms of delay was added. Research suggests thats because webm is a space-based container, and not really well suited to low latency delays. This has brought our total latency delay to approximately 1-1.3 seconds. Adding audio: Ideally, we'd like to stream audio together with video in a container. Unfortunately, this added a even worse delay. I tested multiple codecs, and opus gave us the "best" results, though it was still not great. Audio latency test: $ ./ffmpeg -f x11grab -s 1280x720 -i :0 -f alsa -i pulse -acodec libopus -deadline realtime -vcodec libvpx -deadline realtime -b:v 1M -threads 7 -qmin 0 -qmax 50 -crf 5 -f webm pipe:1 Viewer side: $ nc -l *port* | ffplay -i webm Adding opus audio brought our total latency up to approximately two seconds, though this would vary. From the viewer side, audio and video were synced and not dropping frames, it presented itself as a constant delay. Other codecs and containers were tried, getting roughly similar results across the spectrum. What I'd like to determine is the following: - Is it possible to get better performance from x11grab? As best I can tell, most of the latency appears to becoming from having to transcode from RGB to YUV formats internally within FFmpeg; grabbing a raw YUV video source did not present the same latency delay - The muxer latency hit is painful. If we could get x11grab to go faster, it would be livable, though adding audio has still caused us to be far outside out latency target - Is there a better way to handle this situation? We could handle doing two independent streams, one for audio, and one for video. FFmpeg on the streamer side was compiled as follows: ffmpeg version N-75472-g7086154 Copyright (c) 2000-2015 the FFmpeg developers built with gcc 4.9.2 (Ubuntu 4.9.2-0ubuntu1~14.04) configuration: --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-fontconfig --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-opengl --enable-x11grab --enable-libxvid --enable-libdc1394 --enable-frei0r --enable-libx264 --enable-libsoxr --enable-openal --prefix=/opt/ffmpeg --enable-static libavutil 55. 2.100 / 55. 2.100 libavcodec 57. 3.100 / 57. 3.100 libavformat 57. 2.100 / 57. 2.100 libavdevice 57. 0.100 / 57. 0.100 libavfilter 6. 6.100 / 6. 6.100 libavresample 3. 0. 0 / 3. 0. 0 libswscale 4. 0.100 / 4. 0.100 libswresample 2. 0.100 / 2. 0.100 libpostproc 54. 0.100 / 54. 0.100 Viewer side was using a binary build of FFmpeg for Windows with VP8 but I don't think decoding speed was a huge problem in terms of latency. Any insight anyone could provide would be a huge help. Please Cc replies to me, as I'm not subscribed to the list (though I will check the list archives in case a message gets sent there). Thanks for your time in advance, Michael Casadevall _______________________________________________ ffmpeg-user mailing list [email protected] http://ffmpeg.org/mailman/listinfo/ffmpeg-user
