Hello, I have an interesting problem I am working through and I am hoping to get some insight as to a root cause. I am taking antenna feeds that are converted to multicast into a transcoding server that is running FFMPEG. I am transcoding the streams using h264_nvenc encoder and outputting them onto the network. The issue is that the receiving servers are getting continuity errors on the transcoded streams periodically. You can also see these in the video playback in the form of very minor pixilation and slight audio skips. This only seems to effect HD stream, and only when the receiving server is virtualized.
What I have tried: Testing in 3 separate physical locations/networks Verified that only virtual machines are effected. Physical baremetal linux installed machines do not see continuity errors. Verified that the problem is only present on HD transcoded streams. SD transcoded streams and the original source streams do not display continuity issues. I noticed variation in packet size so I forced consistent packet size with the ffmpeg command. 2 separate physical transcoder servers. 2 separate models of Nvidia cards. P2000 and RTX4000. Changing the virtual machines NIC driver from VMXNET3 to E1000. Increasing buffers of the Linux OS's that are receiving the channels. Verified that it was not a resource issue on the graphics card by reducing to a single stream with one transcoded output. Verified the GPU is not under load. It is currently using less than 10% of the NVENC chip to process one of these streams. Verified that the continuities are an issue on both audio streams and the video stream. Verified the issue is present across multiple versions of FFMPEG including a newly compiled from source version. Command that I am using to transcode: ffmpeg -hwaccel nvdec -i 'udp://@239.200.200.100:6001?fifo_size=1000000&overrun_nonfatal=1' \ -vcodec h264_nvenc -bf:v 2 -g 120 -rc cbr -b:v 5500K -minrate 5500k -maxrate 5500k -bufsize 11000k -profile:v high -level 4.0 -acodec libfdk_aac -ac 2 -b:a 128k -flush_packets 0 -f mpegts 'udp://@239.200.200.103:59103' ==Command on the transcoding server to kick off the process: james@transcoder:~$ ffmpeg -hwaccel nvdec -i 'udp://@239.1.1.93:59093?overrun_nonfatal=1' -vcodec h264_nvenc -bf:v 2 -g 120 -rc cbr -b:v 5500K -minrate 5500k -maxrate 5500k -bufsize 11000k -profile:v high -level 4.0 -acodec libfdk_aac -ac 2 -b:a 128k -flush_packets 0 -f mpegts 'udp://@239.100.100.3:59003' ffmpeg version N-93005-gd92f06e Copyright (c) 2000-2019 the FFmpeg developers built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04) configuration: --prefix=/home/circle/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/home/circle/ffmpeg_build/include --extra-ldflags=-L/home/circle/ffmpeg_build/lib --extra-libs='-lpthread -lm' --bindir=/home/circle/bin --enable-gpl --enable-libaom --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree --enable-nvenc libavutil 56. 26.100 / 56. 26.100 libavcodec 58. 44.100 / 58. 44.100 libavformat 58. 26.100 / 58. 26.100 libavdevice 58. 6.101 / 58. 6.101 libavfilter 7. 48.100 / 7. 48.100 libswscale 5. 4.100 / 5. 4.100 libswresample 3. 4.100 / 3. 4.100 libpostproc 55. 4.100 / 55. 4.100 [mpeg2video @ 0x5630735fc900] Invalid frame dimensions 0x0. Last message repeated 37 times Input #0, mpegts, from 'udp://@239.1.1.93:59093?overrun_nonfatal=1': Duration: N/A, start: 52458.920733, bitrate: N/A Program 3 Stream #0:0[0x31]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], Closed Captions, 59.94 fps, 59.94 tbr, 90k tbn, 119.88 tbc Stream #0:1[0x34](eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 384 kb/s Stream #0:2[0x35](spa): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s Stream mapping: Stream #0:0 -> #0:0 (mpeg2video (native) -> h264 (h264_nvenc)) Stream #0:1 -> #0:1 (ac3 (native) -> aac (libfdk_aac)) Press [q] to stop, [?] for help Output #0, mpegts, to 'udp://@239.100.100.3:59003': Metadata: encoder : Lavf58.26.100 Stream #0:0: Video: h264 (h264_nvenc) (High), nv12, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 5500 kb/s, 59.94 fps, 90k tbn, 59.94 tbc Metadata: encoder : Lavc58.44.100 h264_nvenc Side data: cpb: bitrate max/min/avg: 5500000/0/5500000 buffer size: 11000000 vbv_delay: -1 Stream #0:1(eng): Audio: aac (libfdk_aac), 48000 Hz, stereo, s16, 128 kb/s Metadata: encoder : Lavc58.44.100 libfdk_aac frame=18186 fps= 60 q=19.0 size= 226856kB time=00:05:04.10 bitrate=6111.1kbits/s speed=1.01x ==FFProbe from the receiving server: ffprobe udp://@239.100.100.3:59003 ffprobe version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2007-2018 the FFmpeg developers built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609 configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv libavutil 54. 31.100 / 54. 31.100 libavcodec 56. 60.100 / 56. 60.100 libavformat 56. 40.101 / 56. 40.101 libavdevice 56. 4.100 / 56. 4.100 libavfilter 5. 40.101 / 5. 40.101 libavresample 2. 1. 0 / 2. 1. 0 libswscale 3. 1.101 / 3. 1.101 libswresample 1. 2.101 / 1. 2.101 libpostproc 53. 3.100 / 53. 3.100 [h264 @ 0x208f4e0] decode_slice_header error [h264 @ 0x208f4e0] no frame! [h264 @ 0x208f4e0] non-existing PPS 0 referenced Last message repeated 1 times Input #0, mpegts, from 'udp://@239.100.100.3:59003': Duration: N/A, start: 37.005333, bitrate: N/A Program 1 Metadata: service_name : Service01 service_provider: FFmpeg Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], Closed Captions, 59.94 fps, 59.94 tbr, 90k tbn, 119.88 tbc Stream #0:1[0x101](eng): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 127 kb/s ==Debugging from the receiving VM: james@transcoder:~$ ffmpeg -v debug -i udp://@239.100.100.3:59003 -f null /dev/null ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609 configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv libavutil 54. 31.100 / 54. 31.100 libavcodec 56. 60.100 / 56. 60.100 libavformat 56. 40.101 / 56. 40.101 libavdevice 56. 4.100 / 56. 4.100 libavfilter 5. 40.101 / 5. 40.101 libavresample 2. 1. 0 / 2. 1. 0 libswscale 3. 1.101 / 3. 1.101 libswresample 1. 2.101 / 1. 2.101 libpostproc 53. 3.100 / 53. 3.100 Splitting the commandline. Reading option '-v' ... matched as option 'v' (set logging level) with argument 'debug'. Reading option '-i' ... matched as input url with argument 'udp://@239.100.100.3:59003'. Reading option '-f' ... matched as option 'f' (force format) with argument 'null'. Reading option '/dev/null' ... matched as output url. Finished splitting the commandline. Parsing a group of options: global . Applying option v (set logging level) with argument debug. Successfully parsed a group of options. Parsing a group of options: input url udp://@239.100.100.3:59003. Successfully parsed a group of options. Opening an input file: udp://@239.100.100.3:59003. [udp @ 0x1e4bc20] end receive buffer size reported is 131072 [mpegts @ 0x1e4b400] Format mpegts probed with size=2048 and score=100 [mpegts @ 0x1e4b400] stream=0 stream_type=1b pid=100 prog_reg_desc= [mpegts @ 0x1e4b400] stream=1 stream_type=f pid=101 prog_reg_desc= [mpegts @ 0x1e4b400] Before avformat_find_stream_info() pos: 0 bytes read:8832 seeks:0 [h264 @ 0x1e6fd20] non-existing PPS 0 referenced Last message repeated 1 times [h264 @ 0x1e6fd20] decode_slice_header error [h264 @ 0x1e6fd20] no frame! [h264 @ 0x1e6fd20] non-existing PPS 0 referenced [mpegts @ 0x1e4b400] max_analyze_duration 5000000 reached at 5013333 microseconds st:1 [mpegts @ 0x1e4b400] rfps: 60.000000 0.007957 [mpegts @ 0x1e4b400] rfps: 59.940060 0.000000 Last message repeated 1 times [mpegts @ 0x1e4b400] After avformat_find_stream_info() pos: 3997328 bytes read:3997952 seeks:0 frames:553 Input #0, mpegts, from 'udp://@239.100.100.3:59003': Duration: N/A, start: 389.901333, bitrate: N/A Program 1 Metadata: service_name : Service01 service_provider: FFmpeg Stream #0:0[0x100], 310, 1/90000: Video: h264 (High), 2 reference frames ([27][0][0][0] / 0x001B), yuv420p(left), 1280x720 [SAR 1:1 DAR 16:9], 1001/120000, Closed Captions, 59.94 fps, 59.94 tbr, 90k tbn, 119.88 tbc Stream #0:1[0x101](eng), 243, 1/90000: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 144 kb/s Successfully opened the file. Parsing a group of options: output url /dev/null. Applying option f (force format) with argument null. Successfully parsed a group of options. Opening an output file: /dev/null. Successfully opened the file. detected 8 logical cores [graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'video_size' to value '1280x720' [graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'pix_fmt' to value '0' [graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'time_base' to value '1/90000' [graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'pixel_aspect' to value '1/1' [graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'sws_param' to value 'flags=2' [graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'frame_rate' to value '60000/1001' [graph 0 input from stream 0:0 @ 0x23b51a0] w:1280 h:720 pixfmt:yuv420p tb:1/90000 fr:60000/1001 sar:1/1 sws_param:flags=2 [AVFilterGraph @ 0x1e7c0e0] query_formats: 3 queried, 2 merged, 0 already done, 0 delayed [graph 1 input from stream 0:1 @ 0x1e4b300] Setting 'time_base' to value '1/48000' [graph 1 input from stream 0:1 @ 0x1e4b300] Setting 'sample_rate' to value '48000' [graph 1 input from stream 0:1 @ 0x1e4b300] Setting 'sample_fmt' to value 'fltp' [graph 1 input from stream 0:1 @ 0x1e4b300] Setting 'channel_layout' to value '0x3' [graph 1 input from stream 0:1 @ 0x1e4b300] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3 [audio format for output stream 0:1 @ 0x1e6ba00] Setting 'sample_fmts' to value 's16' [audio format for output stream 0:1 @ 0x1e6ba00] auto-inserting filter 'auto-inserted resampler 0' between the filter 'Parsed_anull_0' and the filter 'audio format for output stream 0:1' [AVFilterGraph @ 0x1e6be20] query_formats: 4 queried, 6 merged, 3 already done, 0 delayed [auto-inserted resampler 0 @ 0x1e74400] [SWR @ 0x1e8e2a0] Using fltp internally between filters [auto-inserted resampler 0 @ 0x1e74400] ch:2 chl:stereo fmt:fltp r:48000Hz -> ch:2 chl:stereo fmt:s16 r:48000Hz Output #0, null, to '/dev/null': Metadata: encoder : Lavf56.40.101 Stream #0:0, 0, 1001/60000: Video: rawvideo, 1 reference frame (I420 / 0x30323449), yuv420p(left), 1280x720 [SAR 1:1 DAR 16:9], 1001/60000, q=2-31, 200 kb/s, 59.94 fps, 59.94 tbn, 59.94 tbc Metadata: encoder : Lavc56.60.100 rawvideo Stream #0:1(eng), 0, 1/48000: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s Metadata: encoder : Lavc56.60.100 pcm_s16le Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native)) Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native)) Press [q] to stop, [?] for help [h264 @ 0x1eb06c0] Frame num gap 27 24 [h264 @ 0x1eb06c0] Frame num gap 27 25 [h264 @ 0x1eb06c0] no picture [mpegts @ 0x1e4b400] Continuity check failed for pid 257 expected 15 got 7 [mpegts @ 0x1e4b400] PES packet size mismatch [aac @ 0x23b4920] Number of bands (67) exceeds limit (40). Error while decoding stream #0:1: Invalid data found when processing input [aac @ 0x23b4920] channel element 2.6 is not allocated Error while decoding stream #0:1: Invalid data found when processing input [mpegts @ 0x1e4b400] Continuity check failed for pid 4096 expected 10 got 12 [mpegts @ 0x1e4b400] Continuity check failed for pid 256 expected 9 got 3 [mpegts @ 0x1e4b400] Continuity check failed for pid 0 expected 10 got 13 [h264 @ 0x1e7b5c0] Frame num gap 2 0 [h264 @ 0x1f43160] concealing 260 DC, 260 AC, 260 MV errors in I frame [mpegts @ 0x1e4b400] Continuity check failed for pid 256 expected 4 got 6 [mpegts @ 0x1e4b400] Continuity check failed for pid 0 expected 10 got 12 [mpegts @ 0x1e4b400] Continuity check failed for pid 4096 expected 10 got 12 [mpegts @ 0x1e4b400] Continuity check failed for pid 256 expected 2 got 6 [mpegts @ 0x1e4b400] Continuity check failed for pid 0 expected 13 got 15 [mpegts @ 0x1e4b400] Continuity check failed for pid 4096 expected 13 got 15 [mpegts @ 0x1e4b400] Continuity check failed for pid 17 expected 10 got 11 [h264 @ 0x20d6480] left block unavailable for requested intra4x4 mode -1 at 0 25 [h264 @ 0x20d6480] error while decoding MB 0 25, bytestream 5224 [h264 @ 0x20d6480] concealing 1649 DC, 1649 AC, 1649 MV errors in P frame frame= 6908 fps= 62 q=-0.0 Lsize=N/A time=00:01:56.09 bitrate=N/A video:648kB audio:21744kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Input file #0 (udp://@239.100.100.3:59003): Input stream #0:0 (video): 6960 packets read (79709163 bytes); 6909 frames decoded; Input stream #0:1 (audio): 5438 packets read (1856092 bytes); 5436 frames decoded (5566464 samples); Total: 12398 packets (81565255 bytes) demuxed Output file #0 (/dev/null): Output stream #0:0 (video): 0 frames encoded; 6908 packets muxed (663168 bytes); Output stream #0:1 (audio): 5436 frames encoded (5566464 samples); 5436 packets muxed (22265856 bytes); Total: 12344 packets (22929024 bytes) muxed 12345 frames successfully decoded, 2 decoding errors [AVIOContext @ 0x1e6c100] Statistics: 88821952 bytes read, 0 seeks Exiting normally, received signal 2. Thanks, -James CONFIDENTIALITY NOTICE: This e-mail including attachments is covered by the Electronic Communications Privacy Act, 18 U.S.C. ??2510-2521, is confidential and may contain information that is privileged, confidential or otherwise protected from use and disclosure. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, or dissemination of this transmission, or taking of any action in reliance on its contents, or other use is strictly prohibited. If you have received this transmission in error, please reply to the sender listed above immediately and permanently delete this message from your inbox. _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".